Datacenter Outage - Atlanta
Incident Report for Linode

At approximately 22:47 EST on November 11, 2016, the Atlanta datacenter experienced a brief utility power sag from Georgia Power. The power sag was attributed to a vehicle striking a nearby power pole. The issue was identified by our colocation provider’s building monitoring system and their on-site staff began investigating immediately. They discovered that one of the UPS modules failed to operate correctly.

When the UPS module dropped out, the remaining modules took longer than expected to absorb and carry the additional load. This produced an extremely brief voltage sag resulting in a loss of critical power to Linode’s hardware deployment. The remaining UPS modules recovered within milliseconds and have been stable since.

The on-site UPS engineers have visually inspected all of the UPS system’s components and verified that no faults exist. The UPS vendor has also performed an on-site inspection of the UPS system and was unable to determine the root cause. The vendor has tested the UPS system and put all modules back into service. All systems have been operating as intended since then.

As per our colocation provider’s facilities team, these particular UPS units have been slated for replacement as a precautionary measure. The anticipated project completion date is Q1 2017.

Several Linode host machines did not survive the sudden loss of power and needed individual attention. Linode engineers worked after the power was restored in order to make these systems operational again. We were able to recover every system.

We do not anticipate any further issues with the Atlanta facility at this time. Thank you for your patience. We are sorry for the outage and the problems it caused.

Posted Dec 14, 2016 - 23:27 UTC

The last remaining host is now operational and is booting.
Posted Nov 12, 2016 - 14:02 UTC
All hosts but one are up and running, and have been for the past few hours. We're still working on restoring one remaining host.
Posted Nov 12, 2016 - 12:40 UTC
93% of the hosts in Atlanta are up and running and we are working to restore the remaining hosts at this time.
Posted Nov 12, 2016 - 07:08 UTC
We've booted 90% of hosts in Atlanta at this time, and we're working to to get the remaining 10% up and running as soon as possible.
Posted Nov 12, 2016 - 05:40 UTC
We've identified the issue as a datacenter-wide power outage. The datacenter now has backup generators on-line and we are working towards booting the affected hosts now. In addition, we are using this boot to update Atlanta hosts where needed instead of rebooting them at a later date.
Posted Nov 12, 2016 - 04:35 UTC
We are aware of connectivity issues affecting Linodes in our Atlanta data center and are currently investigating. We will continue to provide additional updates going forward.
Posted Nov 12, 2016 - 03:56 UTC
This incident affected: Regions (US-Southeast (Atlanta)).