At approximately 22:47 EST on November 11, 2016, the Atlanta datacenter experienced a brief utility power sag from Georgia Power. The power sag was attributed to a vehicle striking a nearby power pole. The issue was identified by our colocation provider’s building monitoring system and their on-site staff began investigating immediately. They discovered that one of the UPS modules failed to operate correctly.
When the UPS module dropped out, the remaining modules took longer than expected to absorb and carry the additional load. This produced an extremely brief voltage sag resulting in a loss of critical power to Linode’s hardware deployment. The remaining UPS modules recovered within milliseconds and have been stable since.
The on-site UPS engineers have visually inspected all of the UPS system’s components and verified that no faults exist. The UPS vendor has also performed an on-site inspection of the UPS system and was unable to determine the root cause. The vendor has tested the UPS system and put all modules back into service. All systems have been operating as intended since then.
As per our colocation provider’s facilities team, these particular UPS units have been slated for replacement as a precautionary measure. The anticipated project completion date is Q1 2017.
Several Linode host machines did not survive the sudden loss of power and needed individual attention. Linode engineers worked after the power was restored in order to make these systems operational again. We were able to recover every system.
We do not anticipate any further issues with the Atlanta facility at this time. Thank you for your patience. We are sorry for the outage and the problems it caused.