Between 18:21 and 19:18 UTC on September 11, 2022, Object Storage services in Atlanta were unreachable.
Linode received initial alerts for Object Storage services in Atlanta starting at 18:25 UTC. These alerts showed associated services repeatedly appearing to go down and up.
Initial investigations into Object Storage health in Atlanta presented with significant load times ending in timeouts to Atlanta OBJ buckets. By 18:39 UTC, Linode staff observed that Cloud Manager was displaying a banner stating that Object Storage buckets in Atlanta were not reachable.
Linode engaged its Incident Response Procedure at 18:51 UTC, for which they paged a subject matter expert at 18:59 UTC and published a status page at 19:01 UTC. Object Storage services in Atlanta fully recovered by 19:18 UTC.
After the immediate impacts of the incident were complete, a deeper investigation into the outage began. At the time of the incident, it was believed that a large volume of traffic inbound to the data center was responsible for this behavior. However, later investigations strongly suggested that the most likely cause was an automated routine maintenance procedure on Object Storage infrastructure not functioning properly.
To mitigate this situation moving forward, Linode will be improving its internal alerting, logging and documentation to better detect and resolve this issue sooner. Linode will also be attempting to prevent this situation from occurring in the future by further reviewing the Object Storage infrastructure for factors that may instigate this behavior.