At 19:59 UTC on November 2, 2023, Atlanta experienced a network disruption after a routine network maintenance procedure on the gateway routers. Immediately upon observing this impact, incident response procedures were formally declared at 20:03 UTC. Working with the incident procedure, the network engineer performing the maintenance identified the problematic aspect on one of the Atlanta gateways.
After isolating this aspect from the production environment at 20:17 UTC, network performance in Atlanta returned to normal. Upon additional monitoring for further immediate issues, it was confirmed that the impact of this event had been successfully mitigated, and the incident was moved to a resolved status at 22:52 UTC.
For a longer-term fix, the network engineering team engaged the vendor, who suggested rebooting one of the gateway routers entirely. A maintenance window was declared for November 8, 2023 between 05:00 and 08:00 UTC. After this reboot was completed, network performance in Atlanta returned to normal without any further need for component isolation.
To prevent this incident from occurring in the future, Linode's network engineering team will explicitly schedule maintenance windows for the type of maintenance which originally resulted in this incident.