Linode Status

Current Status
Connectivity Issue - EU Central (Frankfurt)
Incident Report for Linode
Postmortem

At 16:00 UTC on 1/21/21, the Linode Network Operations team began a routine scheduled network maintenance to replace a failing core switch (“Switch-B”). This switch is part of an active/active redundant pair, and replacement should have been non-customer affecting. The shutdown, removal and installation of Switch-B proceeded as planned. During the operation of turning up and bringing Switch-B online and back into active service, something triggered the primary switch (“Switch-A”) to go into a failed state. This caused an outage, then latency and packet loss for customers downstream from this switch pair. 

Working quickly, the NetOps team began to triage the problem but was unable to determine the immediate cause of the failure. Opting to rollback and shutdown Switch-B brought Switch-A back online, restoring service to affected customers. However, this left the network in a degraded state with only one switch in service. An emergency maintenance for 1/22/2021 at 3:00 UTC was scheduled to bring the faulty switch back online. During this maintenance we experienced the same type of outage. 

The NetOps team continued working to bring Switch-B into full service, thus restoring connectivity and redundancy. A full investigation is underway; however, initially it seems that a mechanism designed to protect the redundancy was triggered and caused Switch-A to enter a failed state.

Posted Jan 27, 2021 - 22:03 UTC

Resolved
We haven’t observed any additional connectivity issues in our Frankfurt data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Jan 22, 2021 - 03:02 UTC
Monitoring
At this time we have been able to correct the issues affecting connectivity in our Frankfurt data center. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Jan 22, 2021 - 01:07 UTC
Investigating
Our team is investigating a connectivity issue in our Frankfurt data center related to emergency network maintenance. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information.
Posted Jan 22, 2021 - 00:40 UTC
This incident affected: Regions (EU-Central (Frankfurt)).