Linode Status

Current Status
Service Issue - Cross-Data Center Migrations (Toronto)
Incident Report for Linode
Postmortem

At approximately 01:15 UTC on October 1, 2022, our administrators were alerted by the Support team that customers were experiencing failures of cross-data center migrations of Linodes out of the Toronto data center, particularly to London and Frankfurt. Affected migrations would have shown as being stuck at 40% progress in Cloud Manager.

An investigation into the failures began, and the cause was ultimately identified as a network configuration inconsistency between Toronto’s compute hosts and the rest of Toronto’s internal network. Following configuration updates at 09:30 UTC on October 1, 2022, migrations of Linodes leaving Toronto for other data centers were successful.

Once the immediate impact was fixed, Linode proceeded with a deeper review of the circumstances leading to this problem. It was identified that this configuration inconsistency had previously existed within Toronto’s infrastructure, but without any negative effects. It was only after an upgrade to the software on Linode’s compute hosts on September 28, 2022 that this inconsistency caused a meaningful disruption of service. Cross-data center migrations from other data centers were unaffected by this issue.

To prevent this issue from occurring again, Linode will be implementing a number of improvements to its software testing procedures and network configuration that will account for this failure method.

Timeline:Sep 29, 2022 01:20 UTC -- First migration failure for Toronto occurs

Oct 01, 2022 18:16 UTC -- Linode Support receives alert about migration failure, starts investigation

Oct 01, 2022 20:45 UTC -- Support escalates to Platform Operations

Oct 01, 2022 21:08 UTC -- Platform Operations acknowledges escalation and continues investigation

Oct 01, 2022 23:45 UTC -- Platform Operations pages to System Operations on-call

Oct 02, 2022 02:30 UTC -- Platform and System Operations pages the on-call network administrator 

Oct 02, 2022 04:00 UTC -- Network administrator identifies the issue and start exploring potential fixes

Oct 02, 2022 05:30 UTC -- Network administrator fixes the problem

Oct 02, 2022 05:40 UTC -- Beginning of test migration from Toronto to verify fix

Oct 02, 2022 06:00 UTC -- Migration successful,  status page moved to Monitoring

Oct 02, 2022 07:00 UTC -- Status page moved to Resolved

Posted Oct 08, 2022 - 04:18 UTC

Resolved
We haven’t observed any additional issues with cross-data center migrations to and from the Toronto DC. The incident has been resolved.
Posted Oct 02, 2022 - 11:00 UTC
Monitoring
At this time we have been able to correct the issues affecting cross-data center migrations in our Toronto data center. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Oct 02, 2022 - 10:03 UTC
Update
Our team has identified the issue affecting cross-data center migrations in our Toronto data center. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Posted Oct 02, 2022 - 09:30 UTC
Update
We are continuing to investigate an issue with cross-data center migrations to and from the Toronto DC.
Posted Oct 02, 2022 - 08:16 UTC
Investigating
Our team is investigating a service issue affecting migrations of Linode from Toronto to other data center. During this time, users may experience issues when attempting to migrate Linodes from Toronto to other data centers, particularly London and Frankfurt. We will share additional updates as we have more information.
Posted Oct 02, 2022 - 06:06 UTC
This incident affected: Regions (CA-Central (Toronto)).