At around 09:00 UTC on 06-03-2021, we began observing networking issues in our London and Frankfurt data centers as customers reported high latency, packet loss, and failed migrations.
During that time one of our carriers was performing scheduled maintenance on a primary link between our London and Frankfurt data centers, and traffic was redirected to a redundant link. For a brief period during this maintenance, the redundant link also failed which caused cascading connection issues between our North America and EU DCs.
Concurrently, though unrelated to the maintenance, we also observed issues on our link between Newark and London. This compounded the connectivity issues we were experiencing between our American and European centers.
At around 11:00 UTC, the issues on our link between Newark and London were resolved. At around 12:00 UTC the scheduled maintenance and the restoration of the primary link was completed, which fully restored our trans-Atlantic and trans-European network connectivity.
As part of Linode's efforts to prevent this type of situation from occurring again, we are commissioning a replacement dark fiber link in London. This postmortem also relates to the post for connectivity issues in Frankfurt which occurred at the same time. We do not anticipate these events to recur.