Starting at approximately October 30, 2022 at 06:30 UTC, Linode first became aware of connectivity issues impacting customers in the London data center. After investigating the customer reports, Linode’s system administrators could not identify the cause of this impact and paged out to the on-call Network Engineering team member at 7:00 UTC.
Network Engineering was able to determine the presence of a network issue starting at approximately 02:00 UTC. Upon investigating the issue, Linode’s system administrators and Network Engineering team were able to provide small improvements to the situation, but could not completely resolve it. As a result, Linode formally began its incident response procedure at approximately 09:11 UTC, with a status page going live at 10:36 UTC.
Linode’s engineering teams implemented various measures that were expected to alleviate the issue. The issue appeared to be resolved by 13:20 UTC, and the status page was updated at this time.
However, connectivity issues returned to London on a fluctuating basis at approximately 15:44 UTC. Due to renewed customer impact, Linode posted a new status page at 16:27 UTC. Linode immediately started reviewing the situation for resolution pathways, making a change to the network configuration at 16:35 UTC.
During this time, an unrelated issue occurred for certain customers trying to access the London data center. This issue received a separate status page:
Similar recurrences of this issue occurred throughout the week, including at the following times:
- November 3 09:57 UTC to November 4 02:00 UTC
- November 6 06:30 UTC to November 6 08:15 UTC
- November 6 12:00 UTC to November 6 13:05 UTC
- November 6 14:35 UTC to November 6 15:45 UTC
Linode will be reviewing the data gathered during these incident periods to implement longer-term changes into its network.