From 23:20 UTC on March 27, 2024 to 2:40 UTC on March 28, 2024, our London data center faced sporadic disruptions in connectivity. Users experienced increased network latency, packet loss, and intermittent periods of complete connectivity loss during these hours.
After investigation, the Network Operations team identified that the connectivity disruptions were due to a cut fiber connection in an upstream provider’s network. This resulted in routes fluctuating and switching to alternative transit providers. At approximately 06:00 UTC, the affected routes were disabled, and stable connectivity was observed.
At 08:08 UTC on March 27, 2024, the Network Operations team entered a monitoring period that lasted until 19:06 UTC on 31 March 2024. During that time, there was a brief connectivity disruption that was resolved automatically.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing in cooperation with our third party providers, and any information stated is subject to change.
The issue was most severe between 2:30 UTC and 3:30 UTC but continued to occur intermittently. The most recent disruption was recorded around 05:45 UTC and is still ongoing. Our network team confirmed this is related to a fiber connection cut in an upstream provider’s network.
Our team is actively mitigating the issue and working to reroute traffic through paths that are currently stable and free from loss. A significant portion of the traffic was routed over to a stable link at 06:30 UTC.