Linode Status

Current Status
Connectivity Issue - Mumbai, IN
Incident Report for Linode

On 2 November 2021 at 01:48 UTC, hardware failure took down one of the edge routers in our Mumbai data center. These routers are configured in a redundant pair, and the loss of a single device should not have affected connectivity for our Mumbai customers.

However, a specific /24 subnet in Mumbai was only reachable via a single edge router due to a misconfiguration. When that router went down, there was a total loss of connectivity for customers using these IPs.

Our engineering team became aware of the impact of the offline device on 2 November at 02:43 UTC. Work began to bring the router back online, which we completed on 2 November at 20:06 UTC. At this point, connectivity to this range of IPs was restored. While the maintenance necessary to bring the device back online was time consuming, other factors also contributed to the extended time between detection and resolution, including blind spots in our monitoring and response policies. We have taken steps to address these issues and reduce the time to resolution for future incidents.

Further investigation revealed that when we made these IP addresses available to customers, only a portion of our upstream Internet providers had accepted the new prefix. The loss of one of our edge routers was therefore sufficient to take any Linodes using IP addresses within this range offline.

This was an oversight on our part and we have taken steps to ensure that provisioning new IP addresses does not impact our customers in this way going forward.

If you have any further questions regarding this incident, please reach out to Linode Support.

Posted Nov 12, 2021 - 23:03 UTC

This incident has been resolved.
Posted Nov 02, 2021 - 21:09 UTC
A fix has been implemented and we are monitoring the results.
Posted Nov 02, 2021 - 20:07 UTC
Our networking team is continuing to work on a fix.
Posted Nov 02, 2021 - 18:19 UTC
Our networking team is continuing to implement a fix.
Posted Nov 02, 2021 - 16:46 UTC
Our networking team is still working on resolving this issue.
Posted Nov 02, 2021 - 15:32 UTC
We are continuing to work on a fix for this issue.
Posted Nov 02, 2021 - 14:18 UTC
The issue has been identified and a fix is being implemented.
Posted Nov 02, 2021 - 12:56 UTC
Our team is investigating a connectivity issue in our Mumbai data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information.
Posted Nov 02, 2021 - 12:44 UTC
This incident affected: Regions (AP-West (Mumbai)).