Linode Status

Current Status
Emergency Maintenance - US-East (Newark)
Scheduled Maintenance Report for Linode
Postmortem

At 05:00 UTC on 2022-01-21, our engineers began routine maintenance on the edge routers in our Newark data center. These routers consist of a redundant pair.

One router in this redundant pair was taken offline at 07:50 UTC as part of the maintenance procedure. At that point, Newark's Internet-bound traffic was handled exclusively by the remaining router.

At 08:30 UTC the upgrade on the offline router concluded and it re-established a connection to the online router. At this point, we had still not brought up the links between the offline router and our internal LAN. However, at 08:32 UTC, the online router began sending traffic bound for some of our intermediate network devices to the offline router. This led to 100% packet loss to the subset of Newark Linodes that were served by these devices.

The link to the offline router was taken down at 08:47 UTC, which restored connectivity to all affected Linodes. After a followup investigation to ensure that there were no additional problems on the offline router, it was brought back online at 10:00 UTC.

This incident was an unforeseen failure scenario, in which the routing configurations on our edge devices led them to forward traffic in an unintended way during what should have been routine maintenance. As a result of this incident, we have audited and updated the configurations on our edge routers to ensure that this failure scenario cannot happen again.

If you have any further questions about this incident, please open a Support ticket for assistance.

Posted Jan 28, 2022 - 20:46 UTC

Completed
The scheduled maintenance has been completed.
Posted Jan 21, 2022 - 10:27 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Jan 21, 2022 - 09:28 UTC
Scheduled
We will be undergoing scheduled maintenance during this time.
Posted Jan 21, 2022 - 09:26 UTC
This scheduled maintenance affected: Regions (US-East (Newark)).