Linode Status

Current Status
Service Issue - Block Storage (Toronto, Frankfurt, Newark, Sydney, and Fremont)
Incident Report for Linode
Postmortem

On December 3, 2024, at around 09:00 UTC, block storage clusters in Toronto and Frankfurt went down, leading to customer-facing issues such as connection timeouts and errors. The problem was resolved by 10:30 UTC after force-restarting the affected devices. A similar issue occurred on December 5, 2024, between 06:40 UTC and 07:49 UTC, impacting the Newark cluster. In this instance, the Subject Matter Experts (SMEs) mitigated the impact by manually restarting the affected machines one at a time. Given the recurrence, the scope of the incident was expanded to evaluate all potentially vulnerable sites and eliminate the risk of recurrence. Despite these efforts, on December 9, 2024, the same software malfunction caused an outage in our Sydney data center at 07:25 UTC, which was mitigated by rebooting the nodes. To resolve the underlying issue and prevent further occurrences, we expedited a kernel and Network Interface Card (NIC) firmware upgrade, which was completed on December 12, 2024, at approximately 22:25 UTC.  This update eliminated the risk of recurrence across all identified sites.

The efforts to determine the root cause and additional long-term preventive actions are in progress.

This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

Posted Dec 18, 2024 - 19:55 UTC

Resolved
We haven’t observed any additional issues with the Block Storage service in the impacted data centers (Toronto, Frankfurt, Newark, Sydney, and Fremont), and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Dec 12, 2024 - 23:08 UTC
Update
Our team identified an additional impact to Block Storage in the US-West (Fremont) data center with an initial, brief period of impact around 14:43 UTC and another period of impact between approximately 15:40 and 15:58 UTC on December 10, 2024. During this time, users may have experienced connection timeouts and errors with this service. This impact was introduced by maintenance being performed across clusters to prevent this impact going forward. We have been able to resolve the issue and continue to monitor to ensure that the service remains stable.
Posted Dec 11, 2024 - 16:16 UTC
Monitoring
At this time we have been able to correct the issues affecting the Block Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Posted Dec 09, 2024 - 08:47 UTC
Investigating
Our team identified an issue affecting the Block Storage service in Sydney, Australia, data center, on December 9, 2024, at 7:25 UTC. During this time, users may experience connection timeouts and errors with this service. We will share additional updates as we have more information.
Posted Dec 09, 2024 - 08:33 UTC
Update
Our team identified an issue affecting the Block Storage service in Sydney, Australia, data center, on December 9, 2024, at 7:25 UTC. During this time, users may experience connection timeouts and errors with this service. We will share additional updates as we have more information.
Posted Dec 09, 2024 - 08:30 UTC
Update
We have identified a recurrence of the issue.
Posted Dec 09, 2024 - 08:24 UTC
Update
Our team identified an issue affecting the Block Storage service in our Newark, New Jersey, data center, on December 5, 2024, between 06:40 and 07:49 UTC. At this time we have been able to correct the issues affecting the service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Posted Dec 05, 2024 - 09:37 UTC
Monitoring
At this time we have been able to correct the issues affecting the Block Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Posted Dec 03, 2024 - 11:52 UTC
Investigating
Our team is investigating a recurrence of the issue affecting the Block Storage service in our Toronto and Frankfurt data centers. During this time, users may experience connection timeouts and errors with this service. We will share additional updates as we have more information.
Posted Dec 03, 2024 - 11:32 UTC
Monitoring
At this time we have been able to correct the issues affecting the Block Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Posted Dec 03, 2024 - 10:46 UTC
Update
Our team is investigating an issue affecting the Block Storage service in our Toronto and Frankfurt data centers. During this time, users may experience connection timeouts and errors with this service. We will share additional updates as we have more information.
Posted Dec 03, 2024 - 09:52 UTC
Investigating
Our team is investigating an issue affecting the Block Storage service in our Toronto data center. During this time, users may experience connection timeouts and errors with this service. We will share additional updates as we have more information.
Posted Dec 03, 2024 - 09:41 UTC
This incident affected: Block Storage (US-East (Newark) Block Storage, US-West (Fremont) Block Storage, CA-Central (Toronto) Block Storage, EU-Central (Frankfurt) Block Storage, AP-Southeast (Sydney) Block Storage).