Linode Status

Current Status
Service Issue - Managed Databases
Incident Report for Linode

At around 23:00 UTC on March 22, 2023, Linode’s Support team noticed a large number of alerts regarding Managed Database clusters and informed the appropriate teams about this issue. An investigation into the issue commenced, and Linode began its incident response procedures to coordinate the response.

By about 00:00 UTC on March 23, 2023, it was clear that a subsystem for Managed Databases had experienced a failure at 21:46 UTC on March 22, 2023, likely due to a general networking disruption shortly before this time. This subsystem does have an automatic recovery process, but this recovery process failed after 1 hour and 15 minutes at around 23:01 UTC on March 22, 2023, matching the timing of the alert flood noticed by Linode Support.

During this investigation, a manual recovery process had begun at 00:01 UTC on March 23, 2023 without an ETA. By 00:20 UTC, an ETA of about 3 hours was available. The recovery was actively monitored during this time and was confirmed to be completed at 03:38 UTC on March 23, 2023.

While this recovery of the subsystem was a necessary prerequisite for service restoration, the step of refreshing the Managed Database clusters themselves was also necessary. This action completed for all Managed Database clusters by 08:53 UTC on March 23, 2023, rendering them fully usable once again. After observing no further reports, the status page was resolved at 12:42 UTC.

To prevent this issue from occurring again, we are exploring ways to improve the resiliency of the subsystems whose failure to automatically recover led to this incident.

Posted Jun 27, 2023 - 20:17 UTC

This incident has been resolved. If you still continue to experience issues please open a Support ticket for assistance.
Posted Mar 23, 2023 - 12:42 UTC
We have resolved the cluster monitoring issue, however as checks are continuing to update you may still see your cluster display a failed status as the updates propagate. If you are encountering any issues connecting to your cluster please reach out to our support team for assistance.
Posted Mar 23, 2023 - 04:48 UTC
A fix has been implemented and we are monitoring the results.
Posted Mar 23, 2023 - 04:16 UTC
We are continuing to work on a fix for this issue.
Posted Mar 23, 2023 - 02:40 UTC
The issue has been identified and a fix is being implemented.
Posted Mar 23, 2023 - 01:16 UTC
Our team is investigating a service issue affecting the Managed Databases service. During this time, users may experience issues when attempting to connect to or provision these systems.

We will share additional updates as we have more information.
Posted Mar 23, 2023 - 00:11 UTC
This incident affected: Managed Databases.