Connectivity Issue -- Platform Operations, Cloud Manager, and API
Incident
Report for Linode
Postmortem
At ~23:30 UTC December 21, 2023, a database became overloaded due to a manual query for an internal project. This caused issues with the API and Cloud Manager and affected jobs issued by customers. Our incident response procedures were formally launched at 23:34 UTC. After an initial investigation, the team took action to return the database to normalized levels. Jobs issued by customers resumed and worked as expected around 00:09 UTC December 22, 2023.
We have taken steps to increase the database performance and prevent this type of issue from recurring in the future. We are also continuing to make improvements to our technical procedures which will further reinforce reliability of this database moving forward.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we are monitoring the results. We will continue to monitor to ensure the stability of all systems before declaring the issue resolved.
If you continue to experience any further problems, please submit a support ticket for assistance.
If you continue to experience any further problems, please submit a support ticket for assistance.
Identified
Our team has identified an issue affecting platform-level operations affecting services in all data centers, as well as Cloud Manager.
During this time, users may experience issues when performing platform-level tasks such as creating services, deleting services, and changing the power state of services. Users may also experience connection timeouts when interacting with Cloud Manager and the API.
Our team has identified the issue causing this issue and is working quickly to implement a fix. We will update with more details as they become available.
These issues may prevent customers from opening tickets, so customers should call 855-454-6633 (U.S.) or +1-609-380-7100 (global) if they need direct assistance from our Support team.
We will share additional updates as we have more information.
During this time, users may experience issues when performing platform-level tasks such as creating services, deleting services, and changing the power state of services. Users may also experience connection timeouts when interacting with Cloud Manager and the API.
Our team has identified the issue causing this issue and is working quickly to implement a fix. We will update with more details as they become available.
These issues may prevent customers from opening tickets, so customers should call 855-454-6633 (U.S.) or +1-609-380-7100 (global) if they need direct assistance from our Support team.
We will share additional updates as we have more information.
This incident affected: Linode Manager and API.