On March 20, 2024 at approximately 16:00 UTC, our administrators encountered 502 errors when attempting to connect to the Linode API. Investigation into the issue began immediately. During the investigation, a quick fix was implemented in order to alleviate the 502 error, which was partially successful. Work continued in order to find and deploy a permanent solution. On March 21, 2024 at around 18:44 UTC, our administrators were able to identify the issue to be an update to the Linode Cloud Controller Manager that was then picked up by other services. At 19:58 UTC, this update was successfully rolled back. At 21:07 UTC on March 21 2024, the issue was considered mitigated by our administrators, and the incident was resolved after a short period of monitoring.
We will continue to take steps to improve the resilience of our API service, including improving our alerting systems for API related issues in order to catch these issues as they happen sooner.