Linode job Issue - ALL: 1 (INVESTIGATING)
Incident
Report for Linode
Postmortem
At 20:04 UTC on October 12, 2023, hosts across multiple data centers alerted for a failing component in their software stack. After initial investigation, this failure was an unexpected side effect of routine maintenance work that had just been completed.
We activated our incident response procedures at 20:12 UTC, with an action plan for remediation performed at 20:21 UTC. Improvements in behavior became evident by 20:25 UTC. Additional remediation actions commenced at 20:55 UTC and were completed by 22:41 UTC for a full resolution.
To prevent this failure from occurring again, we have identified how the component in question failed and are undertaking steps to prevent this type of maintenance from causing this failure in the future.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are aware of an issue across all of our data centers which is affecting new linodes being created, booted, rebooted, and shut down. we are working to assess the impact and resolution, and will update with information as it becomes available
This incident affected: Regions (US-East (Newark), US-Central (Dallas), US-West (Fremont), US-Southeast (Atlanta), US-IAD (Washington), US-ORD (Chicago), CA-Central (Toronto), EU-West (London), EU-Central (Frankfurt), FR-PAR (Paris), AP-South (Singapore), AP-Northeast-2 (Tokyo 2), AP-West (Mumbai), AP-Southeast (Sydney), SE-STO (Stockholm), US-SEA (Seattle), IT-MIL (Milan), JP-OSA (Osaka), IN-MAA (Chennai), ID-CGK (Jakarta), BR-GRU (Sao Paulo), NL-AMS (Amsterdam)).