Service Issue - Reboot Watchdog
Incident
Report for Linode
Postmortem
At 7:30 UTC, on October 4th, 2023, our team discovered that the Lassie watchdog service was unable to successfully reboot Linodes across a small subset of hosts.
Our administrators were able to identify a recent code change that impacted the behavior of the Lassie Watchdog service. Once identified, they reverted the code change, restoring expected behavior at 12:35 UTC, October 4th, 2023.
As a result of this incident, we are improving end to end testing in regards to the Lassie Watchdog service.
Resolved
This incident has been resolved.
Monitoring
Our administrators have found and mitigated an issue where the Shutdown Watchdog was not rebooting Linodes that were shutdown/rebooted via the command line, or which shutdown unexpectedly.
Linodes rebooted via the Linode API/CLI/Cloud-Manager are not affected.
Linodes rebooted via the Linode API/CLI/Cloud-Manager are not affected.
This incident affected: Regions (US-East (Newark), US-West (Fremont), US-Southeast (Atlanta), US-ORD (Chicago), CA-Central (Toronto), EU-West (London), EU-Central (Frankfurt), FR-PAR (Paris), AP-South (Singapore), AP-Northeast-2 (Tokyo 2), AP-West (Mumbai), AP-Southeast (Sydney), SE-STO (Stockholm), US-SEA (Seattle), IT-MIL (Milan)).