Service Issue: Lassie
Incident
Report for Linode
Postmortem
On April 20th, 2022 an update to our hypervisors prevented Lassie reboots from being issued to customer Linodes. At around 00:46 UTC we began receiving reports of Linodes that were not being rebooted by our kernel watchdog service (Lassie). The issue was brought to the attention of our administrators and a fix was implemented at approximately 04:30 UTC.
The issue was initially resolved by rolling back the previous update across our fleet. Following the roll back, administrators identified and fixed the bug that caused this issue. We’ve now added checks to test for similar issues in future updates, and we have not observed any related problems following resolution of this incident.
Resolved
We haven’t observed any additional issues affecting the Lassie service, and now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Monitoring
At this time we have been able to correct the issue affecting the Lassie service. We will be monitoring this to ensure that it remains stable.
Identified
Our team has identified the issue affecting the Lassie service. We are working to implement a fix and will provide an update as soon as the solution is in place.
Investigating
We are currently investigating an issue with Lassie, our reboot helper. At this time if your Linode experiences an unexpected shutdown, it will need to be manually resolved within Cloud Manager or Linode CLI.
This incident affected: Regions (US-East (Newark), US-Central (Dallas), US-West (Fremont), US-Southeast (Atlanta), CA-Central (Toronto), EU-West (London), EU-Central (Frankfurt), AP-South (Singapore), AP-Northeast-2 (Tokyo 2), AP-West (Mumbai), AP-Southeast (Sydney)).