On 20 January, 2024, our administrators became aware of a software update which affected backups between 19:00 UTC on 19 January, 2024 and 3:00 UTC on 20 January, 2024. Our administrators identified that affected backups experienced data loss, as files greater than 500MB had been purged permanently from backups and snapshots made during this timeframe due to a bug in the recent software update.Customers were advised that any backup restores should be made to a new Linode instance to avoid overwriting existing disks causing data loss. At 3:00 UTC, the problematic software update was rolled back, and our administrators began implementing measures to have new backups taken to replace affected backups.
At approximately 22:00 UTC on 20 January, 2024, our administrators determined that the issue was affecting all backups, not only backups made during the affected timeframe. Affected backups were made unavailable to customers, and mitigation instructions were provided to customers who had suffered data loss due to restores that had been made during this time. After rolling back the modified update, internal testing verified the Backup Service was functioning appropriately and no longer risking data loss.
On 22 January, 2024 at approximately 19:55 UTC, it was discovered that a handful of customers were unable to complete a backup. By 3:52 UTC on 23 January 2024, a fix was implemented to resolve this.
Our administrators will continue to refine the procedures associated with the Backup Service. This will involve improvements to the alerting system and rigorous code evaluations during the end to end testing phase before the deployment of code updates.These initiatives will strengthen the resilience of the Backup Service.
Our team has deployed a fix to address any affected backup to address the ability to create new backups. They expect this fix to take up to 48 hours to address all affected backups. We will update this posting when the fix has been implemented.
As a matter of caution, we continue to recommend any restores needed from the Backup Service be deployed to a new compute instance. Should the restore be successful, you can then transfer the IP address between the old and new compute instance. This will alleviate any DNS changes needed and the only downtime should be a result of rebooting the instance.
If you have already used an affected backup to overwrite your instance’s disks, we recommend utilizing the Recovery Image that was created when the compute instance’s disks were deleted. You can deploy an Image to an existing compute instance to recover from any files that were not included in your restore but were present on the instance at the time the disks were deleted.
In the event that there is data from an affected backup that a customer requires, please open a Support ticket for assistance. We can make these affected backups available to you for the next fourteen days.