Between 22:04 UTC and 23:33 UTC Jul 20, 2023, there was a disruption in service to the Object Storage service in the Frankfurt Datacenter. This caused the service to initially become inaccessible for ~15 minutes, and then experience degraded performance for a remaining 1 hour and 15 minutes. 

This incident occurred during a standard upgrade to a redundant piece of the Object Storage infrastructure in the datacenter. The root cause of the incident was a recovery setting that had been changed from the default in the Datacenter and not changed back before the upgrade process began.

In response Linode invoked its incident response procedures, and at 22:48 UTC a Status Page was created. The issue was identified at 23:04 UTC.

Recovery was initiated by returning the recovery settings back to default, and adding the upgraded piece of Object Storage infrastructure back into the rotation. To prevent similar incidents, the upgrade procedures for the Object Storage service have been reviewed and changed. Additionally, our teams are investigating more precise and earlier alerting to ensure we are able to address Object Storage incidents sooner in the future.