On September 6th, at 9:45 UTC during a routine upgrade of Object Storage systems in our Frankfurt Datacenter. The service encountered an issue resulting in a loss of connectivity to customer buckets.
We immediately started our incident response procedures to restore customer connectivity. This in turn began to cause issues with I/O resulting in restored but spotty connectivity for some users. On September 7th, 2023, we performed a second maintenance to the service which again caused issues for customer buckets but this was resolved once maintenance was complete.
On September 18th, we were alerted to new issues, which caused three short spurts of intermittent connectivity for a subset of customers.
These were observed from 16:21 UTC - 16:27 UTC, 17:30 UTC - 17:30 UTC (less than a minute), and 17:39 UTC - 17:48 UTC. This was mitigated by 17:48 UTC and we continued to monitor for any additional impact during recovery until the service issue was resolved.
In addition to the short-term changes we have implemented over the past few months, we are also prioritizing long-term stability improvements of our Object Storage service within the Frankfurt Datacenter to match the rest of our fleet.