On June 20th, 2025, at approximately 10:40 UTC, we observed a sudden increase in alerts affecting the Los Angeles (US-LAX) Block Storage service. These alerts included service unavailability, elevated latency, increased HTTP 429 errors, and spikes in retry attempts.
Upon investigation, we identified a storage drive that was not performing as expected. Although it did not show any obvious signs of failure through standard checks, it was handling data inefficiently and slowing down system operations. This underperformance contributed to the issues customers observed.
We disabled the affected drive from service, and immediately observed stabilization across the system. No further slow operations were detected within the cluster, confirming that the issue had been mitigated.
The incident was fully mitigated by approximately 14:34 UTC on June 20th, 2025.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.