Service Issue - Block Storage US-ORD (Chicago) US-IAD (Washington, D.C.) and Los Angeles (LAX3)
Postmortem
Beginning at ~21:57 UTC on January 9, 2025, customers began to experience delays and an inability to access Block Storage resources in the US-ORD region. Shortly after, at ~22:07 UTC, delays and an inability to access Block Storage resources was also observed in the US-IAD region. This impact was mitigated at ~22:20 UTC. We’ve identified that this degradation was due to an unanticipated effect of a service re-configuration process which required the restart of a Block Storage service component. We are modifying our service re-configuration process for this subset of services to reload updated configurations instead of performing a restart, removing the potential for a recurrence of this issue.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.
Resolved
Monitoring
- Chicago (ORD2): 21:57 UTC to 22:18 UTC
- Washington, D.C. (IAD3): 22:07 UTC to 22:20 UTC
- Los Angeles (LAX3): 21:32 UTC to 21:45 UTC
During this time, users may have experienced slow or failed storage operations for their Block Storage Clients.
We have been able to correct the issues and the service has resumed normal operations. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.