On February 4, 2025, at 18:30 UTC, customers started experiencing latency, intermittent connection timeouts, and errors when attempting to access the Object Storage service in the Washington, Chicago, Seattle, Paris, and Amsterdam data centers. Investigation revealed that the issue was being caused by a misconfiguration in an ongoing firewall change. To mitigate the impact, we rolled back the change at 19:07 UTC, and it fully propagated at 19:25 UTC.
During the monitoring period, the Chicago and Washington data centers were still experiencing problems, so additional actions were taken to address the ongoing impact. We mitigated the issue for Chicago at 20:16 UTC and for Washington at 20:29 UTC. After monitoring our services for an extended period, we confirmed that the incident was fully resolved. In addition, the firewall change that initiated this situation was successfully completed at 13:20 UTC on February 11, 2025.
This summary provides an overview of our current understanding of the incident, given the information available. Our investigation is ongoing, and any information herein is subject to change.