On Tuesday, July 9, 2024 at approximately 1630 UTC, customers using Object Storage in our Singapore region began experiencing connection timeouts and service disruptions.
Prior to these connectivity issues, our engineering team had already identified a problem with a recent firewall configuration change which had been recently implemented around 1545 UTC. In short order, a new code change was pushed out to address the connectivity issues at 1627 UTC. Once this code change took effect across all underlying infrastructure, connectivity was restored around 1740 UTC. Some customers would continue to see latency, with these occurrences diminishing over the next couple of hours.
The firewall change was a part of general infrastructure updates for our Object Storage service in Singapore. To prevent a repeat of this problem moving forward, our team will be implementing more robust testing for code changes around firewall rules.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.