At approximately 09:43 UTC on January 31, 2024, a customer reported encountering difficulties when creating Block Storage volumes in Singapore. Subsequent investigations by Compute Support confirmed the issue on our end and persisted through testing, prompting the initiation of the incident process.
Further examination revealed communication disruptions between the backend API and the Dallas region, indicating a broader impact on other customers. While Block Storage volumes could still be created, communication via the API was impeded, resulting in apparent failed volume deployments for customers.
By 18:19 UTC on January 31, 2024, we successfully resolved the backend API communication issue, restoring connectivity between Singapore and Dallas. As a result, customers regained the ability to seamlessly create Block Storage volumes in Singapore.
After resolving this incident, our team has implemented controls to ensure greater network redundancy. We are also looking to improve monitoring alerts for situations where connectivity becomes impeded between data centers and services that rely on each other.
We remain dedicated to taking all necessary measures to ensure the optimal performance and reliability of our Block Storage offering.