On January 17, at 08:25 UTC, our administrators received notifications regarding service degradation with regard to the Block Storage service in our US-East (Newark) data center. At 08:38 UTC, Compute Support received several customer reports of connectivity issues with Block Storage volumes in Newark.
Our administrator’s investigations revealed that a software bug from an upstream provider was the underlying cause of the Block Storage service degradation, necessitating a manual restart performed by an administrator.
At 10:04 UTC, our administrators successfully addressed the customer impact by restarting the process responsible for the service degradation. By 10:10 UTC, customers confirmed the resolution of their issues, and services were restored to their normal functioning state.
We are continuing to investigate the software error from the upstream provider that caused the slowdown. In the interim, a refined alerting system has been implemented for earlier warning of Block Storage service issues caused by this bug until a more permanent fix has been implemented.