At 04:00 UTC on 10/05/2023, we began emergency maintenance to expand the capacity of our Object Storage service within the Newark Datacenter.
At 07:30 UTC, during the final maintenance steps, a networking race condition began to impact performance and connectivity for customers utilizing the Object Storage and LKE services.
Once identified, our administrators began working to restore the services. Our network team implemented a fix for the impacted networking at 12:55 UTC. After additional monitoring of the services following the changes made, we determined the issue to be resolved.
As a result of this incident, our team is working to implement additional monitoring of our Object Storage infrastructure, and will be implementing additional proactive steps during future maintenance of our Object Storage service.