At approximately 15:00 UTC on July 5, 2023, we began to observe new database cluster deployments failing in our Washington, D.C. (IAD3), Chicago (ORD2), and Paris (PAR3) data centers. By 21:04 UTC, an issue in code was identified and work commenced to address it.
At 00:24 UTC, it was determined that work on the issue would need to pause and resume at the start of US Eastern business hours. At 18:55 UTC on July 6, during the next US Eastern business day, the tested code fix had been deployed and the issue was confirmed to be resolved after a brief monitoring period.
To prevent this incident from occurring in the future, we will be implementing closer integration between our teams to verify the continued proper functioning of services in new data centers. We will also be working to expedite the delivery of code changes for resolving any future incidents involving Managed Databases.