Between October 21st and 28th a change was deployed across Paris, Frankfurt, Mumbai, Atlanta, Newark, Chicago, Osaka, Seattle, and Singapore Data Centers that inadvertently caused deployment failures for customers in these regions.
Our team identified the cause of the issue to be a new Disk Scrubber version introduced on October 21st in Paris and on October 27th in the other 10 Data Centers. The Disk Scrubber cleans our host disk before a new Virtual Machine's storage is provisioned. The issue was not noticed until October 27th.
Due to a race condition, volumes were not always successfully cleaned causing the process to fail and preventing new VMs from being provisioned on the host.
In order to mitigate this issue, we switched the faulty Disk Scrubber to another build and manually cleaned the affected hosts.
To avoid this issue in the future, the faulty Disk Scrubber will be fixed and our scrub failure alerting improved. We will re-deploy the Disk Scrubber after rigorous testing.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.