A scheduled activity to shift the back-end API DNS configuration between DNS providers occurred on September 3rd at approximately 14:00 UTC. The impact began soon after at 14:06 UTC. Following the DNS update, the Block Storage team observed a loss of connectivity between internal services. As a result, customers attempting to create, modify, or delete block storage volumes found their operations stalled in the “creating”, “resizing”, or “deleting” states. This issue was mitigated at approximately 16:32 when the backlog of API requests was processed after rolling back the configuration update applied during this scheduled activity.
Akamai will review, update, and test a comprehensive change plan for the future application of this change to prevent this issue from occurring again. We apologize for the disruption and appreciate your patience and support as we work to improve our systems to prevent future occurrences
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.