At 18:24 UTC on November 7, 2023, a backend change was made to the Compute database records. Internal networking equipment in Toronto took up this change at 19:17 UTC. This change contained a number of improperly formatted records, which caused the software on this networking equipment to crash. This software crash is what caused the impact of this incident.
At 19:22 UTC, alerts for this networking equipment software started arriving. Akamai's Compute team started its initial investigation at 19:23 UTC, finding the presence of the bad record at 19:33 UTC. An incident was declared at 19:34 UTC, with an action plan formulated by 19:44 UTC. Implementation of this action plan began at 19:47 UTC and completed by 19:53 UTC.
This action plan resolved the immediate impact, but a complete fix required correcting the improperly formatted records. The team which manages those records was engaged at 20:07 UTC, beginning work on correcting the records at 20:12 UTC and completing work by 20:20 UTC. A test of the validity of the records successfully completed at 20:24 UTC, and the impacted systems had temporary mitigations for preventing recurrences removed at 20:33 UTC, fully resolving the impact of the issue.
To prevent this issue from occurring in the future, Akamai Compute is implementing improvements to data validation to prevent the insertion of improperly formatted records. Additionally, safeguards will be put in place to prevent the network equipment from accepting invalid records.