Starting at approximately 15:20 UTC on January 16th, 2025, Akamai received internal alerts for our Mumbai data center. As is standard procedure for the specific alerts seen, a reboot of the associated piece of network hardware was initiated.
During this process, an unforeseen issue was encountered, and the aforementioned piece of hardware began to fail forwarding packets through the device. This caused affected hosts to be unable to process host jobs as well as a complete loss of observability during the period of impact.
The issue was mitigated at approximately 17:40 UTC after removing the faulty network hardware piece from the rotation.
In order to prevent this problem from reoccurring, Akamai has replaced the faulty piece of network hardware. We will also use a new and more stable Operating System version in the network devices involved and update the alerting system so that it can better identify when such issues occur in host jobs.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.