At 10:26 UTC on January 16th, our team was alerted to performance degradation for TCP traffic. Upon investigation of these alerts, a routing leak of Akamai prefixes was found to be misconfigured by a third-party Internet Service Provider (ISP). The route advertisements were accepted by Tier 1 and Tier 2 ISPs, which contributed to the packet loss and increased latency for traffic going through the ISPs that had accepted the misconfigured routing prefixes. By 11:25 UTC, our team used an established run book workflow to deactivate Border Gateway Protocol (BGP) sessions with the origin ISP.
As this issue was caused by the actions of a third-party ISP, actions to prevent routing leaks like this are outside of our control. Instead, we will continue to rely on alerting for situations like this and the actions already established to mitigate such an issue.
Affected ISPs include:
MSKIX-LED
SDN-DUS
DECIX-FRA
NETNOD-STO2
SECH-PAR
MSKIX-MOW
M9-SVO
M9-SVO2
FABRIC-LON
SDN-HAM
NEWTELCO-KBP
LINX-LON12
EQUINIX-HKG3
SDN-AMS
PCCW-HKG4
INTX-STO
DECIX-FRA11
DATA4-PAR2
SECH-STO
FABRIC-PAR3
DESTINY-KRK
FABRIC-STO2
FABRIC-AMS
SECH-FRA
FABRIC-FRA3
CECOLO-PRG2
FABRIC-AMS2