On March 12th at 22:43 UTC, we began receiving customer reports regarding DNS resolution issues in the form of time outs affecting multiple domains in northern Italy and Switzerland. These issues prevented users from accessing various websites and disrupted normal operations for the impacted domains.

Following an initial assessment by Akamai, it was determined that the issue was limited to domains utilizing Linode authoritative nameservers and the Akamai Shield NS53 product. Specifically, the affected nameservers were ns1.linode.com, ns2.linode.com, ns3.linode.com, ns4.linode.com, and ns5.linode.com. 

Further investigation revealed that DNS requests to Linode origin nameservers were timing out when routed through the Shield NS53 data center in Rome, which was recently brought online and started receiving traffic on March 11th at 13:32 UTC. Log analysis from this region indicated that Linode’s authoritative nameservers were rejecting requests originating from the Rome Shield NS53 infrastructure. It was subsequently discovered that the backend IPs of the Shield NS53 region in Rome were not present in Linode’s authoritative server ACLs, leading to the service disruption.

To resolve the issue, the missing backend IPs were added to Linode’s authoritative nameserver ACLs on March 13th at approximately 22:30 UTC, restoring normal DNS resolution and mitigating the incident.

Also, to prevent the issue from happening in the future Akamai will create additional alerting and review current procedures to check connectivity from new Shield NS53 datacenters to Linode’s authoritative nameservers and enhance a process for adding new Shield NS53 backend IPs to the Linode origin nameservers’ ACLs.