Connectivity loss in Atlanta
Incident
Report for Linode
Postmortem
Here is an excerpt from the postmortem received from our upstream provider:
At 11:30 AM EST we began to monitor high CPU and latency on the core routers providing the internet access. Network engineers were on the devices and began looking for sources of CPU load. It was determined that multiple IPs were attacking a customer with TCP SYN packets. Once the attacking IPs were determined a filter was applied and traffic began to return to normal.
We are investigating the type of attack and looking to implement filters and rate limiters on the edge network devices to prevent this type of attack resulting in impact to other customers. We are also working to implement better traffic analysis to prevent future attacks from creating such wide spread issues for other customers.
We apologize for any inconvenience that this issue has caused.
Alex Forster
Network Engineer, Linode
Resolved
We have seen stabilized latency, loss, and jitter for approximately 40 minutes from all of our distributed monitoring points, and our upstream provider has indicated that the attack should be largely mitigated at this point.
Update
We have received notice that our upstream provider is currently absorbing an extremely large DDoS attack. Because of the highly specific measures used to filter bad traffic, there may be further intermittent connectivity issues as the attackers adapt their methods to circumvent the filtering. We will update this post when the attack has subsided.
Update
While the frequency of intermittent packet loss has lessened, the issue remains ongoing. We have no new information from our upstream at this time.
Update
The problem has been confirmed to be upstream from our network. We have engaged the Atlanta on-site engineering staff and are expecting more information shortly.
Monitoring
Our monitoring systems are currently indicating that there are partial internet outages in the Atlanta datacenter starting at approximately 11:15am EDT. We are currently working with our upstream provider to find out more information.