Connectivity Issues in Atlanta
Incident Report for Linode
Postmortem

At approximately 5pm EDT on Saturday September 3rd our monitoring systems alerted us to a DDoS attack towards many destinations in our Atlanta data center. The support team immediately began to mitigate the attack while the on call engineer was being paged. As the on call engineer was taking lead in mitigating the attack it became apparent that once the attack was mitigated another attack to many other destinations would start. The on call engineer then sent out a request for other members of the engineering team to assist in the mitigation of the attacks. During this manual cat and mouse mitigation phase of the attack, customers may have seen degraded connectivity to any destination IP that was specifically under attack. Other customers in the DC may have also been impacted by some packet loss and latency resulting from collateral damage due to the attack. Linode engineers then wrote and deployed software to automate the mitigation of these randomized ongoing attacks, once that software was in place the collateral impact to customers lessened . A request was also put out to our upstream to help police the attack and that request was denied. The DDoS attacks continued throughout the night of September 3rd and into the morning of September 4th, customers of the Atlanta datacenter may have seen some packet loss and latency during this time but for the most part the automated mitigation was working. At approximately 8am September 4th the volume of the attacks dramatically increased and was causing widespread latency and packet loss for most of the customers in our Atlanta data center. Another call was placed to the upstream provider and this time our engineers were able to compel the upstream to assist with the placement of policers on our upstream ports. The policers were crafted by our engineers from the data that was collected during the attack and once the policers were implemented by the upstream all network disruption subsided.

We are in the midst of many network upgrades in Atlanta that will allow us to not feel the impact of DDoS attacks like this one in the future. Our engineers were onsite in Atlanta lighting dark fiber two weeks ago, and we expect to have our greatly increased capacity online by the end of this month. This round of DDoS attacks was not limited to just Atlanta, our other locations also saw the exact same DDoS attack vector on their IP ranges. Fortunately our other locations are far along in regards to the network upgrades so negative impact to customer traffic was not felt.

Posted over 2 years ago. Sep 09, 2016 - 21:43 UTC

Resolved
This incident has been resolved.
Posted over 2 years ago. Sep 05, 2016 - 21:34 UTC
Update
Some users may be experiencing delayed DNS resolution in Atlanta due to filtering that has been put in place. We are currently working to reconfigure where necessary and restore normal DNS resolution.
Posted over 2 years ago. Sep 04, 2016 - 20:13 UTC
Monitoring
We have implemented upstream mitigation at this time and Atlanta IP service has returned to normal. We will continue closely watching the situation for further adaptation to these measures.
Posted over 2 years ago. Sep 04, 2016 - 16:45 UTC
Identified
The attacks in Atlanta have returned and are significantly impacting connectivity. We are working toward mitigation.
Posted over 2 years ago. Sep 04, 2016 - 13:54 UTC
Update
Connectivity throughout Atlanta has begun stabilizing and we are currently monitoring for any further issues.
Posted over 2 years ago. Sep 04, 2016 - 13:34 UTC
Monitoring
The attacks have not stopped yet and our Network Engineers are still actively mitigating the DDoS attacks. However, the status of the network has improved significantly, and customers who are not in the affected ranges should now be stable. If you are still experiencing connectivity issues or have a Linode in one of the affected ranges, please reach out to our Customer Support Team for assistance in obtaining an additional IP address as a temporary workaround.
Posted over 2 years ago. Sep 04, 2016 - 07:30 UTC
Update
While we are continuing to receive attacks, our overall network is beginning to stabilize. Customers who are not in the affected ranges may still see small spikes in latency or short periods of packet loss.
Posted over 2 years ago. Sep 04, 2016 - 04:52 UTC
Update
We are continuing to field frequent attacks, but our mitigation response time has improved significantly in the last two hours, to the point where customers which are not in the affected ranges should only be noticing minimal latency spikes or brief periods of packet loss.
Posted over 2 years ago. Sep 04, 2016 - 03:10 UTC
Update
Our Network Engineers are still hard at work mitigating the DDoS attack. We currently do not have an ETA on when full connectivity will be restored, but rest assured that resolving this is our top priority.
Posted over 2 years ago. Sep 04, 2016 - 02:06 UTC
Update
We have been experiencing a catastrophic DDoS attack which is being spread across hundreds of different IP addresses in rapid succession, making mitigation extremely difficult. We are currently working with our upstreams to implement more complete mitigation.
Posted over 2 years ago. Sep 04, 2016 - 00:45 UTC
Identified
We've identified a distributed DoS attack as the cause of impacted connectivity in Atlanta and we're working to mitigate it. Further updates will be provided as they become available.
Posted over 2 years ago. Sep 03, 2016 - 21:14 UTC
This incident affected: Regions (US-Southeast (Atlanta)).