At approximately 5pm EDT on Saturday September 3rd our monitoring systems alerted us to a DDoS attack towards many destinations in our Atlanta data center. The support team immediately began to mitigate the attack while the on call engineer was being paged. As the on call engineer was taking lead in mitigating the attack it became apparent that once the attack was mitigated another attack to many other destinations would start. The on call engineer then sent out a request for other members of the engineering team to assist in the mitigation of the attacks. During this manual cat and mouse mitigation phase of the attack, customers may have seen degraded connectivity to any destination IP that was specifically under attack. Other customers in the DC may have also been impacted by some packet loss and latency resulting from collateral damage due to the attack. Linode engineers then wrote and deployed software to automate the mitigation of these randomized ongoing attacks, once that software was in place the collateral impact to customers lessened . A request was also put out to our upstream to help police the attack and that request was denied. The DDoS attacks continued throughout the night of September 3rd and into the morning of September 4th, customers of the Atlanta datacenter may have seen some packet loss and latency during this time but for the most part the automated mitigation was working. At approximately 8am September 4th the volume of the attacks dramatically increased and was causing widespread latency and packet loss for most of the customers in our Atlanta data center. Another call was placed to the upstream provider and this time our engineers were able to compel the upstream to assist with the placement of policers on our upstream ports. The policers were crafted by our engineers from the data that was collected during the attack and once the policers were implemented by the upstream all network disruption subsided.
We are in the midst of many network upgrades in Atlanta that will allow us to not feel the impact of DDoS attacks like this one in the future. Our engineers were onsite in Atlanta lighting dark fiber two weeks ago, and we expect to have our greatly increased capacity online by the end of this month. This round of DDoS attacks was not limited to just Atlanta, our other locations also saw the exact same DDoS attack vector on their IP ranges. Fortunately our other locations are far along in regards to the network upgrades so negative impact to customer traffic was not felt.