Connectivity Issues - Atlanta
Incident
Report for Linode
Resolved
We have seen stability in Atlanta for enough time to consider this outage resolved. We apologize for the immense disruption to customers during this two-day outage, and we will be following up in the next week with a full post-mortem and plans for the future.
Monitoring
Connectivity has been restored to the Atlanta datacenter. Users may encounter intermittent connectivity issues as we continue to monitor for additional attacks.
Identified
We are currently working on residual issues from the ongoing DoS attacks that have caused ongoing connectivity problems on a subset of our hosts in Atlanta. Thank you very much for your patience.
Monitoring
We believe that we have closed all of the attack vectors that can lead to a DDoS taking down the entire Atlanta datacenter. We are monitoring to confirm this.
Update
The Atlanta data center is still experiencing network connectivity issues due to the DDoS attack. Our Network Engineers have been working throughout the night to restore connectivity.
Update
We are still in the process of mitigating the DoS attack in our Atlanta data center. Our networking team is working with our upstream provider to restore normal connectivity to the data center.
Update
Our Networking team is still actively engaged in restoring network connectivity. We'll continue to keep this status post updated with the latest information.
Update
There has been some progress in restoring network connectivity to our Atlanta location, but our engineers are still hard at work to fully restore functionality to this datacenter. Thank you very much for your patience.
Update
We're still actively working with our upstream provider to get the transit link hardened and getting connectivity restored. We will continue to keep providing updates until the issue is resolved.
Update
The process of hardening our transit link and restoring connectivity in Atlanta is ongoing. We'll continue to provide updates as we work to resolve this issue.
Update
Our networking team is still working to harden our dedicated transit link in Atlanta against further attacks. Once this process has been completed, connectivity should be restored. We'll continue to provide updates as things progress.
Update
Our Network Engineers are still working with the transit provider mentioned below to apply the necessary DDoS mitigation hardening to facilitate restored connectivity to our infrastructure in Atlanta. We will keep you updated as these efforts continue.
Update
Our network operations and systems teams have been working non-stop for the last ~36 hours toward a resolution of the Atlanta outage. We have acquired a dedicated transit link that is now directly connected to the Linode network, and we are waiting for this transit provider to apply DDoS mitigation hardening, after which we believe that Atlanta should be restored to full service. We will keep you updated as things progress.
Alex Forster
Network Engineer, Linode
Alex Forster
Network Engineer, Linode
Update
Our Network Engineers are still actively working with our upstream provider in Atlanta to restore normal network connectivity.
Update
Our Network Engineers are still working with our upstream provider on mitigating this latest DoS attack. We'll post an update here when we believe connectivity has normalized.
Update
Around two hours after bringing things back online, attacks on Atlanta have started again which are affecting the entire datacenter. At this time we are being taken back offline to attempt further mitigation.
Update
We have received word that all of our Atlanta IP space is now being announced. We are still fine tuning our geographic blocking, so people in the affected regions may see intermittent loss of connectivity over the next hour or two.
Update
Zayo has begun bringing our IP ranges online.
For the short term, we will be using BGP communities to attempt to block Asia Pacific, the Middle East, South America, and others, hopefully leaving us only with traffic from North America and Western Europe. Blocking geographic regions this way is the only way to make sure that large botnets won't be able to launch further attacks.
We hope to have all of our Atlanta IP space advertised in the next several hours.
Alex Forster
Network Engineer, Linode
For the short term, we will be using BGP communities to attempt to block Asia Pacific, the Middle East, South America, and others, hopefully leaving us only with traffic from North America and Western Europe. Blocking geographic regions this way is the only way to make sure that large botnets won't be able to launch further attacks.
We hope to have all of our Atlanta IP space advertised in the next several hours.
Alex Forster
Network Engineer, Linode
Update
Our upstream provider has informed us that they plan to restore network connectivity to our systems gradually over an undisclosed period of time. There is no ETA at the moment, but our engineers and leadership team members are working closely with theirs to accelerate this process as much as possible. We will provide another update as soon as more information becomes available.
Update
Our upstream provider reports that they've made some progress on mitigating the numerous DoS on their infrastructure, but due to the varying nature of the target, scope, and size of the attacks, network connectivity has not yet been restored. We are working closely with their team to remain informed on their progress, and will update this page as soon as there are any further developments. Once again, we sincerely thank you all for your patience.
Update
The DoS attack in Atlanta is ongoing, and is still causing major network disruptions in our Atlanta location. We are continuing to work with our upstream provider to restore network functionality as quickly as possible. Thank you again for your patience.
Update
The attack in Atlanta is still ongoing and causing significant connectivity issues. We're closely working with our upstream provider to ensure that connectivity is restored as soon as possible. We apologize for any inconvenience these issues are causing and we thank you for your patience.
Identified
We're currently engaged with our upstream provider in mitigating a large, distributed DoS attack targeting our infrastructure in Atlanta. Users may experience packet loss and issues with connectivity as we work to resolve this. We thank you for your patience.