Connectivity Issues - Frankfurt
Incident
Report for Linode
Resolved
Being that we have not experienced additional connectivity issues affecting our Frankfurt data center, this matter is now resolved.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Monitoring
The connectivity issues from the scheduled network maintenance in our Frankfurt data center has been corrected. Our team is continuing to monitor the situation to ensure connectivity remains stable.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Investigating
We are aware of connectivity issues affecting Linodes in our Frankfurt data center as a result of the scheduled maintenance and are currently investigating. We will continue to provide additional updates as this incident develops.
Monitoring
After many hours of troubleshooting with Cisco, it appears we have hit hardware and software failures on both internet facing edge routers simultaneously. The "A" side router had a module degrade silently, meaning that the module did not fully fail and caused strange forwarding behavior without notifying the operating system of the failure. Once the "A" side router was reloaded, the module failure became apparent during the boot process. Both redundant routers also experienced the same undocumented forwarding table bug that caused the blackholing of some customer traffic. At this time we have been able to normalize connectivity. A separate maintenance will be required to fully resolve this incident, at which point a full post mortem will be made available.
Update
Our Network Operations team is continuing to work with Cisco on a resolution to these issues. We'll continue to provide updates as the situation develops.
Update
Our engineers are still troubleshooting with Cisco. Cisco believes a software bug is causing the reachability problems we are seeing. A workaround is currently being developed. More updates to follow.
Update
Both of our Frankfurt internet edge routers are experiencing issues with their forwarding tables. This is causing sporadic blackholing of customer traffic. Our engineers have reached out to the router vendor and are engaged with their engineering team. More updates will be provided as information become available.
Update
We are currently still investigating the connectivity issues affecting Linodes in our Frankfurt data center. We will continue to provide additional updates as this incident develops.
Investigating
We are aware of connectivity issues affecting Linodes in our Frankfurt data center and are currently investigating. We will continue to provide additional updates as this incident develops.
This incident affected: Regions (EU-Central (Frankfurt)).