Connectivity Issues – Multiple Data Centers
Incident
Report for Linode
Postmortem
At approximately 05:21:00 UTC on May 31, 2019, we were alerted that our upstream provider had experienced a fiber cut that impacted connectivity to our Toronto data center. At around 12:21:00 UTC, in an attempt to lessen the impact of the outage, a change to our route-policy was made which produced high CPU events on our edge routers in Dallas, Fremont, London, Newark, Singapore, and Tokyo 2 data centers.
Although we were able to revert the change immediately, the event did cause brief routing instability across our global backbone. The result was a momentary degradation of routing to the public internet.
We apologize for the connectivity issues experienced as a result of this change.
Resolved
Connectivity to our Dallas, Fremont, London, Newark, Singapore, and Tokyo 2 data centers have remained stable, so we are considering this incident resolved. If you are still experiencing issues, please contact our Customer Support Team for assistance.
Monitoring
Our Dallas, Fremont, London, Newark, Singapore, and Tokyo 2 data centers experienced brief connectivity issues this morning that lasted from approximately 8:21 – 8:25 EDT, which have since been resolved. During this issue, some Linode services in the above-referenced data centers may have been inaccessible. We’re monitoring this issue to ensure stability, and we’ll provide updates here as necessary. If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Once this has been resolved, we’ll provide a post-mortem here.
Once this has been resolved, we’ll provide a post-mortem here.
This incident affected: Regions (US-East (Newark), US-Central (Dallas), US-West (Fremont), EU-West (London), AP-South (Singapore), AP-Northeast-2 (Tokyo 2)).