Linode Status

All Systems Operational

Linode Manager and API Operational

Hosted DNS Service Operational

London Operational

Newark Operational

Atlanta Operational

Dallas Operational

Fremont Operational

Tokyo 1 Operational

Tokyo 2 Operational

Singapore Operational

Frankfurt Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Linode Manager and API Maintenance: October 18, 2017 Oct 18, 13:00-15:00 UTC

A maintenance window has been scheduled for the Linode Manager and API on October 18th from 09:00 AM - 11:00 AM EDT (13:00 - 15:00 UTC). We expect this maintenance to take approximately 2 hours. During this maintenance, running Linodes will not be impacted but customers will not be able to perform tasks such as creating, removing, booting, or shutting down Linodes, or other tasks involving the Linode Manager or Linode API.

Our Support team will be available on our dedicated line at 855-454-6633 (Intl: +1 609-380-7100) for general questions, but will not be able to assist with issues related to the Linode Manager or API during this maintenance.
Posted on Oct 14, 14:20 UTC

Edge Router Upgrade - B Side - Tokyo 2 Datacenter Oct 23, 23:00 - Oct 24, 03:00 UTC

On Monday, October 23, at 23:00 UTC (Tuesday, October 24, 08:00 Tokyo local time), our engineers will be performing an edge router upgrade in our Tokyo 2 datacenter. We will be working exclusively on the "B" side of the network, and do not expect any downtime during this maintenance as we have full router redundancy. However, brief periods of packet loss or increased latency may be observed at times. We expect this maintenance to take 4 hours.
Posted on Oct 13, 14:38 UTC

Edge Router Upgrade - A Side - Tokyo 2 Datacenter Oct 24, 23:00 - Oct 25, 03:00 UTC

On Tuesday, October 24, at 23:00 UTC (Wednesday, October 25, 08:00 Tokyo local time), our engineers will be performing an edge router upgrade in our Tokyo 2 datacenter. We will be working exclusively on the “A” side of the network, and do not expect any downtime during this maintenance as we have full router redundancy. However, brief periods of packet loss or increased latency may be observed at times. We expect this maintenance to take 4 hours.
Posted on Oct 13, 14:45 UTC

Past Incidents

Oct 17, 2017

Connectivity Issues - Inter-DC connectivity between Dallas and our other datacenters

Resolved - Being that we have not experienced additional connectivity issues affecting inter-DC communication with our Dallas data center, this matter is now resolved.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Oct 17, 08:33 UTC

Monitoring - At this time we have been able to correct the connectivity issues between Dallas and our other facilities. We will be monitoring this issue to ensure connectivity remains stable.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Oct 17, 05:18 UTC

Identified - At this time we have identified the connectivity issues between Dallas and our other facilities. Our team is working as quickly as possible to have connectivity restored. We will provide additional updates as they develop.
Oct 17, 04:43 UTC

Investigating - We are aware of connectivity issues between Dallas and our other datacenters and are currently investigating. We will continue to provide additional updates as this incident develops.
Oct 17, 04:30 UTC

Oct 16, 2017

No incidents reported.

Oct 15, 2017

No incidents reported.

Oct 14, 2017

Linode Manager and API Maintenance: October 14th, 2017

Completed - Maintenance has been completed at this time.
Oct 14, 12:54 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 14, 10:00 UTC

Scheduled - A maintenance window has been scheduled for the Linode Manager and API on Saturday, October 14th between 06:00am and 10:00am EDT (10:00am and 2:00pm UTC). We expect this maintenance to take approximately 2 hours, but it is possible it will last the full 4 hour duration of the window. Linodes will not be affected by this maintenance, but the Linode Manager and API will be unavailable during this time.
Oct 3, 20:09 UTC

Support Ticket Outage

Resolved - We haven't observed any additional issues with Linode.com, the Linode Manager, or our API, and will now consider this incident resolved. If you are still experiencing additional issues, please email our Customer Support Team at support@linode.com for assistance.
Oct 14, 10:01 UTC

Monitoring - We have corrected the source of the issue affecting the ability to create Support Tickets via the Linode Manager and the Linode API. We are monitoring the situation and we will provide updates as soon as we can confirm this issue is fully resolved.
Oct 14, 09:56 UTC

Identified - At this time, we have identified the issues affecting opening Support Tickets via the Linode Manager and the Linode API. Our team is working as quickly as possible to resolve this issue. We will provide additional updates as they develop.
Oct 14, 09:32 UTC

Investigating - We are currently experiencing issues affecting opening a Support Ticket via the Linode Manager and the Linode API. Support Ticket responses are delayed as a result. Our team is investigating this issue and we will provide additional updates as this incident develops. If you have any urgent concerns, please contact us at our dedicated Support line (855-454-6633).
Oct 14, 09:17 UTC

Oct 13, 2017

Connectivity Issues - Linode Manager and Linode API

Monitoring - We have identified the source of the issue affecting the ability to create Support Tickets via the Linode Manager and the Linode API. We are monitoring the situation and we will provide updates as soon as we can confirm this issue is resolved.
Oct 13, 14:19 UTC

Update - We are currently investigating this issue.
Oct 13, 13:53 UTC

Investigating - We are currently experiencing issues affecting opening a Support Ticket via the Linode Manager, and the Linode API. Support Ticket responses are delayed as a result. Our team is investigating this issue and we will provide additional updates as this incident develops. If you have urgent concerns, please contact us at our dedicated Support line (855-454-6633).
Oct 13, 13:52 UTC

Lish Maintenance in Dallas

Completed - Lish Dallas maintenance is complete.
Oct 13, 12:05 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 13, 12:00 UTC

Scheduled - We will be performing maintenance on Lish in the Dallas data center on Oct 13, 2017 at 12:00 UTC for approximately 30 minutes. Dallas Lish will be inaccessible during this time. We will update this post once the work is complete.

Linodes within the Dallas data center can still be accessed by using Lish in a different data center. https://www.linode.com/docs/networking/using-the-linode-shell-lish
Oct 11, 18:39 UTC

Oct 12, 2017

Linode Critical Xen Maintenance

Resolved - We have successfully completed this maintenance.
Oct 12, 12:44 UTC

Investigating - Linode has received several Xen Security Advisories (XSAs) that requires us to perform updates to our legacy Xen host servers. In order to apply the updates, hosts and the Linodes running on them must be rebooted. The XSAs will be publicly released by the Xen project team on October 12th. We must complete this maintenance before then.

These security advisories only affect legacy Xen Linodes. KVM Linodes are not affected and do not require a reboot. Any customer that upgrades to KVM prior to the maintenance can avoid it entirely. You can use the “Upgrade to KVM” link in your Linode’s dashboard to move to KVM. More KVM upgrading information can be found here:

https://www.linode.com/docs/platform/kvm-reference

These updates are required to protect the security and safe operations of not only our infrastructure but yours as well. We understand that a disruption with such limited notice is inconvenient, and we hope you understand that we have no control over the disclosure date of this XSA. As such, we must adhere to this strict timeline to ensure the security of both our customers and platform. We strongly recommend you read through our Reboot guide, which will help you prepare your Linode for these reboots. The Reboot guide is available here:

https://www.linode.com/docs/uptime/reboot-survival-guide

Each Linode's maintenance window will be communicated to you via email and will also be visible within the Linode Manager. Unfortunately, unless you’re able to upgrade to KVM, due to the logistical demands of this effort your assigned maintenance windows are not changeable and the host reboots are mandatory. KVM is currently not available in Tokyo 1, which means Linodes in Tokyo will have to undergo maintenance on-schedule.

During the maintenance window, Linode instances will be cleanly shut down while we perform the updates. Your Linode will be inaccessible during this time. A two-hour window is allocated, however, the actual downtime should be much less. After the maintenance, each Linode will then be returned to its last state (running or powered off).

Stay tuned for more information.
Sep 28, 20:30 UTC

Oct 11, 2017

Service Issues - Linode Manager Jobs in Dallas

Resolved - Being that all jobs issued for Linodes within the Dallas datacenter are processing once more, this matter is now resolved. If you are still experiencing issues, please reach out to our Customer Support Team for assistance.
Oct 11, 09:28 UTC

Monitoring - At this time we have been able to correct the issues affecting jobs processing in our Dallas datacenter. If you are experiencing any further issues issuing jobs for your Linodes, please reach out to our Customer Support Team for assistance.
Oct 11, 07:41 UTC

Identified - At this time we have identified the service issues affecting Manager jobs for Linodes in Dallas. Our team is working as quickly as possible to have this service restored. We will provide additional updates as they develop.
Oct 11, 06:31 UTC

Investigating - We are currently experiencing issues processing jobs for Linodes in our Dallas datacenter. You may observe delays in the execution of boot, shut down, or other jobs. Other connectivity to Linodes in Dallas should not be affected. Our team is investigating this issue and we will provide additional updates as this incident develops.
If you are experiencing issues with your Linode, please email our Customer Support Team at support@linode.com.
Oct 11, 05:45 UTC

Connectivity Issues - Dallas

Resolved - Being that we have not experienced additional connectivity issues affecting our Dallas data center, this matter is now resolved. If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Oct 11, 05:13 UTC

Monitoring - Between approximately 02:39 - 02:43 UTC there was an interruption of network connectivity to our Dallas data center. Connectivity to Linodes, the Linode Manager, and the Linode API may have been impacted. Our Network Engineers were able to quickly identify and resolve the issue.

We are continuing to monitor the situation to ensure connectivity remains stable. If you are still experiencing connectivity issues, please reach out to our Customer Support Team.
Oct 11, 03:42 UTC

Oct 10, 2017

Connectivity Issues - Linode.com, Linode Manager and Linode API

Monitoring - At this time we have been able to correct the issues affecting Linode.com, the Linode Manager, and our API. We will be monitoring this issue to ensure that connectivity remains stable. If you are still experiencing connectivity issues with Linode.com, the Linode Manager or our API, please email our Customer Support Team at support@linode.com for assistance.
Oct 10, 15:50 UTC

Identified - We have identified the cause of the issues affecting Linode.com, the Linode Manager, and our API, and our team is currently working to correct this matter. This also impacts the ability for our Support team to respond to support tickets. If you have urgent concerns, please contact us at our dedicated Support line (855-454-6633). We will provide additional updates as this incident develops.
Oct 10, 15:28 UTC

Investigating - We are currently experiencing connectivity issues affecting Linode.com, the Linode Manager, and our API. Our team is investigating this issue and we will provide additional updates as this incident develops.
If you are experiencing issues with your Linode, please email our Customer Support Team at support@linode.com.
Oct 10, 14:15 UTC

Oct 9, 2017

No incidents reported.

Oct 8, 2017

No incidents reported.

Oct 7, 2017

No incidents reported.

Oct 6, 2017

Service Issue - Lish Dallas

Resolved - Our team has corrected the issue affecting the Lish service in Dallas. If you experience any further issues with Lish in our Dallas data center, please reach out to our Customer Support Team for assistance.
Oct 6, 18:22 UTC

Identified - Our team has identified the issue as a loss of power affecting the Lish service. We are working towards correcting this and we will provide updates as we move forward.
Oct 6, 17:59 UTC

Investigating - We are currently experiencing a service issue with Lish in our Dallas data center. Our team is investigating this issue and we will provide updates as we obtain additional information.
Oct 6, 17:34 UTC

Oct 5, 2017

No incidents reported.

Oct 4, 2017

Connectivity Issues - Frankfurt

Resolved - Being that we have not experienced additional connectivity issues affecting our Frankfurt data center, this matter is now resolved.

If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Oct 4, 02:52 UTC

Monitoring - The connectivity issues from the scheduled network maintenance in our Frankfurt data center has been corrected. Our team is continuing to monitor the situation to ensure connectivity remains stable.

If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Oct 4, 01:57 UTC

Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center as a result of the scheduled maintenance and are currently investigating. We will continue to provide additional updates as this incident develops.
Oct 3, 23:55 UTC

Monitoring - After many hours of troubleshooting with Cisco, it appears we have hit hardware and software failures on both internet facing edge routers simultaneously. The "A" side router had a module degrade silently, meaning that the module did not fully fail and caused strange forwarding behavior without notifying the operating system of the failure. Once the "A" side router was reloaded, the module failure became apparent during the boot process. Both redundant routers also experienced the same undocumented forwarding table bug that caused the blackholing of some customer traffic. At this time we have been able to normalize connectivity. A separate maintenance will be required to fully resolve this incident, at which point a full post mortem will be made available.
Oct 2, 03:32 UTC

Update - Our Network Operations team is continuing to work with Cisco on a resolution to these issues. We'll continue to provide updates as the situation develops.
Oct 2, 02:16 UTC

Update - Our engineers are still troubleshooting with Cisco. Cisco believes a software bug is causing the reachability problems we are seeing. A workaround is currently being developed. More updates to follow.
Oct 2, 01:01 UTC

Update - Both of our Frankfurt internet edge routers are experiencing issues with their forwarding tables. This is causing sporadic blackholing of customer traffic. Our engineers have reached out to the router vendor and are engaged with their engineering team. More updates will be provided as information become available.
Oct 1, 23:52 UTC

Update - We are currently still investigating the connectivity issues affecting Linodes in our Frankfurt data center. We will continue to provide additional updates as this incident develops.
Oct 1, 23:26 UTC

Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center and are currently investigating. We will continue to provide additional updates as this incident develops.
Oct 1, 14:14 UTC

Emergency Network Maintenance in Frankfurt

Completed - The scheduled maintenance has been completed.
Oct 4, 00:00 UTC

Update - We are extending this maintenance by 1 hour.
Oct 3, 23:52 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 3, 22:00 UTC

Scheduled - On Wednesday, October 4th, 2017 at approximately 00:00 CEST, Frankfurt local time, Linode will be conducting an emergency non-service affecting maintenance on both internet edge routers located in our Frankfurt data center. During this maintenance, engineers will be upgrading the software on each router to rectify a Cisco bug that caused packet loss and/or reachability issues for some customer traffic earlier in the week. Due to the redundant configuration of the routers, we do not anticipate any impact to customer traffic. The maximum duration for this maintenance is 2 hours.
Oct 3, 19:04 UTC

Subscribe to Incident

Subscribe to Incident

Subscribe to Incident