Linode Status

Current Status
Connectivity Issue - US-East (Newark)
Incident Report for Linode
Postmortem

On November 22, 2024 at 21:43 UTC, our Compute Operations Center received alerts indicating that DHCP daemons responsible for assigning IP addresses to customer’s virtual machines were not running on ~20 compute hosts from our US-East (Newark) data center. This issue affected customers relying on DHCP for network configuration, potentially resulting in downtime for virtual machines once their DHCP lease expired. Approximately 2,500 customer virtual machines resided on the affected compute hosts during the impact window, but not all were impacted.

The investigation revealed that two services (rad-unnumbered and DHCP-unnumbered) were not restarted on these ~20 hosts following the routine security patch deployments, while the same updates were running on the remaining ~9000 hosts without any issues. For immediate mitigation, we identified and restarted the impacted daemons on the affected hosts to restore DHCP functionality. The issue was mitigated at 23:16 UTC on November 22, 2024. Akamai concluded the root cause investigation and found that the ~20 affected hosts had missed a prerequisite update, which would have ensured the DHCP daemons restarted correctly. This discrepancy rendered the hosts vulnerable during the patch deployment. We were able to run a query and determined there were 110 other hosts with this same out-of-date configuration making them susceptible to issues upon restart; the work to update the configurations for all machines is now completed. 

This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

Posted Dec 04, 2024 - 19:22 UTC

Resolved
We haven’t observed any additional connectivity issues in our US-East (Newark) data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Nov 27, 2024 - 19:19 UTC
Update
We will be monitoring this issue over the weekend to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Nov 23, 2024 - 02:05 UTC
Update
We are continuing to monitor for any further issues.
Posted Nov 23, 2024 - 00:21 UTC
Monitoring
Our team has identified an issue affecting connectivity in our US-East (Newark) data center. During this time, users may have experienced intermittent connection timeouts and errors for all services deployed in this data center. The issue appears to have started around 21:43 UTC on November 22, 2024. As of 23:16 UTC we have been able to correct the issue affecting connectivity. We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues and unable to open a Support ticket, please call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Posted Nov 22, 2024 - 23:47 UTC
This incident affected: Regions (US-East (Newark)).