At 1:23 AM EDT our team was alerted to connectivity issues affecting services for a subset of customers in our Dallas, Fremont, Atlanta, Newark and Toronto data centers. An investigation determined that invalid DHCP lease configurations were the issue, and only customers using DHCP were affected. Additional resources were engaged to work on a fix. This work continued until approximately 10:45 AM EDT, when the fix was merged and DHCP configurations were regenerated, restoring service to all Linode infrastructure.
This incident was triggered by a series of changes introduced in preparation for Linode’s next data center launch. The latest deployment did not include a necessary update to our codebase which is partly responsible for the generation of our DHCP configurations. This resulted in the service that generates DHCP configurations to fail due to the error not being properly handled, which prevented Linodes from renewing their leases. We mitigated the event by adding the necessary changes to our codebase.
To prevent this from occurring again, additional monitoring has been put in place to alert us if the service that generates our DHCP configurations fails. Our team’s runbooks have been updated with additional preventative measures and tooling has been updated to better handle upcoming changes to Linode’s infrastructure.
This DHCP issue was not related to the other ongoing network incident affecting Linode’s London data center.
Thank you for your patience as we worked to resolve this. If you are still experiencing issues, enabling network helper and rebooting your Linode should resolve this issue for most customers. Please open a Support ticket if you need any further assistance.