On January 1, 2025, customers started experiencing intermittent connection timeouts and errors for all services in our Washington Data Center (US-IAD).

‌

Investigation revealed that a large usage of Virtual Private Cloud (VPC) was causing significant disruptions. In order to mitigate the immediate impact, we requested customers to limit the size of their VPC and reduce the number of Linodes customers were creating and deleting.

‌

Additionally, we implemented a software update in our Washington Data Center (US-IAD) on January 9, 2025, during an emergency maintenance window between 04:00 UTC to 09:00 UTC. During this maintenance, customers experienced unexpected intermittent connectivity disruptions. After implementing this update, we monitored our data center for several days.

‌

During the monitoring period, some customers reported they were experiencing network disruptions while they were creating new Linodes. After further investigation, it was found that the update that was previously implemented in our data center had a software defect that was causing this disruption.

‌

To mitigate the impact caused by this software malfunction, we rolled back this deployment on January 23, 2025, between 04:00 UTC and 09:00 UTC, during a scheduled maintenance in our Washington Data Center (US-IAD). After monitoring our data center for a few hours, we confirmed that the issue introduced by the software defect had been resolved.

‌

We continue working on a fix to resolve the issue that the large VPC usage has been causing. In the meantime, our recommendation is to limit the size of the VPC and reduce the number of Linodes customers are creating and deleting.

‌

This summary provides an overview of our current understanding of the incident, given the information available. Our investigation is ongoing, and any information herein is subject to change.