On October 6, 2024, at approximately 11:33 UTC, our administrators received multiple alerts notifying us of database connectivity issues in Toronto. During the impact window, customers were intermittently unable to create new services, delete services, or reboot their Linodes in our CA-Central (Toronto) data center.
Investigation revealed that a faulty router was causing the issue, and the device was promptly taken offline at 17:15 UTC. This quickly mitigated the impact on customers.
After addressing the immediate issue, we initiated the process of replacing the faulty router to restore capacity and redundancy. We conducted an emergency network maintenance at our CA-Central (Toronto) data center from October 7, 2024, 22:00 UTC to October 8, 2024, 02:00 UTC, to install the new router. After encountering some unexpected complications with this process, an additional round of maintenance was required. This second maintenance started on October 8, 2024, at 22:00 UTC and was successfully completed on October 9, 2024, at 00:15 UTC.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.