On November 12th, 2023, at 21:49 UTC our administrators were alerted to network issues within the Dallas Datacenter that would impact Platform level operations of customer Linodes, such as Linode creation, reboot, boot and shutdown via the Cloud Manager and API.
After identifying the general cause, our administrators were able to mitigate the impacted performance issue and declared the incident resolved.
On November 13th, 2023, at 21:36 UTC we were again alerted to Platform action performance issues within the Dallas Datacenter and investigation began anew. We performed additional actions in an effort to fully resolve the issue. After monitoring for a short period of time, performance appeared to stabilize.
On November 14th, 2023 at 02:13 UTC, we were alerted to issues for a third time. Our network administrators made a third change that was expected to fix the performance issue.
However, at 19:13 UTC, we were once again alerted to a recurrence of the issue impacting customers. By this time, our administrators were able to identify a pattern pointing to the true root cause. Once identified, appropriate actions were taken to fully resolve the problem impacting customers.
Out of an abundance of caution, we expanded our monitoring time frame over the next two days to ensure that the impact was completely mitigated. We determined the issue to be fully resolved on November 17th, 2023 at 18:50 UTC.
As a result of this series of incidents, our network administrators are reviewing additional stability options for Platform service performance within the Dallas Datacenter.