At approximately 0630 UTC January 3, 2024, infrastructure that supports a subset of the Block Storage service in the Dallas region experienced a loss of connectivity. Customers experienced failures with Linodes attempting to access data located on their Block Storage devices; additionally, the loss of connectivity resulted in boot failures for Linodes attached to this storage server.
Our administrators rebooted the affected node and it was back online at approximately 0730 UTC which restored normal working service functionality. Linode Support monitored for additional customer reports for a few hours and updated the status page to indicate the official resolution at 1444 UTC.
The loss of connectivity was due to the affected node in our storage infrastructure locking up unexpectedly. Our monitor caught this lockup, and we rebooted the server as a mitigation measure. After the reboot, an error was discovered in the time synchronization process which prevented the node from syncing with the rest of the storage cluster causing further connectivity issues until the synchronization process completed. Our Compute Storage Reliability Team has analyzed the root cause of the time synchronization failure and has implemented a fix to prevent a future recurrence.