On November 6, 2024, at 5:30 UTC, we started upgrading an internal LKE component. At approximately 10:30 UTC, customers started reporting that they could not attach block storage volumes to LKE nodes.
Investigation revealed that the issue was caused by a software malfunction introduced during an internal LKE component upgrade. This resulted in errors due to a bad check intended to confirm volumes and Linodes were in the same region.
We began rolling back the internal upgrade region by region at 14:55 UTC and completed it at approximately 15:25 UTC. After monitoring our systems post-rollback, we confirmed that the issue was fully mitigated.
This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.