Service Issue - Linode Kubernetes Engine
Incident
Report for Linode
Postmortem
Starting at ~15:00 UTC on Apr 20, 2022, a change was rolled out to the LKE platform that introduced rate limits to the API Server Gateway. These rate limits resulted in kubelets being blocked from communicating to the API server inadvertently. This caused the nodes to enter a NotReady state and spike in CPU usage. Internal alerting notified the teams starting at ~17:00 UTC on April 20. The teams investigated and identified the root cause of the issue by ~13:00 UTC on April 21, 2022. By ~15:30 UTC on Apr 21, 2022 that change had been reverted which immediately unblocked the nodes that had been blocked.
Resolved
We haven’t observed any additional issues with the LKE service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Monitoring
At this time we have been able to correct the issues affecting the LKE service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Identified
Our team has identified the issue affecting the LKE service. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
This incident affected: Linode Kubernetes Engine (US-East (Newark) Linode Kubernetes Engine, US-Central (Dallas) Linode Kubernetes Engine, US-West (Fremont) Linode Kubernetes Engine, US-Southeast (Atlanta) Linode Kubernetes Engine, CA-Central (Toronto) Linode Kubernetes Engine, EU-West (London) Linode Kubernetes Engine, EU-Central (Frankfurt) Linode Kubernetes Engine, AP-South (Singapore) Linode Kubernetes Engine, AP-Northeast-2 (Tokyo 2) Linode Kubernetes Engine, AP-West (Mumbai) Linode Kubernetes Engine, AP-Southeast (Sydney) Linode Kubernetes Engine).