Service Issue - Linode Kubernetes Engine
Incident
Report for Linode
Resolved
We haven’t observed any additional issues with the LKE service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Monitoring
Our team has identified the cause of the issue, and applied a fix to restore node creation on newly provisioned LKE deployments. With the fix now applied, we have also reintroduced the problematic infrastructure which was temporarily disabled while working to mitigate initial impact to customers.
We will now be monitoring to ensure that service remains stable. If you continue to experience problems, please open a Support ticket for assistance.
We will now be monitoring to ensure that service remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Update
Our team continues to investigate an issue impacting newly provisioned LKE deployments. The error results in clusters having an incorrect number of nodes in their pool, or no nodes at all. While investigating the cause of the issue, our team was able to detect occurrences in a distinct section of our infrastructure within our Toronto data center. Once identified, our team disabled its current use for creating new LKE clusters. This will prevent customer impact while our team continues to identify a root cause and then develop, test, and then apply a fix.
This issue continues to occur with higher frequency in our Toronto data center, specifically with the distinct backend resource. During our investigation, we have seen instances of errors appearing in other data centers, though with tremendously less frequency. The behavior customers would see if affected would be a timeout in the Cloud Manager after creating a new LKE cluster. Upon refreshing the Cloud Manager, customers would see a newly created cluster, though with no node pools included.
This issue continues to occur with higher frequency in our Toronto data center, specifically with the distinct backend resource. During our investigation, we have seen instances of errors appearing in other data centers, though with tremendously less frequency. The behavior customers would see if affected would be a timeout in the Cloud Manager after creating a new LKE cluster. Upon refreshing the Cloud Manager, customers would see a newly created cluster, though with no node pools included.
Update
We are aware of an issue which may cause new LKE deployments in the Toronto datacenter to fail. Our team is continuing to investigate and we will provide another update within the next 24 hours.
Investigating
We are currently investigating this issue.
Update
We are continuing investigate this issue.
Monitoring
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
Our team is investigating an issue affecting the Linode Kubernetes Engine (LKE). We will share additional updates as we have more information.
This incident affected: Linode Manager and API.