Service Issue - LKE and Object Storage - All Regions
Incident
Report for Linode
Postmortem
On July 17, Akamai began upgrading its internal Kubernetes control plane used to implement several core services of our Compute products. In the course of troubleshooting a problem with the upgrade, our technical team unexpectedly triggered an automatic process that incorrectly rebuilt several resources in active use and critical to the operation of our products. As a result, the following products experienced downtime or degraded performance starting at 19:47 UTC.
- Object Storage Key and bucket management were unavailable via the Linode Cloud Manager and Linode APIv4.
- Object Storage: E2 and E3 endpoints were unavailable in Chicago, Frankfurt, London, Melbourne, Mumbai, Seattle, Singapore, Tokyo, and Washington.
- LKE and LKE-E: the operations to create new clusters or restore clusters from backup were unavailable.
- Image Service: the operations to create and restore images were unavailable.
- Managed Databases: the operations to provision new instances, resize, resume suspended databases, or restore from backup were unavailable. In addition, the automatic recreation of failed nodes in multi-node databases was unavailable.
To address the problem, we initiated a comprehensive process to correctly rebuild the affected resources and to gradually bring back systems into operation. Throughout this process, we conducted extensive testing and validation operations.
At approximately 00:20 UTC on July 18, 2025, the following services began to come back online:
- Image Service: full operation restored.
- LKE and LKE-Enterprise: new cluster provisioning available, except in London and Chicago.
- Managed Databases: all operations fully available.
The affected Object Storage E2 and E3 endpoints were re-established at the following hours. It is important to note that no data stored in these Object Storage systems was lost or damaged during this incident.
| Region | Mitigation Time (UTC) | Date |
|---|---|---|
| Melbourne (au-mel) | 22:44 | July 18, 2025 |
| London (gb-lon) | 07:55 | July 19, 2025 |
| Tokyo (jp-tyo-3) | 11:40 | July 19, 2025 |
| Washington (us-iad) | 12:13 | July 19, 2025 |
| Singapore (sg-sin-2) | 12:30 | July 19, 2025 |
| Frankfurt (de-fra-2) | 13:20 | July 19, 2025 |
| Mumbai (in-bom-2) | 13:40 | July 19, 2025 |
| Seattle (us-sea) | 15:14 | July 19, 2025 |
| Chicago (us-ord) | 15:24 | July 19, 2025 |
To prevent a recurrence of this problem, we have frozen changes to the Kubernetes systems involved in this incident and are reviewing all designs and supporting operational processes, with the expectation that we will implement additional safeguards.
This summary provides an overview of our current understanding of the incident, given the information available. Our investigation is ongoing, and any information herein is subject to change.
Resolved
We haven’t observed any additional issues with the LKE service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Update
We have successfully corrected the issue affecting the Object Storage E2/E3 endpoints in Melbourne, London, Tokyo, Washington, Singapore, Frankfurt, Mumbai, Seattle, and Chicago. Also, the Linode Kubernetes Engine (LKE) service has been restored and is working as expected. We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues and unable to open a Support ticket, please call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Monitoring
At this time, we have successfully corrected the issue affecting the Object Storage E2/E3 endpoints in Melbourne, London, Tokyo, Washington, Singapore, Frankfurt, Mumbai, and Seattle. Also, the Linode Kubernetes Engine (LKE) service has been restored and is working as expected. We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues and unable to open a Support ticket, please call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Update
The efforts to establish Object Storage E2/E3 endpoints in Seattle have been successfully completed, following similar achievements in Melbourne, London, Tokyo, Washington, Singapore, Frankfurt, and Mumbai. Work on the remaining Object Storage E2/E3 endpoints is still ongoing, and it will require additional time to fully resolve these issues. We appreciate your patience and will provide further updates as progress is made.
Update
The efforts to establish Object Storage E2/E3 endpoints in Frankfurt and Mumbai have been successfully completed, following similar achievements in Melbourne, London, Tokyo, Washington, and Singapore. Work on the remaining Object Storage E2/E3 endpoints is still ongoing, and it will require additional time to fully resolve these issues. We appreciate your patience and will provide further updates as progress is made.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
The efforts for Object Storage E2/E3 endpoints from London, Melbourne, Tokyo, Washington and Singapore are completed. For the remaining Object Storage E2/E3 endpoints, the mitigation efforts are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
The efforts for Object Storage E2/E3 endpoints from London, Melbourne and Tokyo are completed. For the remaining Object Storage E2/E3 endpoints, the mitigation efforts are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
The efforts for Object Storage E2/E3 endpoints from London and Melbourne are completed. For the remaining Object Storage E2/E3 endpoints, the mitigation efforts are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
Our subject matter experts are continuing to work on mitigating the impact. The efforts for Object Storage E2/E3 endpoints from Melbourne are completed. For the remaining Object Storage E2/E3 endpoints, the mitigation efforts are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
Our subject matter experts are continuing to work on mitigating the impact. The efforts for Object Storage E2/E3 endpoints from Melbourne are completed. For the remaining Object Storage E2/E3 endpoints, the mitigation efforts are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
Our subject matter experts are continuing to work on mitigating the impact. The Linode Kubernetes Engine (LKE) service has been largely restored, and we continue to work on mitigating the impact in the London and Chicago regions. Mitigation efforts for Object Storage E2/E3 endpoints are ongoing and will require additional time to fully resolve the issue. We appreciate your patience and will provide further updates as progress continues.
Update
We are continuing to work on mitigating the impact of this incident. The Linode Kubernetes Engine (LKE) service has been largely restored and is operating normally for most customers. Mitigation efforts for Object Storage E2/E3 endpoints are ongoing and will require additional time to fully resolve. We appreciate your patience and will provide further updates as progress continues.
Update
Our subject matter experts are actively working on the recovery of the impacted Object Storage E2/E3 endpoints. We will share further updates as we make progress. We apologize for the impact and thank you for your patience and continued support. If you have any questions, please contact our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Update
Current Status
Our teams are actively working to resolve issues impacting Object Storage E2 and E3 endpoints. While Standard (E0/E1) endpoints were not affected, we are working on system recovery for all E2 and E3 endpoints. These issues are being prioritized, and expert teams are engaged in resolving the impact.
Object Storage E2 and E3 Endpoint Recovery Plan
We are employing an incremental approach to ensure a safe and methodical recovery.
> Recovery order by Datacenter: We understand that you may have questions about the recovery process for your specific region. At this time, we are taking a step-by-step approach, and as a result, we do not yet have a determined order for datacenter recovery. Our focus is on restoring systems safely, working to minimize impact across all customers. We will share updates regularly as recovery progresses and will notify you promptly once we can confirm timelines for your specific E2/E3 Endpoint region(s).
> Testing and Validation: To ensure stability and avoid service disruptions, all systems will undergo extensive testing prior to resuming regular operations.
Impact and Timelines
While we are devoting 24/7 engineering resources to this recovery operation, complete resolution will depend on the results of our ongoing diagnostics. Full recovery for affected regions may require multiple days and our teams are looking for ways to optimize and parallelize recovery operations. As we make further progress over the next few hours, we expect to be able to provide more specific updates on estimated recovery times.
Our Commitment
We deeply regret the impact being caused by this disruption and want to assure you of our commitment to restoring full operations as quickly and safely as possible. We will keep you updated with progress and key milestones throughout the recovery process. If you have any questions or concerns, please do not hesitate to reach out to our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Our teams are actively working to resolve issues impacting Object Storage E2 and E3 endpoints. While Standard (E0/E1) endpoints were not affected, we are working on system recovery for all E2 and E3 endpoints. These issues are being prioritized, and expert teams are engaged in resolving the impact.
Object Storage E2 and E3 Endpoint Recovery Plan
We are employing an incremental approach to ensure a safe and methodical recovery.
> Recovery order by Datacenter: We understand that you may have questions about the recovery process for your specific region. At this time, we are taking a step-by-step approach, and as a result, we do not yet have a determined order for datacenter recovery. Our focus is on restoring systems safely, working to minimize impact across all customers. We will share updates regularly as recovery progresses and will notify you promptly once we can confirm timelines for your specific E2/E3 Endpoint region(s).
> Testing and Validation: To ensure stability and avoid service disruptions, all systems will undergo extensive testing prior to resuming regular operations.
Impact and Timelines
While we are devoting 24/7 engineering resources to this recovery operation, complete resolution will depend on the results of our ongoing diagnostics. Full recovery for affected regions may require multiple days and our teams are looking for ways to optimize and parallelize recovery operations. As we make further progress over the next few hours, we expect to be able to provide more specific updates on estimated recovery times.
Our Commitment
We deeply regret the impact being caused by this disruption and want to assure you of our commitment to restoring full operations as quickly and safely as possible. We will keep you updated with progress and key milestones throughout the recovery process. If you have any questions or concerns, please do not hesitate to reach out to our support team at support@linode.com or call us at +1-855-454-6633 (US) or +1-609-380-7100 (Global) for assistance.
Identified
Affected services have begun to come back online. Currently, the following services are up:
• Creating and restoring images to Linodes
• LKE and LKE-Enterprise new cluster provisioning, except in London
• Managed Databases
We are working to restore service to all affected Object Storage E2 and E3 endpoints and will continue to provide updates as more services come back online.
• Creating and restoring images to Linodes
• LKE and LKE-Enterprise new cluster provisioning, except in London
• Managed Databases
We are working to restore service to all affected Object Storage E2 and E3 endpoints and will continue to provide updates as more services come back online.
Update
We are continuing to investigate this issue. The appropriate subject matter experts are engaged. We will continue to provide hourly updates until we have a clearer picture about the mitigation plan. Subsequent updates around mitigation status will be posted as progress is made.
Update
We have confirmed that the issue is impacting the following services:
• Object Storage Key and bucket management using the Linode Cloud Manager and Linode APIv4
• The following Object Storage E2 and E3 endpoints are down:
• jp-osa-1.linodeobjects.com
• us-sea-9.linodeobjects.com
• au-mel-1.linodeobjects.com
• gb-lon-1.linodeobjects.com
• in-bom-1.linodeobjects.com
• jp-tyo-1.linodeobjects.com
• us-iad-13.linodeobjects.com
• us-ord-10.linodeobjects.com
• de-fra-1.linodeobjects.com
• sg-sin-1.linodeobjects.com
• Linode Kubernetes Engine (LKE) and LKE-Enterprise
• Creating clusters
• Restoring clusters from a backup
• Creating and restoring images, including deployment of Linodes from images
• Managed Databases
• Provisioning new clusters
• Resizing existing clusters
• Resuming a suspended cluster
• Restoring existing clusters from backup
• Multi-node clusters will be able to recover from a primary node failure automatically as long as cluster standby Linodes are online
• Automatic recreation of failed nodes is unavailable at this time
We will continue to share updates as we have more information.
• Object Storage Key and bucket management using the Linode Cloud Manager and Linode APIv4
• The following Object Storage E2 and E3 endpoints are down:
• jp-osa-1.linodeobjects.com
• us-sea-9.linodeobjects.com
• au-mel-1.linodeobjects.com
• gb-lon-1.linodeobjects.com
• in-bom-1.linodeobjects.com
• jp-tyo-1.linodeobjects.com
• us-iad-13.linodeobjects.com
• us-ord-10.linodeobjects.com
• de-fra-1.linodeobjects.com
• sg-sin-1.linodeobjects.com
• Linode Kubernetes Engine (LKE) and LKE-Enterprise
• Creating clusters
• Restoring clusters from a backup
• Creating and restoring images, including deployment of Linodes from images
• Managed Databases
• Provisioning new clusters
• Resizing existing clusters
• Resuming a suspended cluster
• Restoring existing clusters from backup
• Multi-node clusters will be able to recover from a primary node failure automatically as long as cluster standby Linodes are online
• Automatic recreation of failed nodes is unavailable at this time
We will continue to share updates as we have more information.
Investigating
Our team is investigating an emerging service issue affecting LKE and Object Storage in all regions. We will share additional updates as we have more information.
This incident affected: Object Storage (US-East (Newark) Object Storage, US-Southeast (Atlanta) Object Storage, US-IAD (Washington) Object Storage, US-ORD (Chicago) Object Storage, EU-Central (Frankfurt) Object Storage, AP-South (Singapore) Object Storage, FR-PAR (Paris) Object Storage, SE-STO (Stockholm) Object Storage, US-SEA (Seattle) Object Storage, JP-OSA (Osaka) Object Storage, IN-MAA (Chennai) Object Storage, ID-CGK (Jakarta) Object Storage, BR-GRU (Sao Paulo) Object Storage, ES-MAD (Madrid) Object Storage, GB-LON (London 2), AU-MEL (Melbourne), NL-AMS (Amsterdam) Object Storage, IT-MIL (Milan) Object Storage, US-MIA (Miami) Object Storage, US-LAX (Los Angeles) Object Storage, GB-LON (London 2) Object Storage, AU-MEL (Melbourne) Object Storage, IN-BOM-2 (Mumbai 2) Object Storage, DE-FRA-2 (Frankfurt 2) Object Storage, SG-SIN-2 (Singapore 2) Object Storage, JP-TYO-3 (Tokyo 3) Object Storage) and Linode Kubernetes Engine (US-East (Newark) Linode Kubernetes Engine, US-Central (Dallas) Linode Kubernetes Engine, US-West (Fremont) Linode Kubernetes Engine, US-Southeast (Atlanta) Linode Kubernetes Engine, US-IAD (Washington) Linode Kubernetes Engine, US-ORD (Chicago) Linode Kubernetes Engine, CA-Central (Toronto) Linode Kubernetes Engine, EU-West (London) Linode Kubernetes Engine, EU-Central (Frankfurt) Linode Kubernetes Engine, FR-PAR (Paris) Linode Kubernetes Engine, AP-South (Singapore) Linode Kubernetes Engine, AP-Northeast-2 (Tokyo 2) Linode Kubernetes Engine, AP-West (Mumbai) Linode Kubernetes Engine, AP-Southeast (Sydney) Linode Kubernetes Engine, SE-STO (Stockholm) Linode Kubernetes Engine, US-SEA (Seattle) Linode Kubernetes Engine, JP-OSA (Osaka) Linode Kubernetes Engine, IN-MAA (Chennai) Linode Kubernetes Engine, ID-CGK (Jakarta) Linode Kubernetes Engine, BR-GRU (São Paulo) Linode Kubernetes Engine, NL-AMS (Amsterdam) Linode Kubernetes Engine, ES-MAD (Madrid) Linode Kubernetes Engine, IT-MIL (Milan) Linode Kubernetes Engine, US-MIA (Miami) Linode Kubernetes Engine, US-LAX (Los Angeles) Linode Kubernetes Engine, GB-LON (London 2) Linode Kubernetes Engine, AU-MEL (Melbourne) Linode Kubernetes Engine, IN-BOM-2 (Mumbai 2) Linode Kubernetes Engine, DE-FRA-2 (Frankfurt 2) Linode Kubernetes Engine, SG-SIN-2 (Singapore 2) Linode Kubernetes Engine, JP-TYO-3 (Tokyo 3) Linode Kubernetes Engine).