Linode Status

Current Status
Service Issue - Object Storage
Incident Report for Linode
Postmortem

Beginning 02:52 on Mar 3, 2023 an incident occurred impacting access to the linodeobjects.com domain name. This impacted all customers without cached DNS entries attempting to access the Linode Object Storage service, and customers who utilized the domain via a CNAME record. The internal teams were notified at 02:57, and after initial triage, further teams were paged by 03:10 to investigate further.

The root cause of the incident was a non-billing hold placed on the linodeobjects.com domain by the registrar which prevented DNS from functioning for the domain. After initial attempts to contact the vendor were unsuccessful, attempts were made to begin mitigating this issue through other means. By 05:43 UTC, access to the Object Storage service was restored for customers accessing the service via the internal network (such as from a Linode instance to the Object Storage service). Full service restoration was not achieved until 11:56 UTC when the domain was moved to a new registrar. Customers may not have seen this restoration for up to 30 minutes depending on local caching.

We are still investigating with the former registrar to determine why the non-billing hold was placed on the domain.

As a result of this, we will be conducting an internal audit of critical services and the vendors they are tied to in order to better assess support accessibility and establish escalation paths to more easily reach these vendors during urgent incidents. As this domain is no longer on the same registrar, we do not anticipate a further recurrence of this issue impacting this service.

Posted Mar 03, 2023 - 22:34 UTC

Resolved
We haven’t observed any additional issues with the Object Storage service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Mar 03, 2023 - 11:51 UTC
Monitoring
At this time we have been able to correct the issues affecting the Object Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Posted Mar 03, 2023 - 09:08 UTC
Identified
Our team has identified the issue affecting our Object Storage service and are working quickly to implement a fix. At this time we have restored internal DNS requests, however, external DNS requests are still experiencing issues. We will provide an update once a solution is in place.
Posted Mar 03, 2023 - 06:20 UTC
Update
Our team is still continuing to investigate this issue.
Posted Mar 03, 2023 - 05:45 UTC
Update
We are continuing to investigate this issue.
Posted Mar 03, 2023 - 04:32 UTC
Investigating
Our team is investigating an issue affecting the Object Storage service. During this time, users may experience connection timeouts and errors with this service.
Posted Mar 03, 2023 - 03:23 UTC