Service Issue - Object Storage - Newark
Incident
Report for Linode
Postmortem
At 03:00 UTC on November 1, 2023, we observed degraded performance with the Object Storage Service in our Newark Datacenter after experiencing a very large amount of inbound traffic. This resulted in customers experiencing 503 errors and connection timeouts. By 09:33 UTC, we identified and restricted the problematic traffic and restarted key infrastructure components. These actions resolved the immediate impact of the issue.
To provide further resiliency against future impacts, we are upgrading the Object Storage infrastructure to higher-performance hardware and have made improvements to our monitoring systems.
Resolved
We haven’t observed any additional issues with the Object Storage service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Monitoring
At this time we have been able to correct the issues affecting the Object Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Update
We are still investigating this issue.
Update
We are continuing to investigate the issue.
Update
Our administrators are still investigating this problem.
Update
At this time, we are continuing to investigate the issue.
Investigating
Our team is investigating degraded performance of the Object Storage service in our Newark datacenter. During this time, users may experience connection timeouts and errors with this service.
This incident affected: Object Storage (US-East (Newark) Object Storage) and Linode Kubernetes Engine (US-East (Newark) Linode Kubernetes Engine, US-Central (Dallas) Linode Kubernetes Engine, US-West (Fremont) Linode Kubernetes Engine, US-Southeast (Atlanta) Linode Kubernetes Engine, US-IAD (Washington) Linode Kubernetes Engine, US-ORD (Chicago) Linode Kubernetes Engine, CA-Central (Toronto) Linode Kubernetes Engine, EU-West (London) Linode Kubernetes Engine, EU-Central (Frankfurt) Linode Kubernetes Engine, FR-PAR (Paris) Linode Kubernetes Engine, AP-South (Singapore) Linode Kubernetes Engine, AP-Northeast-2 (Tokyo 2) Linode Kubernetes Engine, AP-West (Mumbai) Linode Kubernetes Engine, AP-Southeast (Sydney) Linode Kubernetes Engine, SE-STO (Stockholm) Linode Kubernetes Engine, US-SEA (Seattle) Linode Kubernetes Engine, JP-OSA (Osaka) Linode Kubernetes Engine, IN-MAA (Chennai) Linode Kubernetes Engine).