Service Issue - Object Storage
Incident
Report for Linode
Postmortem
On 7 September, 2024 at 05:10 UTC customers using Object Storage in the Singapore datacenter began seeing HTTP 503 errors. Investigation revealed that the hosts providing the service were running out of memory, which caused the Object Storage service to become unavailable. This was mitigated by adding additional swap space to the hosts. Further investigation revealed that some servers had an incorrect service configuration, which caused the Object Storage cluster to be unable to recover after the memory issues were rectified.
To address this issue, we have corrected the configuration and optimized our Object Storage service for better memory usage.
Resolved
We haven’t observed any additional issues with the Object Storage service, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Update
We are continuing to monitor for any further issues.
Monitoring
At this time we have been able to correct the issues affecting the Object Storage service. However, customers may experience a few millisecond of latency until recovery completes. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.
Update
We are continuing to work on a fix for this issue.
Identified
Our team has identified the issue affecting the Object Storage service. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Update
We are continuing to investigate this issue and will provide the next update as we make progress.
Update
We are continuing to investigate this issue and will provide the next update as we make progress.
Investigating
Our team is investigating an issue affecting the Object Storage service. During this time, users may experience connection timeouts and errors with this service.
This incident affected: Object Storage (AP-South (Singapore) Object Storage).