On November 29, 2024, at approximately 14:54 UTC, we encountered an issue affecting connectivity in one of the clusters of our US-LAX (Los Angeles) data center. It resulted in customers experiencing degraded performance across services while connecting to the affected cluster during the period of impact. The following investigation revealed that the issue was due to resource exhaustion in the affected cluster.
Once the cause of the situation was identified, we added additional capacity to the cluster at 23:00 UTC on November 29, 2024. After monitoring our systems for 24 hours to ensure no recurrences, we confirmed that the issue was mitigated.
This summary provides an overview of our current understanding of the incident, given the information available. Our investigation is ongoing, and any information herein is subject to change.