On November 20th, 2023 at 19:21 UTC, our team deployed a change to the DNS configuration of our API. This caused unintended behavior for customers attempting to interact with certain parts of the Cloud Manager and API and we immediately began our incident response procedure.
Our team was able to mitigate customer impact shortly after it was reported. We completed a full rollback of the change at 20:21 UTC.
We continued to monitor for any outlying behavior before determining the incident resolved at 22:39 UTC.
As a result of this incident, our team has reviewed the testing environment and identified some discrepancies with the production environment, which have been addressed.
We also are working to implement changes to DNS behavior to further decrease response times to changes made.