On October 3rd, 2023 at 18:30 UTC, we observed degraded performance across multiple hosts in Sydney.
Upon further investigation it was discovered that a recent code change was preventing processes from properly terminating. This led to resource contention which degraded performance across the affected hosts.
After identifying the cause, our administrators were able to revert the changes and clear the affected hosts of issues. This was completed at 00:18 UTC October 4th, 2023.
From here, we are improving our testing environment to more comprehensively account for changes in hosting configurations.