On September 5th, 2019, at approximately 11:30 AM EDT (15:30 UTC) our Systems Administrators began troubleshooting an issue impacting customer DNS resolution. During this process we initiated a restart of the underlying systems on our resolvers, however, those systems failed to come back online as expected. This, in turn, caused DNS resolution in our Dallas and Atlanta data centers to begin to fail. We were quickly notified by our monitoring systems and were able to address the issue shortly after it began.
Access to the resolvers in Dallas was restored by 11:55 AM EDT, and access to our Atlanta resolvers was restored by 12:08 PM EDT.
Going forward, our Systems Administrators will be adjusting our deployment plan to limit the risk of potential impact.