On 05-25-2021 at 22:00 UTC, our Network Operations team began scheduled maintenance on one of our aggregate switches in the Frankfurt data center. This switch is a member of a redundant switch pair. The maintenance required a reboot of the switch, and upon coming back online the switch was missing part of its configurations.
While the team was working to correct this, our monitoring system alerted us to connectivity issues in the Frankfurt data center. This began at 00:15 UTC 05-26-2021. Investigation showed that the switch under maintenance was accepting traffic but unable to forward it or fail over. This resulted in a partial loss of connectivity in the data center, as only the A-side of the redundant pair was forwarding traffic.
We completed the manual configuration of the switch at 1:37 UTC, which restored full connectivity to the data center. We have also corrected the issue on the switch that caused it to lose configurations on reboot. We have not experienced any technical issues on this switch since the completion of the maintenance and we believe this issue to be fully resolved at this point.