The root cause of the issues experienced on Thursday May 5th occurred while bringing new transit routers into service. Once IPv6 peering was established we noticed unusual behavior in cli output and peering stability.
The problem centered around the supervisor card on the primary router cloning of certain ipv6 bgp peering sessions in the realm of 400+ replicas of 3 sessions.) This led to periodic tear down of the actual bgp sessions. We shutdown all v6 bgp on the router, and gracefully moved v4 traffic to gw2, in order to reboot the primary gateway. Once we had the gateway up and stable, we re-enabled ipv6 and monitored for any recurrence of the condition that led to the initial instability. We've not seen any 'cloning' behavior since for ipv6 sessions.
Dealing with and fixing that issue meant that 2 further issues with v6 and v4 advertisements from these routers went incorrectly diagnosed while we dealt with the first issue. Once we identified both issues, we corrected the configuration on the new routers to ensure that this doesn't happen again.
These incidents would have resulted firstly in losing IPv6 connectivity to all customers in Fremont. IPv4 connectivity was also affected between Frankfurt/Fremont and Newark/Fremont. We were able to rollback the IPv4 migration and reschedule those for 05/09/16.
During the outage we considered rolling back, but felt that this may have been ultimately more impactful, as we had already communicated changes to our upstream provider and planned for this work, who had executed their changes already.