At 14:38 UTC on August 23rd, 2023, we began to receive reports from customers that rate-limiting errors were being encountered when pulling images from Dockerhub. After initial investigation into these reports, our administrators determined that the issue was also impacting LKE deployments on our platform and incident response procedures began at 16:51 UTC.
It was discovered that new rate-limiting policies implemented by the Docker team treated customers on our platform utilizing default IPv6 addresses as a single entity. This was due to how default IPv6 addresses are assigned to customers on our platform.
Each Linode is assigned a single /128 IPv6 address from a default /64 IPv6 range within each of our data centers. This not only affected LKE worker nodes, but any workload deployed by a customer that utilized Dockerhub without a login. A status page was created at 17:44 UTC to relay this information to our customers.
We were able to have this change reverted for our default /64 IPv6 ranges, effectively removing customer impact. The incident was moved to a Monitoring status at 20:28 UTC after the exception was implemented. After stability was confirmed, the incident was considered Resolved at 12:26 UTC on August 27th, 2023.
While immediate mitigation of the issue has been addressed, we are working on long-term solutions to improve default IPv6 implementation and prevent future impact.