On February 5, 2025, at approximately 21:38 UTC, DockerHub imposed rate limits on image pulls from our IPv6 IP space across multiple regions. This resulted in 429 Too Many Requests errors when making both unauthenticated and authenticated image pull requests over IPv6. As a consequence, several products, including LKE and LKE-E, as well as other services relying on DockerHub image pulls, were impacted.

The issue was traced back to DockerHub’s abuse prevention measures, which rate-limited traffic based on IPv6 address checks. DockerHub clarified that the failure occurred due to exceeding their abuse limit, which was triggered by how IPv6 checks the first 64 bits of an address. As an interim fix to get around this block, we’ve modified our DNS resolvers to return an 'A' record instead of an 'AAAA' record, ensuring image pull requests to DockerHub use IPv4 instead. This action successfully mitigated the issue at approximately 04:30 UTC on February 6, 2025. To prevent a recurrence of this issue, we are actively exploring long-term solutions that help ensure that our services are not impacted by DockerHub’s abuse prevention measures going forward. 

This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.