Beginning March 20th, 2024, we began to receive customer reports of an issue with certificate chains for NodeBalancers in different Compute regions. Our investigation revealed that a recent code change caused the problem.
The code change was intended to mitigate cases where the provided certificate was corrupted and would result in a NodeBalancer not starting at all. This change resurfaced an issue with validation of chain certificates in the APINext component. In its turn the validation issue was caused by a python cryptography library that only treats a single certificate in a chain.
To mitigate the incident, we rolled back the code change. Affected customers are advised to reconfigure their chain certificates and validate SSL termination capabilities of their services.
Our team is diligently working on implementing essential changes to our infrastructure to provide proper support for chain certificates. In addition, we are carefully planning the reintroduction of the code change responsible for the incident to ensure a smooth deployment upon the next attempt.