Skip to content

fix(auth): add startup polling for Cloud Run SPIFFE mTLS certificates - #19161

Open
daniel-sanche wants to merge 1 commit into
mainfrom
rollback_google_auth_release_google-auth_02_revert-19159
Open

daniel-sanche wants to merge 1 commit into
mainfrom
rollback_google_auth_release_google-auth_02_revert-19159

Conversation

@daniel-sanche

Copy link
Copy Markdown
Contributor

Rolls back #19159 to restore #18924:

A startup latency can occur where the environment variable pointing to the certificate configuration is set before the files are actually mounted on the filesystem. While token refresh already polls for the certificate file, mTLS transport discovery did not, causing transports initialized at container startup to fall back to standard TLS and fail with 401 once a bound token was minted.

  • Return True from check_use_client_cert() when GOOGLE_API_CERTIFICATE_CONFIG points inside the well-known SPIFFE directory so mTLS endpoint selection is enabled at startup without blocking on disk I/O.
  • Poll up to 30 seconds (50 retries at 100ms followed by 50 retries at 500ms) in _get_workload_cert_and_key() and _get_workload_cert_and_key_paths() for the config, certificate, and private key files to be written, failing fast on invalid config schemas, or PermissionError.
  • Add unit tests in TestSpiffeWorkloadPolling covering cold-start polling recovery, timeout, and fail-fast conditions.

fixes b/580018256

@daniel-sanche
daniel-sanche requested review from a team as code owners November 21, 2026 03:43
@daniel-sanche daniel-sanche added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 21, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces SPIFFE workload polling for mutual TLS (mTLS) configurations, allowing up to a 30-second wait for SPIFFE configurations, certificates, and keys to become ready, and adds comprehensive tests for these scenarios. The review feedback recommends catching exceptions.ClientCertError in default_client_encrypted_cert_source to ensure consistent exception wrapping, and resetting _spiffe_poll_timed_out to False upon successful credential loading to support future certificate rotations.

Comment on lines 139 to +149
try:
(
_,
has_cert,
cert_bytes,
key_bytes,
passphrase_bytes,
) = _mtls_helper.get_client_ssl_credentials(generate_encrypted_key=True)
if not has_cert:
raise exceptions.MutualTLSChannelError(
"Default client cert source did not provide a certificate"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In default_client_encrypted_cert_source, the except block (which is just below this hunk) is not updated to catch exceptions.ClientCertError. This is inconsistent with default_client_cert_source (lines 96-101) and will cause ClientCertError (e.g., from a SPIFFE polling timeout) to propagate directly instead of being wrapped in MutualTLSChannelError as expected.

Please update the except block of default_client_encrypted_cert_source to catch exceptions.ClientCertError as well:

        except (
            exceptions.ClientCertError,
            OSError,
            RuntimeError,
            ValueError,
        ) as caught_exc:
            new_exc = exceptions.MutualTLSChannelError(caught_exc)
            raise new_exc from caught_exc

Comment on lines +657 to +660
try:
_check_spiffe_file_ready(cert_path)
_check_spiffe_file_ready(key_path)
return cert_path, key_path, spiffe_config_path

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Once the SPIFFE credentials are successfully loaded and validated, we should reset _spiffe_poll_timed_out to False. This ensures that if a temporary timeout occurs during startup but eventually recovers, any future certificate rotations can still be polled normally instead of failing immediately.

Note: If you apply this change, you will also need to update the assertion in test_poll_after_timeout_succeeds_once_files_appear (line 2743 in test__mtls_helper.py) to expect _spiffe_poll_timed_out to be False instead of True.

Suggested change
try:
_check_spiffe_file_ready(cert_path)
_check_spiffe_file_ready(key_path)
return cert_path, key_path, spiffe_config_path
try:
_check_spiffe_file_ready(cert_path)
_check_spiffe_file_ready(key_path)
_spiffe_poll_timed_out = False
return cert_path, key_path, spiffe_config_path

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant