Skip to content

fix(auth): send IAM Credentials calls over mTLS for certificate-bound agent identity tokens - #15027

Open
macastelaz wants to merge 8 commits into
googleapis:mainfrom
macastelaz:impersonated-agent-identity-mtls
Open

macastelaz wants to merge 8 commits into
googleapis:mainfrom
macastelaz:impersonated-agent-identity-mtls

Conversation

@macastelaz

@macastelaz macastelaz commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #15014.

Summary

On GKE and Cloud Run workloads with an agent identity, ComputeEngineCredentials (the credential ADC returns there) requests access tokens bound to the workload certificate. IAM accepts a bound token only over mTLS, presenting that certificate, to the mTLS endpoint.

ImpersonatedCredentials (generateAccessToken, signBlob, generateIdToken) and ComputeEngineCredentials.sign() sent the token to plain iamcredentials.googleapis.com without a client certificate, so they failed with 401. #15015 fixed the related error-body parsing bug that hid this as an NPE for sign() and idTokenWithAudience().

With this change, those calls go to iamcredentials.mtls.googleapis.com and present exactly the certificate the token was bound to.

Design

  • One verified certificate/key snapshot. AgentIdentityUtils already loads the certificate and private key together and verifies that they match, with bounded retries and a cache. CertInfo now also holds a KeyStore built from those same verified objects (leaf, additional chain certificates, and key), so there's no second read of files that may be mid-rotation.

    • Building the key store can't fail token binding. Like MtlsKeyStoreUtils, it skips unparsable and duplicate additional certificates, and it uses PKCS12 if JKS isn't available. If it still fails, the key store is left unset, and IAM calls use the default endpoint as before.
  • The token remembers its certificate. ComputeEngineCredentials.refreshAccessToken() returns a package-private BoundAccessToken (an AccessToken subclass) that carries the CertInfo it was bound to.

    • It serializes (writeReplace) as a plain AccessToken that is already expired. Serialized credentials stay readable by older versions, and deserialized credentials mint a new bound token on first use instead of sending a bound token without its certificate.
    • Only agent-identity workloads with binding on ever hold a BoundAccessToken. For them, a round-tripped credential refreshes once on first use, and its token no longer compares equal to the original (the expiry differs).
  • IAM call routing. IamUtils.resolveCallTarget() is used by ImpersonatedCredentials.refreshAccessToken/sign/idTokenWithAudience and ComputeEngineCredentials.sign. When the authenticating credential is a ComputeEngineCredentials whose current token is a BoundAccessToken, the call:

    • goes to iamcredentials.mtls.<universe>;
    • over MtlsHttpTransportFactory(certInfo.getKeyStore());
    • authenticated with a PinnedTokenCredentials instance refreshed in-memory once (preserving x-goog-user-project from sourceCredentials while starting with token ready and preventing a concurrent refresh on the source credential from attaching a token bound to a different certificate), with HttpCredentialsAdapter's 401 retry disabled on the pinned token so a rejected bound token isn't resent 10 times.

    The token and the presented certificate therefore always match, including across certificate rotation.

  • iamEndpointOverride:

    • generateAccessToken keeps using the override, with no mTLS switch.
    • signBlob and generateIdToken never used the override (it's a full generateAccessToken URL), so they go to the mTLS endpoint whether or not an override is set.
  • Unchanged cases: a custom HttpTransportFactory, a non-default universe, an unbound token (opted out, or not an agent identity), and Workload Identity Federation sources (their existing IdentityPoolCredentials mTLS path).

Testing

  • Unit tests:

    • IamMtlsRoutingTest: ImpersonatedCredentials.refreshAccessToken, sign, idTokenWithAudience (with and without an endpoint override, using a DefaultHttpTransportFactory test subclass) and ComputeEngineCredentials.sign, each with a bound source, send one request to iamcredentials.mtls.googleapis.com with the bound token and x-goog-user-project header over a transport built from the token's key store, and fail fast without retrying the pinned token on 401.
    • IamUtilsTest: routing for a bound token, x-goog-user-project header preservation on sync and async getRequestMetadata, pinned-token credential seeding and refresh retention, cold-start refreshIfExpired() when getAccessToken() is null, certificate rotation (Cert A -> Cert B) and mid-flight refresh isolation, an unbound token, a CertInfo without a key store, a non-default universe, non-Compute-Engine credentials, a custom endpoint, and a custom transport factory, plus the real MtlsHttpTransportFactory wiring.
    • ComputeEngineCredentialsTest:
      • a bound refresh returns a BoundAccessToken whose key store holds the verified key, retrievable with the empty password the transport uses, and the leaf certificate;
      • valid extra certificates in the chain are retained while duplicate, empty, or unparsable PEM blocks are skipped without breaking binding;
      • the opt-out returns a plain token;
      • BoundAccessToken serializes as an expired AccessToken;
      • deserialized credentials mint a new bound token on first use.
  • Full oauth2_http suite (JDK 21): 1363 tests, 0 failures, 2 skipped.

  • Live, GKE agent-identity pod (patched build):

    Call 1.57.0 Patched
    ImpersonatedCredentials.refreshAccessToken() 401 OK
    sign() NPE OK
    idTokenWithAudience() NPE OK
    ComputeEngineCredentials.sign() NPE 400 "Invalid form of account ID <project>.svc.id.goog"

    The ComputeEngineCredentials.sign() 400 is an existing limitation: GKE's metadata server reports the workload identity pool as the account. With the opt-out, behavior is unchanged.

  • Live, Cloud Run job with an agent identity (patched build): all four calls reach iamcredentials.mtls.googleapis.com and get 403 (permission) rather than 401, because the job's principal has no grants on the target. So the bound token is accepted, and ComputeEngineCredentials.sign() gets past the account check on Cloud Run.

…thout a body

IamUtils.sign() and IamUtils.getIdToken() parse the JSON body of every 4xx
response. HttpURLConnection discards the body of a 401 response to a request
sent in streaming mode (NetHttpTransport sends every POST with content in
streaming mode), so the response has no content and parseAs() throws a
NullPointerException. Callers such as ImpersonatedCredentials.sign(),
ImpersonatedCredentials.idTokenWithAudience() and ComputeEngineCredentials.sign()
then surface a bare NPE instead of the 401.

Use the HTTP status message when the error response has no content, so these
calls fail with an IOException (or SigningException) that includes the status
code.
- Fall back to the status message when the 4xx body is empty but not null
  (the JSON parser throws IllegalArgumentException: no JSON input found).
- Treat an empty reason phrase like a missing one.
- Tests: empty-string body, missing reason phrase, sign() with a JSON body,
  and an IAM-shaped 401 (Bearer challenge) for getIdToken that asserts no
  credential refresh.
… agent identity tokens

On GKE and Cloud Run workloads with an agent identity, ComputeEngineCredentials
requests access tokens that are bound to the workload certificate. A bound token
is accepted only over mTLS, presenting that certificate, to the mTLS endpoint.
ImpersonatedCredentials (generateAccessToken, signBlob, generateIdToken) and
ComputeEngineCredentials.sign() sent the token to the plain
iamcredentials.googleapis.com host without a client certificate, so they failed
with 401.

- AgentIdentityUtils.CertInfo now also holds a KeyStore built from the same
  certificate chain and private key that were loaded and verified together.
- ComputeEngineCredentials returns a package-private BoundAccessToken that
  records the CertInfo the token was bound to. It serializes as a plain
  AccessToken.
- IamUtils.resolveCallTarget() sends calls authenticated with a bound token to
  iamcredentials.mtls.googleapis.com over an MtlsHttpTransportFactory with that
  KeyStore, authenticated with exactly that token, so the presented certificate
  always matches the token. Calls with a custom IAM endpoint, a custom transport
  factory, a non-default universe, or an unbound token are unchanged.
- ImpersonatedCredentials and ComputeEngineCredentials.sign() use it.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for certificate-bound access tokens over mTLS. It adds a new BoundAccessToken class that retains the CertInfo (including an in-memory KeyStore with the private key and certificate chain) to ensure the correct certificate is presented during mTLS calls. ComputeEngineCredentials, ImpersonatedCredentials, and IamUtils are updated to resolve the appropriate call target (mTLS endpoint and transport) when a bound token is used. Additionally, error handling for 4xx responses in IamUtils is improved to handle empty or null response bodies gracefully. The feedback suggests initializing pinnedTokenCredentials with the access token directly in the builder to avoid an unnecessary refresh call on first use.

- Building the mTLS key store can no longer fail token binding: unparsable and
  duplicate extra certificates are skipped (as in MtlsKeyStoreUtils), PKCS12 is
  used if JKS is unavailable, and any remaining failure leaves the key store
  unset (IAM calls then use the default endpoint).
- BoundAccessToken serializes as an expired AccessToken, so deserialized
  credentials mint a new bound token on first use instead of sending a bound
  token without its certificate.
- sign() and idTokenWithAudience() use the mTLS endpoint even when
  iamEndpointOverride is set; the override only ever applied to
  generateAccessToken.
- Look up the universe domain before the source refresh in
  ImpersonatedCredentials.refreshAccessToken so its errors keep their type.
- Tests: call-site routing for ImpersonatedCredentials (generateAccessToken,
  signBlob, generateIdToken, endpoint override) and ComputeEngineCredentials.sign
  through a test-only mTLS transport factory provider; key store contents; chain
  with extra certificates; credential serialization round trip.
Avoids a refresh on first use. Adds a test that the pinned credentials start
with the bound token and keep it after a refresh.
…bound tokens

- Populate PinnedTokenCredentials via an in-memory refresh() after construction
  instead of Builder.setAccessToken(token), so GoogleCredentials attaches the
  x-goog-user-project header via getAdditionalHeaders() while still starting
  with the bound token cached.
- Disable HttpCredentialsAdapter's 401 retry handler when authenticating IAM
  generateAccessToken or generateIdToken calls with PinnedTokenCredentials,
  preventing 10 futile retries with the same pinned token on a 401.
- Store usesMtls explicitly on IamCallTarget, reuse OAuth2Utils.PEM_CERT_PATTERN
  in AgentIdentityUtils.buildKeyStore, and match MtlsKeyStoreUtils by skipping
  null returns or RuntimeExceptions from CertificateFactory.generateCertificate
  on extra chain blocks.
- Expand unit tests in IamMtlsRoutingTest, IamUtilsTest, and
  ComputeEngineCredentialsTest to cover quotaProjectId preservation, 401
  fail-fast behavior across all 4 call sites, DefaultHttpTransportFactory
  endpoint-override routing, idTokenWithAudience with iamEndpointOverride,
  cold-start token refresh, certificate rotation isolation, and
  multi-certificate chain retention.
@macastelaz
macastelaz marked this pull request as ready for review November 21, 2026 03:03
@macastelaz
macastelaz requested review from a team as code owners November 21, 2026 03:03
@macastelaz
macastelaz requested review from lqiu96 and nbayati November 21, 2026 03:03
@macastelaz

Copy link
Copy Markdown
Contributor Author

@lqiu96 @nbayati one known limitation I'd like your input on before this merges:

The mTLS transport is created per call. On the bound-token path, IamUtils.resolveCallTarget() returns a new MtlsHttpTransportFactory, and each sign(), idTokenWithAudience() or generateAccessToken call runs create() on it. That builds a new NetHttpTransport with a new SSLContext each time: MtlsHttpTransportFactory.create() → NetHttpTransport.Builder.trustCertificates(...) → SslUtils.getTlsSslContext(...) / sslContext.getSocketFactory(). NetHttpTransport sets that socket factory on every HttpsURLConnection it opens (setSSLSocketFactory). As a result:

No TLS session resumption. The JDK keeps the client session cache per SSLContext (SSLContextImpl.clientCache, created in its constructor), so every call does a full mTLS handshake.

No connection reuse, and idle connections pile up under load. The JDK's HTTPS keep-alive cache is keyed by protocol, host, port and the socket factory instance, compared by identity:

  • HttpsClient uses kac.get(url, sf) and kac.put(url, sslSocketFactory, this);
  • KeepAliveKey.equals checks this.obj == kae.obj.

When a response is fully read and closed, KeepAliveStream.close() → HttpClient.finished() returns the connection to the cache. Because every call has a new socket factory, each idle connection goes under its own key and is never reused. It stays open until the keep-alive timeout, which is 5 seconds by default when the server sends no Keep-Alive timeout (KeepAliveCache.put). So a loop making N calls per second holds roughly 5×N idle connections to iamcredentials.mtls.googleapis.com at any time.

(Sources: JDK 17u sun.net.www.protocol.https.HttpsClient, sun.net.www.http.KeepAliveCache / KeepAliveStream / HttpClient, sun.security.ssl.SSLContextImpl; google-http-java-client 2.2.0 NetHttpTransport.)

Context:

It isn't a regression: on agent identities these calls failed before this PR.
generateAccessToken runs about once per token lifetime, so the cost there is negligible.
It matters for bulk sign() use, such as generating many signed URLs in a loop.

Options:

  1. Leave it out of this PR and track it in a follow-up issue.
  2. Build the transport once per certificate snapshot: keep one NetHttpTransport with the CertInfo (created when the certificate and key are loaded, so it's replaced on rotation) and reuse it for every IAM call bound to that certificate. That restores session resumption and keep-alive reuse. It's small, but it adds state to CertInfo.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IAM Credentials calls fail from GKE/Cloud Run agent identities with certificate-bound tokens (ImpersonatedCredentials 401, sign/idTokenWithAudience NPE)

1 participant