Repository navigation
fix(auth): send IAM Credentials calls over mTLS for certificate-bound agent identity tokens - #15027
macastelaz wants to merge 8 commits into
Conversation
…thout a body IamUtils.sign() and IamUtils.getIdToken() parse the JSON body of every 4xx response. HttpURLConnection discards the body of a 401 response to a request sent in streaming mode (NetHttpTransport sends every POST with content in streaming mode), so the response has no content and parseAs() throws a NullPointerException. Callers such as ImpersonatedCredentials.sign(), ImpersonatedCredentials.idTokenWithAudience() and ComputeEngineCredentials.sign() then surface a bare NPE instead of the 401. Use the HTTP status message when the error response has no content, so these calls fail with an IOException (or SigningException) that includes the status code.
- Fall back to the status message when the 4xx body is empty but not null (the JSON parser throws IllegalArgumentException: no JSON input found). - Treat an empty reason phrase like a missing one. - Tests: empty-string body, missing reason phrase, sign() with a JSON body, and an IAM-shaped 401 (Bearer challenge) for getIdToken that asserts no credential refresh.
… agent identity tokens On GKE and Cloud Run workloads with an agent identity, ComputeEngineCredentials requests access tokens that are bound to the workload certificate. A bound token is accepted only over mTLS, presenting that certificate, to the mTLS endpoint. ImpersonatedCredentials (generateAccessToken, signBlob, generateIdToken) and ComputeEngineCredentials.sign() sent the token to the plain iamcredentials.googleapis.com host without a client certificate, so they failed with 401. - AgentIdentityUtils.CertInfo now also holds a KeyStore built from the same certificate chain and private key that were loaded and verified together. - ComputeEngineCredentials returns a package-private BoundAccessToken that records the CertInfo the token was bound to. It serializes as a plain AccessToken. - IamUtils.resolveCallTarget() sends calls authenticated with a bound token to iamcredentials.mtls.googleapis.com over an MtlsHttpTransportFactory with that KeyStore, authenticated with exactly that token, so the presented certificate always matches the token. Calls with a custom IAM endpoint, a custom transport factory, a non-default universe, or an unbound token are unchanged. - ImpersonatedCredentials and ComputeEngineCredentials.sign() use it.
There was a problem hiding this comment.
Code Review
This pull request introduces support for certificate-bound access tokens over mTLS. It adds a new BoundAccessToken class that retains the CertInfo (including an in-memory KeyStore with the private key and certificate chain) to ensure the correct certificate is presented during mTLS calls. ComputeEngineCredentials, ImpersonatedCredentials, and IamUtils are updated to resolve the appropriate call target (mTLS endpoint and transport) when a bound token is used. Additionally, error handling for 4xx responses in IamUtils is improved to handle empty or null response bodies gracefully. The feedback suggests initializing pinnedTokenCredentials with the access token directly in the builder to avoid an unnecessary refresh call on first use.
- Building the mTLS key store can no longer fail token binding: unparsable and duplicate extra certificates are skipped (as in MtlsKeyStoreUtils), PKCS12 is used if JKS is unavailable, and any remaining failure leaves the key store unset (IAM calls then use the default endpoint). - BoundAccessToken serializes as an expired AccessToken, so deserialized credentials mint a new bound token on first use instead of sending a bound token without its certificate. - sign() and idTokenWithAudience() use the mTLS endpoint even when iamEndpointOverride is set; the override only ever applied to generateAccessToken. - Look up the universe domain before the source refresh in ImpersonatedCredentials.refreshAccessToken so its errors keep their type. - Tests: call-site routing for ImpersonatedCredentials (generateAccessToken, signBlob, generateIdToken, endpoint override) and ComputeEngineCredentials.sign through a test-only mTLS transport factory provider; key store contents; chain with extra certificates; credential serialization round trip.
Avoids a refresh on first use. Adds a test that the pinned credentials start with the bound token and keep it after a refresh.
…bound tokens - Populate PinnedTokenCredentials via an in-memory refresh() after construction instead of Builder.setAccessToken(token), so GoogleCredentials attaches the x-goog-user-project header via getAdditionalHeaders() while still starting with the bound token cached. - Disable HttpCredentialsAdapter's 401 retry handler when authenticating IAM generateAccessToken or generateIdToken calls with PinnedTokenCredentials, preventing 10 futile retries with the same pinned token on a 401. - Store usesMtls explicitly on IamCallTarget, reuse OAuth2Utils.PEM_CERT_PATTERN in AgentIdentityUtils.buildKeyStore, and match MtlsKeyStoreUtils by skipping null returns or RuntimeExceptions from CertificateFactory.generateCertificate on extra chain blocks. - Expand unit tests in IamMtlsRoutingTest, IamUtilsTest, and ComputeEngineCredentialsTest to cover quotaProjectId preservation, 401 fail-fast behavior across all 4 call sites, DefaultHttpTransportFactory endpoint-override routing, idTokenWithAudience with iamEndpointOverride, cold-start token refresh, certificate rotation isolation, and multi-certificate chain retention.
|
@lqiu96 @nbayati one known limitation I'd like your input on before this merges: The mTLS transport is created per call. On the bound-token path, IamUtils.resolveCallTarget() returns a new MtlsHttpTransportFactory, and each sign(), idTokenWithAudience() or generateAccessToken call runs create() on it. That builds a new NetHttpTransport with a new SSLContext each time: MtlsHttpTransportFactory.create() → NetHttpTransport.Builder.trustCertificates(...) → SslUtils.getTlsSslContext(...) / sslContext.getSocketFactory(). NetHttpTransport sets that socket factory on every HttpsURLConnection it opens (setSSLSocketFactory). As a result: No TLS session resumption. The JDK keeps the client session cache per SSLContext (SSLContextImpl.clientCache, created in its constructor), so every call does a full mTLS handshake. No connection reuse, and idle connections pile up under load. The JDK's HTTPS keep-alive cache is keyed by protocol, host, port and the socket factory instance, compared by identity:
When a response is fully read and closed, KeepAliveStream.close() → HttpClient.finished() returns the connection to the cache. Because every call has a new socket factory, each idle connection goes under its own key and is never reused. It stays open until the keep-alive timeout, which is 5 seconds by default when the server sends no Keep-Alive timeout (KeepAliveCache.put). So a loop making N calls per second holds roughly 5×N idle connections to iamcredentials.mtls.googleapis.com at any time. (Sources: JDK 17u sun.net.www.protocol.https.HttpsClient, sun.net.www.http.KeepAliveCache / KeepAliveStream / HttpClient, sun.security.ssl.SSLContextImpl; google-http-java-client 2.2.0 NetHttpTransport.) Context: It isn't a regression: on agent identities these calls failed before this PR. Options:
|
Fixes #15014.
Summary
On GKE and Cloud Run workloads with an agent identity,
ComputeEngineCredentials(the credential ADC returns there) requests access tokens bound to the workload certificate. IAM accepts a bound token only over mTLS, presenting that certificate, to the mTLS endpoint.ImpersonatedCredentials(generateAccessToken,signBlob,generateIdToken) andComputeEngineCredentials.sign()sent the token to plainiamcredentials.googleapis.comwithout a client certificate, so they failed with 401. #15015 fixed the related error-body parsing bug that hid this as an NPE forsign()andidTokenWithAudience().With this change, those calls go to
iamcredentials.mtls.googleapis.comand present exactly the certificate the token was bound to.Design
One verified certificate/key snapshot.
AgentIdentityUtilsalready loads the certificate and private key together and verifies that they match, with bounded retries and a cache.CertInfonow also holds aKeyStorebuilt from those same verified objects (leaf, additional chain certificates, and key), so there's no second read of files that may be mid-rotation.MtlsKeyStoreUtils, it skips unparsable and duplicate additional certificates, and it uses PKCS12 if JKS isn't available. If it still fails, the key store is left unset, and IAM calls use the default endpoint as before.The token remembers its certificate.
ComputeEngineCredentials.refreshAccessToken()returns a package-privateBoundAccessToken(anAccessTokensubclass) that carries theCertInfoit was bound to.writeReplace) as a plainAccessTokenthat is already expired. Serialized credentials stay readable by older versions, and deserialized credentials mint a new bound token on first use instead of sending a bound token without its certificate.BoundAccessToken. For them, a round-tripped credential refreshes once on first use, and its token no longer compares equal to the original (the expiry differs).IAM call routing.
IamUtils.resolveCallTarget()is used byImpersonatedCredentials.refreshAccessToken/sign/idTokenWithAudienceandComputeEngineCredentials.sign. When the authenticating credential is aComputeEngineCredentialswhose current token is aBoundAccessToken, the call:iamcredentials.mtls.<universe>;MtlsHttpTransportFactory(certInfo.getKeyStore());PinnedTokenCredentialsinstance refreshed in-memory once (preservingx-goog-user-projectfromsourceCredentialswhile starting withtokenready and preventing a concurrent refresh on the source credential from attaching a token bound to a different certificate), withHttpCredentialsAdapter's 401 retry disabled on the pinned token so a rejected bound token isn't resent 10 times.The token and the presented certificate therefore always match, including across certificate rotation.
iamEndpointOverride:generateAccessTokenkeeps using the override, with no mTLS switch.signBlobandgenerateIdTokennever used the override (it's a fullgenerateAccessTokenURL), so they go to the mTLS endpoint whether or not an override is set.Unchanged cases: a custom
HttpTransportFactory, a non-default universe, an unbound token (opted out, or not an agent identity), and Workload Identity Federation sources (their existingIdentityPoolCredentialsmTLS path).Testing
Unit tests:
IamMtlsRoutingTest:ImpersonatedCredentials.refreshAccessToken,sign,idTokenWithAudience(with and without an endpoint override, using aDefaultHttpTransportFactorytest subclass) andComputeEngineCredentials.sign, each with a bound source, send one request toiamcredentials.mtls.googleapis.comwith the bound token andx-goog-user-projectheader over a transport built from the token's key store, and fail fast without retrying the pinned token on 401.IamUtilsTest: routing for a bound token,x-goog-user-projectheader preservation on sync and asyncgetRequestMetadata, pinned-token credential seeding and refresh retention, cold-startrefreshIfExpired()whengetAccessToken()isnull, certificate rotation (Cert A -> Cert B) and mid-flight refresh isolation, an unbound token, aCertInfowithout a key store, a non-default universe, non-Compute-Engine credentials, a custom endpoint, and a custom transport factory, plus the realMtlsHttpTransportFactorywiring.ComputeEngineCredentialsTest:BoundAccessTokenwhose key store holds the verified key, retrievable with the empty password the transport uses, and the leaf certificate;BoundAccessTokenserializes as an expiredAccessToken;Full
oauth2_httpsuite (JDK 21): 1363 tests, 0 failures, 2 skipped.Live, GKE agent-identity pod (patched build):
ImpersonatedCredentials.refreshAccessToken()sign()idTokenWithAudience()ComputeEngineCredentials.sign()<project>.svc.id.goog"The
ComputeEngineCredentials.sign()400 is an existing limitation: GKE's metadata server reports the workload identity pool as the account. With the opt-out, behavior is unchanged.Live, Cloud Run job with an agent identity (patched build): all four calls reach
iamcredentials.mtls.googleapis.comand get 403 (permission) rather than 401, because the job's principal has no grants on the target. So the bound token is accepted, andComputeEngineCredentials.sign()gets past the account check on Cloud Run.