E2E test: vMCP cross-pod restore with upstreamInject outgoing auth

[tgrunnagle]

Context

PR #5650 fixed two bugs in the cross-pod session restore path for Redis-backed multi-replica vMCP deployments:

1. Nil identity from RestoreSession — RestoreSession was fabricating a partial *auth.Identity with empty Token/UpstreamTokens; it now passes nil.
2. Context propagation — loadSession now threads the incoming request's context (carrying a live *auth.Identity with populated UpstreamTokens) through context.WithoutCancel to RestoreSession, so backend Initialize handshakes during restore are authenticated.

The cross-pod restore E2E test in test/e2e/thv-operator/virtualmcp/virtualmcp_redis_session_test.go exercises the restore path with an anonymous vMCP (no outgoing auth), but does not verify that an authenticated session with upstreamInject outgoing auth continues to work after restore. See the TODO(#5336) comment above the "Should allow a session established on pod A to be reconstructed on pod B" test.

The code fix is complete. The E2E test is blocked only by the lack of an in-cluster OIDC provider in the test environment.

Goal

1. Add an in-cluster OIDC provider deployment helper to the E2E test infrastructure.
2. Add a new Ginkgo It block (or a new Context) in virtualmcp_redis_session_test.go that verifies an authenticated vMCP session — one backed by upstreamInject outgoing auth — is fully functional after cross-pod Redis restore.

What already exists

The helpers in test/e2e/thv-operator/virtualmcp/helpers.go already provide:

• DeployParameterizedOIDCServer — deploys a Python Flask OIDC server in-cluster. It issues RSA-signed JWTs with caller-controlled subject and exposes a JWKS endpoint. This may be sufficient as the upstream OIDC provider if it supports the discovery URL and authorization/token endpoints needed by the embedded auth server.
• DeployMockOAuth2Server / GetMockOAuth2Stats — already used by the tokenExchange E2E tests.
• InstrumentedBackendScript — a Python Flask MCP backend that logs every inbound Authorization: Bearer token to a /stats endpoint. This is the correct backend for validating that the upstream token was injected on backend requests.
• portForwardToPod, countReadyPods, GetVirtualMCPServerPods — already used by the cross-pod restore tests.
• Redis deployment/cleanup helpers — deployRedis, cleanupRedis, readRedisSessionBackendIDs.

What needs to be added

Option A — reuse DeployParameterizedOIDCServer

If the existing Python Flask server can act as the upstream OIDC provider for the embedded auth server (serving a proper /.well-known/openid-configuration with authorization_endpoint, token_endpoint, and jwks_uri), no new deployment helper is needed. The embedded auth server would be configured to use it as the upstream IDP.

Option B — deploy Dex

Dex is a CNCF-project OIDC provider that is easy to deploy in-cluster via a single Deployment + ConfigMap. It supports static passwords, which avoids browser-based consent flows in tests. Add a deployDex(name, namespace string) / cleanupDex(name string) helper pair alongside the existing deployRedis/cleanupRedis pair.

The choice between A and B is left to the implementor; B is recommended if the parameterized server lacks the endpoints the embedded AS requires.

Test scenario

The test should run inside the existing Redis-session-sharing Describe block (or a new one) and follow this structure:

1. Setup (BeforeAll):

   • Deploy Redis (as in the existing cross-pod restore tests).
   • Deploy the OIDC provider.
   • Create an MCPExternalAuthConfig with type: upstreamInject and upstreamInject.providerName pointing to the upstream provider configured in the embedded AS.
   • Deploy a backend MCPServer using InstrumentedBackendScript with ExternalAuthConfigRef pointing to the MCPExternalAuthConfig above.
   • Create a VirtualMCPServer with:
     • replicas: 2
     • Redis session storage
     • SessionAffinity: None
     • Incoming auth: OIDC, using the in-cluster provider as issuer
     • Outgoing auth: source: discovered (picks up the backend's ExternalAuthConfigRef)
     • Embedded auth server (AuthServerConfig) configured with the OIDC provider as an upstream provider under the same providerName.
   • Wait for 2 ready pods.

2. Test (It "Should inject upstream tokens after cross-pod restore"):

   • Port-forward to pod A and pod B independently.
   • Obtain an authenticated token from the OIDC provider (or the embedded AS), wire it into an MCP client.
   • Initialize the MCP session on pod A. Verify a tool call succeeds and the instrumented backend's /stats shows a Bearer token on the request.
   • Connect to pod B with the same session ID (triggering RestoreSession from Redis).
   • Make the same tool call through pod B. Assert it succeeds — not a 401.
   • Query the instrumented backend's /stats endpoint (via in-cluster curl pod as in the existing auth tests). Assert the backend received a second request with a Bearer token, confirming upstreamInject fired correctly after restore.

Acceptance criteria

• A new in-cluster OIDC provider helper (or confirmation that the existing parameterized server is sufficient) is present in helpers.go.
• The new It block compiles and passes in the E2E suite (task -d cmd/thv-operator thv-operator-e2e-test).
• The test fails if context.WithoutCancel in loadSession is reverted to context.Background() (i.e., it actually exercises the fix from Stop fabricating partial identity in RestoreSession #5650).
• The test fails if upstreamInject is misconfigured or the upstream token is not injected (i.e., the instrumented backend /stats check is a meaningful assertion, not just "the request succeeded").
• Remove or update the TODO(#5336) comment in virtualmcp_redis_session_test.go once the test is added.

Related

• PR Stop fabricating partial identity in RestoreSession #5650 — fixes the underlying bugs
• Issue [vMCP] RestoreSession reconstructs a partial auth.Identity (Subject only), breaks cross-pod failover with upstream-auth strategies #5336 — original bug report; this issue tracks the E2E validation gap
• test/e2e/thv-operator/virtualmcp/virtualmcp_external_auth_test.go — reference for how upstreamInject, tokenExchange, and instrumented backend tests are structured
• test/e2e/thv-operator/virtualmcp/virtualmcp_authserver_config_test.go — reference for embedded auth server config