feat: support the Vault (OpenBao) credential-driver e2e lane on OpenShift

[jgarciao]

Follow-up of #3183 (feat(e2e): make e2e:kubernetes work transparently on OpenShift), which brought the general OpenShift e2e path up; this extends that work to the Vault (OpenBao) credential-driver lane.

User Story

As an OpenShell contributor validating the Vault credential driver, I want mise run e2e:kubernetes with the Vault credential driver to pass on OpenShift/ROSA clusters, so that I can verify provider credential storage on OpenShift the same way I already can on kind/k3d.

Problem Statement

Running the Kubernetes e2e suite with the Vault (OpenBao) credential driver on OpenShift fails in two independent ways:

1. OpenBao fixture pod never schedules. The OpenBao Helm chart pins the pod's security context (a fixed runAsUser/fsGroup and a deprecated seccomp annotation). OpenShift's restricted-v2 SCC rejects those pinned fields, so the StatefulSet controller never creates the pod and the readiness wait in the e2e driver fails with error: no matching resources found.

2. Provider-creating tests are denied by OpenBao auth. After the pod is fixed, tests that create providers fail with Vault Kubernetes auth denied the configured role — an HTTP 403 from auth/kubernetes/login. The OpenBao Kubernetes auth method and login role were provisioned only inside a single feature-gated e2e test (e2e/rust/tests/credential_drivers.rs).

   This single coupling surfaces the failure two ways:

   • Deterministically — when that test is not compiled/scoped in (e.g. a full-suite run that forces the Vault backend without the credential-driver feature), the auth method is never enabled and every provider create is rejected with HTTP 403.
   • Order-dependently — even when the test is included, correctness depends on it running and finishing before every other provider-creating test. That ordering is not guaranteed: under cargo nextest (parallel, isolated test processes) execution order has been observed to be decisive, with provider/workspace tests running — and failing — before the login role was created. A test that races ahead of provisioning fails with HTTP 403 (before the auth method is enabled) or HTTP 400 (after auth-enable but before the role write).

Impact / Why This Matters

• Consequence of current behavior: the Vault credential-driver e2e lane is unusable on OpenShift. Contributors cannot validate credential storage on the platform that OpenShell productization targets.
• Current workaround: run credential-driver validation only on kind/k3d. This is insufficient because the failures live precisely in OpenShift-specific behavior — SCC/PodSecurity admission, and (on ROSA) a custom OIDC service-account-token issuer — that kind/k3d never exercise. A lane that only runs on kind gives false confidence for OpenShift.
• Latent, not OpenShift-only: problem (2) is a design gap in the e2e fixture setup. Coupling vault-auth provisioning to one test means the full suite can never reliably run against a Vault-backed gateway on any cluster; the same footgun bites anyone forcing the Vault driver for a broader run, and it makes results order-dependent (and flaky) under parallel runners such as cargo nextest.

Proposed Design

• Fixture scheduling: when the e2e driver detects an OpenShift cluster, deploy the OpenBao fixture in the chart's OpenShift-compatible mode so the pod drops its pinned UID/fsGroup/seccomp fields and inherits a namespace-assigned, SCC-compliant security context — no manual SCC grant required. OpenShift detection must run before the credential-driver fixtures are deployed.
• Auth provisioning: provision OpenBao's KV store, Kubernetes auth method, storage policy, and the gateway login role (bound to the gateway ServiceAccount in the gateway namespace) as part of fixture setup, so it is available to every test that uses the Vault backend — not only the feature-gated credential-driver test.
• Externally observable behavior: a Vault-backed gateway works for the whole e2e suite on both kind/k3d and OpenShift; the dedicated credential-driver test remains green; the default (non-vault) lanes are untouched.

Internal implementation choices (exact chart flag, where the provisioning code lives, bao invocation mechanics) are left open.

Acceptance Criteria

• mise run e2e:kubernetes with OPENSHELL_E2E_CREDENTIAL_DRIVERS=1 and OPENSHELL_E2E_CREDENTIAL_DRIVER=vault passes on an OpenShift/ROSA cluster.
• The OpenBao fixture pod schedules under restricted-v2 with no manual SCC grant and no cluster-admin action.
• Provider-creating tests authenticate to OpenBao (no HTTP 403), including workspace/provider tests that are not part of the credential-driver test file.
• The vanilla-Kubernetes (kind/k3d) path is unchanged and still green.
• No secrets are committed; the fixture remains dev-mode OpenBao only.

Alternatives Considered

• Grant the OpenBao ServiceAccount a custom SCC (e.g., anyuid) to keep the chart's pinned UID/fsGroup. Rejected: verified on ROSA that the pod still failed to schedule (the fsGroup/seccomp rejections remained and anyuid was not even selected by admission), and it requires a cluster-admin SCC grant. The chart's OpenShift mode is cleaner and grant-free.
• Keep vault-auth provisioning in the test and require every Vault run to scope to it (OPENSHELL_E2E_KUBE_TEST=credential_drivers). Rejected: this prevents running the full suite against a Vault backend and preserves the latent gap for anyone who forces the Vault driver without that exact scoping.

Agent Investigation

Root cause and fix were reproduced and confirmed on a live ROSA cluster: missing auth method → HTTP 403 (permission denied, the observed failure); missing role → HTTP 400; fully provisioned auth → HTTP 200 login, even with ROSA's custom OIDC-issuer SA-token audience. A full mise run e2e:kubernetes run with the Vault driver then passed end-to-end with zero test failures.

An agent codebase review confirmed the two failure modes map to real pre-existing code and that the fixture/provisioning logic lives in the e2e bash driver (e2e/with-kube-gateway.sh), not the Rust harness crate:

• The 403 → error-string mapping is exact in crates/openshell-driver-vault/src/lib.rs (FORBIDDEN → Vault Kubernetes auth denied the configured role).
• Before the fix, configure_vault_storage() in e2e/rust/tests/credential_drivers.rs enabled the KV store, Kubernetes auth method, storage policy, and openshell-gateway login role only inside one vault-gated test — matching the described coupling.
• The exact OpenBao chart pinned values are external-chart behavior and unverifiable in-repo; the proposed fix defers entirely to the chart's OpenShift mode rather than depending on specific values.

The order-dependent pathway under cargo nextest was observed on a separate branch (login role created after some tests had already run and failed); it was not independently re-reproduced here, but it is consistent with the same single-test coupling and is likewise resolved by fixture-side provisioning.