feat(k8s): verify the tenant can override the GPU Operator driver (K8S25-02)

[abegnoche]

Closes #220.

What

K8S25-01 only asserts that the GPU Operator namespace exists and has a running pod. K8S25-02 is the substantive half of the requirement: "Provider-default accelerator operators and drivers shall be replaceable or overridable to allow installation of tenant-required operator and driver versions." It had no validation class and no wiring.

K8sGpuOperatorOverrideCheck proves the tenant holds that authority, without mutating the cluster:

1. Locate the operator-managed driver version — clusterpolicies.nvidia.com, falling back to nvidiadrivers.nvidia.com (the per-node-pool CR GPU Operator 24.6+ adds). When neither exists there is no operator-managed driver version to override, and the check fails rather than passing on the absence of evidence. A query error (unreachable API) is reported separately from a genuinely missing configuration.
2. Ask the API server whether the tenant may rewrite it — kubectl auth can-i for patch/update (override in place) and delete/create (swap the configuration out) on the driver configuration, plus patch/update on Deployments and DaemonSets in the operator namespace (installing a different operator version rewrites its own workloads). auth can-i answers "no" with a non-zero exit, so the answer is parsed from stdout and an inconclusive probe fails instead of being read as a pass.
3. Dry-run the override through admission — kubectl patch --dry-run=server is evaluated by RBAC and by every mutating and validating webhook but never persisted. The requested version is then read back off the returned object, which is what distinguishes a provider that permits tenant overrides from one that accepts the write and pins its own version anyway.

The tenant-required version comes from steps.setup.kubernetes.driver_version — the same inventory field K8sDriverVersionCheck asserts is installed — so the check proves the tenant can reach exactly the version the stack is expected to run, rather than a guessed placeholder.

Files

• isvtest/src/isvtest/validations/k8s_gpu_operator.py — the new check, alongside the two existing K8S25-01 checks.
• isvtest/tests/test_k8s_gpu_operator.py — 10 new cases: happy path, NVIDIADriver fallback, missing configuration, query error, RBAC denial on the configuration, read-only operator workloads, inconclusive probe, admission rejection, silent version pin-back, and the no-target-version skip.
• isvctl/configs/suites/k8s.yaml — wires K8S25-02 next to K8S25-01.
• docs/test-plan.yaml / docs/test-plan.adoc — adds the kubernetes label the coverage guardrail requires for a K8S* id, and the regenerated row.

isvtest/src/isvtest/released_tests.json is deliberately untouched: the check ships unreleased and lands in the manifest via a release commit. Run it with ISVTEST_INCLUDE_UNRELEASED=1 until then.

Verification

No Kubernetes cluster is reachable from the authoring environment, so the check was exercised two ways: mocked-kubectl unit tests, and end-to-end through the real orchestrator against a throwaway kubectl-compatible CLI selected with KUBECTL. The latter drives the actual step output and Jinja2 wiring — the fake cluster's node labels report the tenant-required driver 580.82.07 while its ClusterPolicy carries the provider default 550.54.15, so the run proves the wiring end to end and not just the class.

$ KUBECTL=<fake> FAKE_MODE=<scenario> ISVTEST_INCLUDE_UNRELEASED=1 \
    uv run isvctl test run -f isvctl/configs/suites/k8s.yaml -- -k K8sGpuOperatorOverrideCheck

Cluster scenario	Result
Tenant may rewrite the operator stack	PASSED — clusterpolicies.nvidia.com/cluster-policy spec.driver.version accepts '580.82.07' (currently '550.54.15')
Admission accepts the write, then restores the provider version	FAILED — Admission kept the provider-default driver version: requested '580.82.07' ..., admitted object reports '550.54.15'
Tenant may not delete the driver configuration	FAILED — Tenant is not authorized to replace the provider-default GPU Operator: cannot delete clusterpolicies.nvidia.com
No operator-managed driver version exists	FAILED — No GPU Operator driver configuration found (...); the driver version is not operator-managed, so a tenant override cannot be proven

• make test — 1717 isvtest + 124 scripts tests pass
• make demo-test — all my-isv suites pass
• make lint, uvx pre-commit run -a — clean, including the plan-coverage and validate-suites guardrails

Scope note

docs/references/GPU-Operator-Self-Certify-Integration-Solution-for-AI-Cloud-Validation-Suite-PRD.md defines GPUOP-03 "Driver Update", owned by the GPU Operator team per ENT-REQ-000. This check is deliberately scoped to whether the tenant may replace or override — install ownership, authorization, and admission — and leaves "does a driver update work end to end" to that track.

  

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}