refactor: trim GPU provisioning critical path (skip redundant pull, async cleanup, defer DCGM)

[ganeshkumarashok]

Summary

Three small, independent, low-risk optimizations that remove redundant waits from the GPU node provisioning critical path on Ubuntu. None of these touch the dominant cost — the per-node DKMS driver compile + cp -a /opt/gpu inside configGPUDrivers, which is ~95% of GPU CSE time. The intent here is hygiene and tail-latency protection, not the headline speedup. The large win (moving the driver build off the boot path / pre-laying the driver tree on the VHD) is tracked separately.

Expected impact: low single-digit seconds off the median for a default GPU node, plus meaningful p99/throttling-tail protection. The DCGM change (#3) only affects the managed-GPU experience path.

1. Skip the redundant driver image pull (configGPUDrivers)

The aks-gpu-cuda image is pre-fetched into the VHD at build time (install-dependencies.sh → image-fetcher), so it is already present locally at boot. configGPUDrivers() nonetheless ran ctr image pull unconditionally — a wasted manifest/layer round trip to MCR and exposure to MCR throttling. Now we only pull when the image is genuinely absent locally; otherwise we go straight to ctr run. Median savings are small; the real value is avoiding a multi-second-to-minutes stall when MCR is throttling.

2. Async image cleanup (drop --sync)

The post-install ctr images rm --sync blocked CSE waiting for containerd GC to finish before returning. Dropping --sync removes the image reference immediately and lets GC reclaim space asynchronously — same disk outcome, no blocking. Small, on-path saving.

3. Defer DCGM telemetry off the critical path

nvidia-dcgm and nvidia-dcgm-exporter are monitoring only and don't gate GPU workload scheduling, yet they were started with the blocking systemctlEnableAndStart and hard-exited CSE on a slow/failed start (up to a 30s timeout each). They now use systemctlEnableAndStartNoBlock and are non-fatal. The nvidia-device-plugin start stays blocking and fatal because it gates the node advertising GPUs to the scheduler. Note: this path only runs when the managed-GPU experience is enabled, so it does not affect default GPU nodes today.

Tests

• New shellspec coverage for startNvidiaManagedExpServices: asserts device-plugin stays blocking while dcgm/dcgm-exporter are enqueued off the critical path and don't fail provisioning. (7/7 GPU-service examples pass.)
• go test ./pkg/agent/... — pass.
• make generate — no snapshot diffs.

Risk / behavior notes

• Initial Setup #1 keeps a full fallback pull when the image is missing — no regression on non-prebaked images. Image-existence check uses fixed-string whole-line matching (grep -qxF) to avoid regex false positives from dots in the image ref.
• Add vhdbuilder and publisher #2 still removes the image; only the synchronous wait is gone. Worst case is slightly delayed disk reclamation on very tight disks.
• import: remove iovisor apt remove from vhd #3 makes DCGM/exporter start non-fatal. This is the intended trade-off (telemetry should never block or fail node provisioning). Because systemctlEnableAndStartNoBlock returns before the service actually starts, the dcgm-exporter=enabled node label is now a feature-intent label rather than a "metrics confirmed up" signal. Reviewers who want DCGM failures to remain fatal should flag it.

Coordination

Touches configGPUDrivers(), which the held PR #8612 (prebuild GPU kernel module) also edits — these two will need a light rebase against each other whenever both land. This PR is independent and can merge on its own.

---

Re-opened from the same repo (was #8613 from a fork; AgentBaker requires same-repo branches for CI/secrets).

[Copilot]

Pull request overview

This PR optimizes the Ubuntu GPU provisioning path in cse_config.sh by avoiding unnecessary container registry pulls, removing a synchronous containerd GC wait, and moving DCGM telemetry service startup off the CSE critical path (non-fatal, non-blocking).

Changes:

• Skip ctr image pull for the NVIDIA driver image when it already exists locally.
• Remove --sync from ctr images rm to avoid blocking on containerd garbage collection.
• Start nvidia-dcgm and nvidia-dcgm-exporter via systemctlEnableAndStartNoBlock and treat failures as non-fatal; keep nvidia-device-plugin blocking/fatal.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
parts/linux/cloud-init/artifacts/cse_config.sh	Adds conditional image pull, async image removal, and defers DCGM/DCGM-exporter startup off the provisioning critical path.
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh	Adds ShellSpec coverage validating device-plugin remains blocking while DCGM services are enqueued non-blocking and are non-fatal on failure.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.