Harden hypervisor process liveness checks

[yummybomb]

Summary

Layer 1 of the vendor VFIO vGPU stack (generalize-vgpu-device ← this ← vendor-vfio-backend ← vendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.

The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:

• Unify liveness checks on ProcessExists — one exported, EPERM-aware, zombie-filtering definition instead of scattered bare kill(pid, 0) probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.
• Wait for non-child hypervisor exit before finishing kill — after a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.
• Verify process identity before teardown — a bare PID probe treats any process that reused a stored hypervisor PID as the owning VMM. On Linux, require confirmed socket ownership or a matching PID/start-time token scoped to the current host boot, including the graceful stop/delete wait path.
• Treat provable death as death — two disproofs unwedge stop/delete instead of failing closed forever: an identity token recorded under a different host boot cannot be the recorded VMM, and a socket with no owning listener proves the recorded VMM is gone even when a recycled PID is live. The recycled PID is never signaled.
• Delete what the hardened resolver made redundant — the startup identity backfill (correctness never depended on persisted tokens: token-less metadata confirms through the expected-owner fast path, and an empty listener scan is provable death) and stop's unconditional force-kill after a shutdown that already confirmed exit.
• Drop the command-line fallback from socket resolution — a process matching the socket path by argv only could never authorize teardown, so its sole effect was turning provable death into a wedge: a debug client (ch-remote, socat) holding the path in its argv blocked stop/delete with a 500 until it exited. The fd scan runs with root/CAP_SYS_PTRACE (documented on ResolveProcessPID), so a missing listener is proof of death on its own. If hypeman is ever de-privileged below that, this needs revisiting.

Failure contract

When the hypervisor can be confirmed dead or killed, delete completes synchronously (worst case ~2s graceful + SIGKILL waits). When ownership is genuinely unprovable — the /proc socket scan itself fails, or a process survives SIGKILL — stop/delete return an error (API 500) with metadata retained and the restart policy blocked, so a retried delete is safe and converges once the ambiguity clears. An earlier revision handled this with a background delete finalizer; it was removed in favor of the synchronous error after review — the ambiguous cases are rare, self-resolving, and better surfaced loudly than retried silently.

Testing

• go build ./..., go vet clean
• Identity/kill/shutdown suites additionally cross-compiled and run as root on a dev host against real /proc: all pass. Validated against the host's production firecracker binary: a live VMM resolves with no hint, the correct owner hint, and a wrong hint; a SIGKILLed VMM whose socket file remains on disk classifies as provably dead; a dead socket path held in a live process's argv resolves as provably dead instead of wedging
• go test -race ./lib/instances/ targeted suites pass (TestCreateInstanceWithNetwork requires image pulls + iptables and fails in this environment on the unmodified base as well; TestStandbyAndRestore/TestForkCloudHypervisorFromRunningNetwork boot real VMs and cannot run in this sandbox — CI covers them)

---

Note

High Risk
Changes core stop/delete/standby and hypervisor kill paths; incorrect ownership logic could leave VMs running or signal wrong processes, though extensive tests target PID reuse and socket edge cases.

Overview
Hypervisor process identity and Linux socket resolution are reworked so stop, delete, and standby only signal processes that truly own the instance control socket.

On Linux, ResolveProcessPID now maps the bound path to a listening inode in /proc/net/unix, scans process fds (with optional ResolveProcessPIDForOwner when a stored PID might share the fd with a child), and returns ErrNoOwningProcess when no listener exists. The old cmdline fallback is removed so debug clients that only mention the socket path cannot block teardown.

Instance metadata gains embedded HypervisorProcessIdentity (PID + start time + boot ID, flat JSON keys unchanged). Boot/restore mints tokens only for a live spawned child or a confirmed socket owner; destructive paths use resolveLiveHypervisorPID to prefer matching identity, disprove stale PIDs after reboot or when the socket has no owner, and error if the fd scan fails ambiguously.

Lifecycle behavior changes: killHypervisor / shutdownHypervisor resolve the live owner before SIGKILL, wait for exit via shared killProcessAndWait / ProcessExists (EPERM-aware, zombie-aware), and remove the socket only after the VMM is gone. Failed hypervisor kill on delete or shutdown on standby now abort with metadata retained instead of continuing cleanup. Stop waits on the socket owner, not a bare stored PID, and drops the redundant force-kill after a successful shutdown.

^{Reviewed by Cursor Bugbot for commit 5bad493. Bugbot is set up for automated code reviews on this repo. Configure here.}