vm: cpuset fence — reserve host cores from the VM population, optional per-VM core placement

[CMGS]

Follow-up to #182. Under high density the VM population can occupy every host core; the CPU scope caps how much time VMs burn but not where — clone/wake execution, vk, and the OS still compete for CPU with the fleet on all cores. Field data (vk three-tier wake runs): CH clone median 6.98s but p95 24.9s / max 69.2s under load.

Neither backend can express this natively: CH --cpus affinity= pins only vCPU threads while virtio/io-wq workers (the 10-13% overhead measured in #182) roam every core; FC has nothing. The mechanism is cgroup v2 cpuset, and it lands on the parent/scope infrastructure #182 built.

Proposal

Two layers, both raw cpuset encodings, zero hot-path cost (one control-file write at parent setup / VM launch):

P0 — machine fence (config): cgroup_cpus config key (e.g. "0-14"; default empty = all cores = today's behavior). ensureParent enables +cpuset along the chain and writes cpuset.cpus on the parent once. Every VM scope inherits — VM threads physically cannot touch reserved cores regardless of per-VM quota planning. Overcommit then happens inside the fence; reserved cores stay available to the host, vk, and the clone path. Same shape as K8s reserved-cpus / systemd AllowedCPUs.

P1 — per-VM placement (flag): --cpuset-cpus on vm run/clone (range or specific cores) → the scope's own cpuset.cpus. Documented as non-work-conserving (#182 listed pinning as a non-goal for defaults — that reasoning stands; this is an explicit opt-in for NUMA affinity / isolation-sensitive workloads, never a default). Snapshot semantics follow #182's rule: the knob is host policy — a record, never applied from snapshots; clone takes it from flags alone, restore keeps the VM's.

Composition: fence (where) × cpu.weight (share) × cpu.max (ceiling) are orthogonal; all three combine.

Validation is cocoon's job — the kernel does not reject out-of-range requests

cgroup v2 cpuset.cpus is a requested set; cpuset.cpus.effective is what the parent actually grants, and the kernel contract allows the effective set to silently shrink — a request none of whose CPUs can be granted behaves like an empty request (inherit parent), not an error. So enforcement is explicit, all driven by one parsed set:

• machine fence must be validated ⊆ the parent's ancestor cpuset.cpus.effective;
• each VM placement must be validated ⊆ the parent's cpuset.cpus.effective;
• shrinking the fence is refused while an existing VM's non-empty placement would be invalidated by it;
• the same parsed set drives the queue_affinity clamp — no extra abstraction.

Convergence: clearing the fence must converge

The parent cgroup persists across runs and #182's cleanup removes only vm-*.scope children — so "empty config never writes" would leave a previously-set fence at 0-14 forever, breaking empty = all cores. Parent reconciliation is read-gated, mirroring #182's subtree_control handling: fresh/steady empty stays zero-write, but a stale non-empty cpuset.cpus on the parent is cleared once when the config is empty. The controller itself is never disabled.

Interactions to handle

• Disk queue_affinity: args.go pins virtio-blk queues to host CPUs 0..N-1 unconditionally. With a fence (or per-VM cpuset) excluding low cores, those targets fall outside the allowed set — queue affinity targets must clamp to the effective cpuset or be dropped when they don't intersect. The clamp lives on the shared storageConfigToDisk path so it covers initial launch, clone's post-restore disk addition, and runtime DiskAttach alike — not just first boot.
• Delegation: systemd user slices typically do not delegate cpuset (testbed: user@1000.service delegates cpu/memory/pids only). Non-root test environments need widened delegation or root; production (root) is unaffected. Preflight follows vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism #182: a missing cpuset interface file with the knob set is an actionable error naming the file.
• Hotplug ceiling: CH max=NumCPU boot arg is unrelated (guest topology); no change.

Acceptance

• Fence 0-14 on a 16-core host + saturated VM fleet: mpstat shows core 15 idle; Cpus_allowed_list of every VMM thread (incl. io-wq) ⊆ 0-14.
• Clone-latency A/B under the same saturation, fence off vs on: reserved-core headroom must show in the tail (this is the motivating number).
• --cpuset-cpus placement visible in /proc/<pid>/status; out-of-fence placement → cocoon rejects with an actionable error (kernel would silently degrade); fence shrink under a live conflicting placement → refused.
• Fence rollback: set "0-14", restart with empty config → parent cpuset.cpus cleared once, subsequent empty runs zero-write.
• queue_affinity targets stay inside the effective cpuset on launch, clone, and DiskAttach.
• Fresh host + empty cgroup_cpus + no flag → behavior byte-for-byte identical to vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism #182 as merged (no cpuset writes, no +cpuset enablement).
• Darwin build stays green.

[CMGS]

Reviewed against #183 head 9ec6368d. I do not think this is ready to freeze yet; two reachable lifecycle gaps remain:

1. Out-of-fence writes cannot rely on kernel EINVAL. In cgroup v2, cpuset.cpus is a requested set and cpuset.cpus.effective is constrained by the parent. The kernel documentation explicitly allows the effective set to be smaller; when none of the requested CPUs can be granted, it is treated like an empty request. Thus parent 0-14 plus VM request 15 may silently fall back to the parent's effective set instead of failing, contradicting both the proposal and acceptance criteria. The implementation should explicitly validate the machine fence against its ancestor effective set, validate each VM placement as a subset of the parent effective set, and reject a parent-fence shrink that would invalidate an existing non-empty VM placement before applying it. The same parsed set can drive queue_affinity; no extra abstraction is needed. See the kernel cpuset contract.

2. Empty cgroup_cpus does not converge after a previous fence. The parent cgroup persists; vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism #182 cleanup removes only vm-*.scope children. If the host first runs with cgroup_cpus: "0-14" and the operator later clears the key, “empty config never touches cpuset” leaves the parent at 0-14 indefinitely. That violates “empty = all cores” and makes rollback non-converging. Minimal fix: read-gate parent reconciliation—fresh/steady empty remains zero-write, but a stale non-empty parent value is cleared once. There is no need to disable the controller.

The overall split remains sound: machine fence as host policy, optional per-VM placement, clone-from-flags/restore-keeps-record semantics, and the queue_affinity interaction. For affinity, make sure the shared storageConfigToDisk path covers launch, clone post-restore disk addition, and runtime DiskAttach, rather than fixing only initial boot.

[CMGS]

Both gaps are real; body updated.

1. You caught a factual error in my draft — I claimed the kernel EINVALs out-of-fence requests, but cpuset.cpus is requested-vs-effective and silently degrades (a fully ungrantable request behaves as empty → inherits the parent). Enforcement is now explicitly cocoon's: fence validated against the ancestor effective set, placements validated against the parent effective set, fence shrink refused while a live placement would be invalidated, and the same parsed set drives the queue_affinity clamp. Acceptance now tests cocoon's rejection, not the kernel's.

2. Convergence gap folded in: parent reconciliation is read-gated like vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism #182's subtree_control — steady empty stays zero-write, a stale non-empty fence is cleared once, controller never disabled. Added a rollback acceptance item (set fence → clear config → parent cleared once, then zero-write).

3. queue_affinity clamp pinned to the shared storageConfigToDisk path so launch, clone's post-restore disks, and runtime DiskAttach all get it.