Skip to content

sandboxd: tenant managed volumes that outlive claims - #370

Open
minatoaquaMK2 wants to merge 3 commits into
cocoonstack:mainfrom
minatoaquaMK2:feat/tenant-managed-volumes-300
Open

minatoaquaMK2 wants to merge 3 commits into
cocoonstack:mainfrom
minatoaquaMK2:feat/tenant-managed-volumes-300

Conversation

@minatoaquaMK2

@minatoaquaMK2 minatoaquaMK2 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Tenants create node-local data disks with PUT /v1/volumes/{name}, attach one to a claim (warm VMs included), and keep it across release, restart, template changes and hibernate. One claim at a time holds a writable volume, and the hold is derived from claims.json and the #349 stale-VM inventory. A claim for a volume on another mesh node redirects there. This implements #300. Online growth of an attached volume, idle hibernation and archive export stay open.

What changed

  • PUT, GET and DELETE /v1/volumes/{name}, and POST /v1/volumes/{name}/grow, manage raw sparse files under managed_volume_dir. registry.json records each volume's owner, size and backing file, and nothing else.
  • A claim attaches a managed volume the same way as a catalog volume. Holds survive a restart with no second journal.
  • With managed_volume_hibernate set, hibernate of a claim with one mounted writable volume runs a guest hook, unmounts, detaches and captures. Wake reverses the steps. The claim row records the phase as volume_sleep.
  • Each mesh node gossips a tenant-scoped key per volume, so peers can redirect a claim without learning names.

Scope

  • Growth is offline only. A held volume answers 409 naming the holder, because cocoon cannot resize an attached disk yet.
  • Archive refuses a claim that holds a volume. Idle hibernation and sleep of attach-only or multi-volume claims are not supported.

Points we'd like to talk through

We made a call on each of these so the branch works end to end. Any of them can change if you see it differently.

  • Sleep phase in claims.json. It sits on the claim row as volume_sleep. An older sandboxd would wake such a claim without its disk, so the docs ask for a drain before a rollback.
  • Error replies. The managed routes return {"error","code"} with statuses picked in pool (VolumeError). We can move them to sentinels in poolErrHTTP if you prefer that style.
  • Hook contract. The hook runs as a host-built flock and timeout exec of hook_argv in the guest, set per node. A silkd-native hook would work too.
  • Growth. Growth is offline only for now. Online growth needs cocoon to resize an attached disk (Cloud Hypervisor /vm.resize-disk).
  • Placement. Root claims are not redirected to managed volumes, and name uniqueness across the mesh holds once gossip catches up.
  • Stacked mounts on main. Release unmounts the topmost mount at a volume path, so a mount stacked on a writable volume is popped and its dirty marker cleared. This predates the PR, and we can open a separate issue for it.

Blast radius

Nodes without managed_volume_dir keep their behavior, with two shared changes. A guest mount conflict or a too-old cocoon on hibernate or wake answers 409 or 503 instead of 500. The Python SDK rejects a bare string or dict as a volume list. Nodes that ran earlier builds of this branch must be drained before upgrade, because the formats have no migration.

Verification

  • go test -race passes for sandboxd and e2e, except two tests that also fail on main when run as root. golangci-lint passes on linux and darwin, and ruff, pytest and mypy pass.
  • TestManagedVolumeIssue300Acceptance walks the acceptance line over HTTP. TestManagedSleepRoundTripAndExclusiveHold covers hibernate and wake, and two e2e tests cover placement.
  • Not yet run on real Cloud Hypervisor.

The first comment explains how we built this and the decisions behind it.

@minatoaquaMK2

Copy link
Copy Markdown
Contributor Author

How we built it and what we decided

This replaces #368 and #369. We re-checked the branch against every edge case #339 settled, then reworked it.

Process. We listed 32 cases from the #339 review (busy and stacked mounts, a release racing a change, restarts mid-change, error codes, Retry-After, journal rules) and checked each one against every managed path. Five parallel reviews found 12 must-fix bugs, each proven by a failing test. We compared two designs before any rework. Every fix landed as a failing test, then the fix. Two independent reviews followed, and the second approved.

Decisions.

Decision Why
One journal: holds come from claims.json and the #349 stale-VM inventory. registry.json keeps only volume records. Two journals had to stay in step, and most must-fix bugs came from the gaps.
No node-wide fence. A leftover VM that cannot be removed is queued like any stale VM. If cocoon cannot inspect it, it holds every managed name until a later pass. One failed vm rm used to make every managed volume on the node fail until a restart. This answers the stale-VM question from #369.
The sleep phase lives on the claim row. A failed final wake write keeps memory and retries. Matches main's claim store. A crash before the write replays the resume hook once, and the hook is required to be idempotent.
Release and expiry use main's paths and do not wait behind an in-flight hibernate or wake. Waiting behind the hook could stall maintenance on the whole node.
Offline growth replaces the cocoon vm disk grow path. No cocoon release has that command.
A tenant claim naming an unknown name, a forbidden catalog volume, or another tenant's managed volume gets the same 400. /v1/volumes/{name} answers 404. Privacy needs identical answers, not a particular status, and the claim contract stays as on main.
An interrupted create stays creating. The same PUT finishes it, and DELETE removes it. The file was never handed to a VM, so it is safe to replace.
volume_hook_failed (409) only for a guest hook exit. Transport errors stay a generic 500. Tenants must not see host paths.
UnmountVolume stays as on main. The device-aware version added guest round trips to every catalog release.

Deferred. Pool-side error statuses (question 2), online growth, idle and attach-only sleep, re-arming the egress proxy on wake, an e2e hibernate test, and archive export.

Tenants create node-local data disks with `PUT /v1/volumes/{name}`,
attach one to a claim (warm VMs included), and keep it across release,
restart, template changes and hibernate. One claim at a time holds a
writable volume, and the hold is derived from `claims.json` and the
cocoonstack#349 stale-VM inventory. A claim for a volume on another mesh node
redirects there. This implements cocoonstack#300. Online growth of an attached
volume, idle hibernation and archive export stay open.
The test asserted the retried PUT left a different inode, but ext4 on
the CI runner reuses the freed inode number, so os.SameFile reported the
fresh file as the old one. Seed the leftover file with bytes instead and
assert the new 1 GiB file starts with zeros, which is what a reader of
the volume observes.
The test removed the backing and recreated it at the same size. ext4 on
the CI runner reused the inode number, so the replacement matched the
recorded identity and the claim succeeded. Build the replacement beside
the original and rename it over, which is a real replacement the
identity check must catch.
@minatoaquaMK2
minatoaquaMK2 force-pushed the feat/tenant-managed-volumes-300 branch from a2858eb to 2ebd52d Compare October 22, 2026 01:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant