Repository navigation
sandboxd: tenant managed volumes that outlive claims - #370
Open
minatoaquaMK2 wants to merge 3 commits into
Open
minatoaquaMK2 wants to merge 3 commits into
minatoaquaMK2 wants to merge 3 commits into
Conversation
Contributor
Author
How we built it and what we decidedThis replaces #368 and #369. We re-checked the branch against every edge case #339 settled, then reworked it. Process. We listed 32 cases from the #339 review (busy and stacked mounts, a release racing a change, restarts mid-change, error codes, Retry-After, journal rules) and checked each one against every managed path. Five parallel reviews found 12 must-fix bugs, each proven by a failing test. We compared two designs before any rework. Every fix landed as a failing test, then the fix. Two independent reviews followed, and the second approved. Decisions.
Deferred. Pool-side error statuses (question 2), online growth, idle and attach-only sleep, re-arming the egress proxy on wake, an e2e hibernate test, and archive export. |
Tenants create node-local data disks with `PUT /v1/volumes/{name}`,
attach one to a claim (warm VMs included), and keep it across release,
restart, template changes and hibernate. One claim at a time holds a
writable volume, and the hold is derived from `claims.json` and the
cocoonstack#349 stale-VM inventory. A claim for a volume on another mesh node
redirects there. This implements cocoonstack#300. Online growth of an attached
volume, idle hibernation and archive export stay open.
The test asserted the retried PUT left a different inode, but ext4 on the CI runner reuses the freed inode number, so os.SameFile reported the fresh file as the old one. Seed the leftover file with bytes instead and assert the new 1 GiB file starts with zeros, which is what a reader of the volume observes.
The test removed the backing and recreated it at the same size. ext4 on the CI runner reused the inode number, so the replacement matched the recorded identity and the claim succeeded. Build the replacement beside the original and rename it over, which is a real replacement the identity check must catch.
minatoaquaMK2
force-pushed
the
feat/tenant-managed-volumes-300
branch
from
October 22, 2026 01:59
a2858eb to
2ebd52d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tenants create node-local data disks with
PUT /v1/volumes/{name}, attach one to a claim (warm VMs included), and keep it across release, restart, template changes and hibernate. One claim at a time holds a writable volume, and the hold is derived fromclaims.jsonand the #349 stale-VM inventory. A claim for a volume on another mesh node redirects there. This implements #300. Online growth of an attached volume, idle hibernation and archive export stay open.What changed
PUT,GETandDELETE /v1/volumes/{name}, andPOST /v1/volumes/{name}/grow, manage raw sparse files undermanaged_volume_dir.registry.jsonrecords each volume's owner, size and backing file, and nothing else.managed_volume_hibernateset, hibernate of a claim with one mounted writable volume runs a guest hook, unmounts, detaches and captures. Wake reverses the steps. The claim row records the phase asvolume_sleep.Scope
Points we'd like to talk through
We made a call on each of these so the branch works end to end. Any of them can change if you see it differently.
claims.json. It sits on the claim row asvolume_sleep. An older sandboxd would wake such a claim without its disk, so the docs ask for a drain before a rollback.{"error","code"}with statuses picked inpool(VolumeError). We can move them to sentinels inpoolErrHTTPif you prefer that style.flockandtimeoutexec ofhook_argvin the guest, set per node. A silkd-native hook would work too./vm.resize-disk).Blast radius
Nodes without
managed_volume_dirkeep their behavior, with two shared changes. A guest mount conflict or a too-old cocoon on hibernate or wake answers 409 or 503 instead of 500. The Python SDK rejects a bare string or dict as a volume list. Nodes that ran earlier builds of this branch must be drained before upgrade, because the formats have no migration.Verification
go test -racepasses for sandboxd and e2e, except two tests that also fail on main when run as root. golangci-lint passes on linux and darwin, and ruff, pytest and mypy pass.TestManagedVolumeIssue300Acceptancewalks the acceptance line over HTTP.TestManagedSleepRoundTripAndExclusiveHoldcovers hibernate and wake, and two e2e tests cover placement.The first comment explains how we built this and the decisions behind it.