feat(sandboxd): reconcile live read-only volume attachments

README.md

@@ -121,7 +121,8 @@ TEMPLATE=rt:24.04 scripts/sandboxd-e2e.sh
 #   without a Go toolchain.
 # VOLUME_IMAGE=/absolute/dataset.img enables the volumes proof (ro sharing;
 #   the image contains volume-e2e.txt); VOLUME_RW_IMAGE=/absolute/scratch.img
-#   adds the rw claim/release leg. Prebuilt runs also set VOLUME_SMOKE_BIN.
+#   adds the rw claim/release leg; VOLUME_LIVE=1 adds the live volume leg.
+#   Prebuilt runs also set VOLUME_SMOKE_BIN.
 ```
 
 ## CI

docs/deploy.md

@@ -13,7 +13,8 @@ the cocoon CLI and needs a template image with silkd baked in.
   [performance](performance.md) numbers assume; v0.6.9 prints `[]` for an
   empty `snapshot list --format json`, which sandboxd's snapshot reconcile
   reads. sandboxd logs a warning at startup when the detected cocoon is below
-  v0.6.9 (a dev/`master-<sha>` build is assumed current)
+  v0.6.9 (a dev/`master-<sha>` build is assumed current); changing volumes on
+  a running claim needs v0.7.2
 - The sandbox boot artifact installed where cocoon finds it
   (`/boot/vmlinuz-sandbox`, `/boot/initrd.img-sandbox` — from
   `ghcr.io/cocoonstack/sandbox/boot:<kernel-ver>`)
@@ -156,8 +157,36 @@ fragment the warm pools):
 
 ### Dataset volumes
 
+Running claims can change mounted read-only catalog entries through the
+[GET, PUT, and PATCH volume API](sandboxd-api.md#get-v1sandboxesidvolumes)
+and both SDKs. The image must already exist on the owner node. Live changes
+require Cocoon v0.7.2 or newer for idempotent disk attach and detach. PUT and
+PATCH reject an older release, or a runtime whose version cannot be queried,
+with 503 before any intent, hold, or device change; a dev build is assumed
+current. sandboxd reads the version once, so restart it after upgrading cocoon.
+GET and release remain available.
+
+The claim journal stores confirmed applied `Volumes`, plus desired
+`VolumesWant` while it differs.
+Admission holds cover their union across daemon restart and teardown. Retry
+an interrupted operation with the same PUT target. Release cancels convergence
+and transfers holds to teardown. Capture is refused while either set is
+nonempty.
+
+A sandbox whose VMM exits while holding volumes is release-only. Release it and
+create a new sandbox. Daemon restart recovery assumes the original VMM remains
+alive and does not recreate guest processes. Unfinished changes report an
+interrupted diagnostic after restart and require a caller retry.
+
+`e2e/cmd/volumesmoke -live` (`VOLUME_LIVE=1` for `scripts/sandboxd-e2e.sh`)
+exercises GET, repeated full-set PUT, PATCH add and remove, and clearing a new
+claim.
+
 Each catalog path must name a disk image containing a mountable whole-device
-filesystem. A read-only entry's image must stay immutable: do not replace,
+filesystem. Keep catalog names bound to the same image while any claim holds
+them. Use versioned names such as `dataset-v1` and
+`dataset-v2`; sandboxd cannot detect a repointed catalog entry. A read-only
+entry's image must stay immutable: do not replace,
 truncate, or delete it while attached; publish a new catalog name or path
 instead. A missing path produces a startup warning and fails only claims that
 request it, allowing images to be distributed after sandboxd starts.
@@ -681,7 +710,8 @@ using prebuilt binaries, also set `VOLUME_SMOKE_BIN` beside `SANDBOXD_BIN`,
 
 Add `VOLUME_RW_IMAGE=/srv/datasets/scratch.img` (a second, writable image) to
 also run the writable leg: a durable write across release, second-writer
-exclusion, and a clean read-only claim afterward.
+exclusion, and a clean read-only claim afterward. `VOLUME_LIVE=1` adds the live
+volume leg on cocoon v0.7.2 or newer.
 
 ## TLS for SDK clients
 

docs/live-volumes.md

@@ -0,0 +1,93 @@
+# Replace a dataset in a running sandbox
+
+This example replaces `dataset-v1` with `dataset-v2` at `/datasets/current`
+while preserving the sandbox, session, guest boot, and independent process.
+
+Register both read-only catalog entries on the owning node and grant the
+sandbox's tenant access to them. Put `v1` and `v2` in each image's `/version`
+file. Keep each name bound to the same immutable image while held. Use Cocoon
+v0.7.2 or newer and set `SANDBOX_ADDR` and `SANDBOX_API_TOKEN` for the client.
+A sandbox token alone cannot manage volumes.
+
+Pause application reads before replacement. Close files, mmap mappings, cwd,
+and bind mount references on the old disk. Replacement unmounts the old disk
+before mounting the new one, so the path has a short unavailable interval.
+The independent process below uses `/tmp` and can continue running.
+
+```python
+import os
+
+from cocoonsandbox import Client
+
+
+def process_identity(sb, session, pid):
+    stat = sb.exec("cat", f"/proc/{pid}/stat")
+    starttime = stat.rsplit(") ", 1)[1].split()[19]
+    boot = sb.exec("cat", "/proc/sys/kernel/random/boot_id").strip()
+    return sb.id, session.id, boot, pid, starttime
+
+
+client = Client(os.environ["SANDBOX_ADDR"], api_token=os.environ["SANDBOX_API_TOKEN"])
+v1 = [{"name": "dataset-v1", "mount": "/datasets/current"}]
+v2 = [{"name": "dataset-v2", "mount": "/datasets/current"}]
+
+with client.new("rt:24.04", ttl_seconds=900) as sb:
+    with sb.session(cwd="/tmp", env={"DATASET_SESSION": "same"}) as session:
+        pid = sb.spawn("sleep", "3600", cwd="/tmp")
+        try:
+            identity = process_identity(sb, session, pid)
+            sb.set_volumes(v1)
+            assert sb.exec("cat", "/datasets/current/version").strip() == "v1"
+
+            sb.patch_volumes(remove=["dataset-v1"], add=v2)
+            sb.set_volumes(v2)
+
+            assert sb.exec("cat", "/datasets/current/version").strip() == "v2"
+            assert process_identity(sb, session, pid) == identity
+            assert session.id in sb.sessions()
+            assert session.exec("printenv", "DATASET_SESSION").strip() == "same"
+            view = sb.get_volumes()
+            assert view["volumes_want"] == view["volumes"] == v2
+            assert view["converging"] is False
+
+            sb.set_volumes([])
+            assert process_identity(sb, session, pid) == identity
+        finally:
+            sb.kill(pid)
+```
+
+A successful run reads `v1`, then `v2`, and preserves every identity assertion.
+The repeated full target verifies that a settled mount can be retained.
+This example starts without other volumes. If your sandbox has writable or
+attach-only volumes, retain them in a full PUT or use PATCH to replace only
+the dataset names. `None`, a nil Go slice, and a missing `volumes` field do not
+mean an empty target.
+
+Go uses the same owning handle and client API token.
+
+```go
+view, err := sb.PatchVolumes(ctx, sandbox.VolumePatch{
+	Remove: []string{"dataset-v1"},
+	Add: []sandbox.Volume{{Name: "dataset-v2", Mount: "/datasets/current"}},
+})
+if err != nil {
+	return err
+}
+if view.Converging {
+	return errors.New("volume request still running")
+}
+```
+
+For a 409 mount conflict, use the guest path in `states[].error` to close
+references or remove unexpected mounts, then retry. For a transport failure or
+timeout, GET the current target and confirmed set before retrying the target.
+A failed call can already have changed devices. `VolumeError` retains a
+complete failure view in Go's `Response` or Python's `.response`.
+
+`pending` or `detaching` with `converging: false` needs a caller retry. GET does
+not run device repair. After sandboxd restarts with the original VM still
+alive, unfinished entries report an interrupted operation and keep their
+admission holds. Detailed errors survive only within one daemon lifetime.
+If the VMM exits, a sandbox holding volumes is release-only. Release it and
+create a new sandbox; neither PUT with an empty target nor daemon restart
+restores the original processes.

docs/sandboxd-api.md

@@ -255,6 +255,132 @@ list; the field is emitted (as `true`) only for a writable entry this node
 declares and omitted otherwise (peer-only rows never carry it), so a read-only
 entry's response is byte-identical to v1.
 
+## GET /v1/sandboxes/{id}/volumes
+
+Returns the persisted desired set (`volumes_want`), confirmed applied set
+(`volumes`), per-volume `states`, and `converging`. Auth requires a client API
+token. Root can access every sandbox; a tenant can access only its own.
+Sandbox tokens are rejected when authentication is enabled. The owning node
+handles the request without redirecting.
+
+```json
+{
+  "volumes_want": [{"name": "imagenet", "mount": "/datasets/training"}],
+  "volumes": [{"name": "imagenet", "mount": "/datasets/training"}],
+  "states": [{"name": "imagenet", "state": "attached"}],
+  "converging": false
+}
+```
+
+All three arrays are present, including when empty. A state is `attached`
+when the volume is in both sets, `pending` when desired but not yet confirmed,
+or `detaching` when applied but no longer desired. An interrupted device call
+can leave the actual guest state different from the confirmed set. A failed
+PUT or PATCH adds a top-level `error` naming the refusal, and the affected
+state may carry its own `error`. `converging` is true only while a request worker is running.
+`pending` with `converging: false` requires an explicit retry; there is no
+background convergence. GET retains the last safe per-volume error for the
+current daemon lifetime. After daemon restart, unfinished states report an
+interrupted operation that needs retry. Successful retries clear resolved errors.
+The sandbox index exposes both sets for operator inspection.
+
+## PUT /v1/sandboxes/{id}/volumes
+
+Replaces the full desired set and converges synchronously. Auth matches GET.
+The owner tenant's catalog access list applies to newly admitted names, including
+for root. Retaining, repairing, or removing held names does not reauthorize
+them. A new image must exist on the owning node. The request must include a non-null `volumes`
+array. An explicit empty array removes all mutable entries.
+
+```json
+{"volumes": [{"name": "imagenet", "mount": "/datasets/training"}]}
+```
+
+Only mounted read-only catalog entries can change live (`mode` omitted or
+`ro`). A PUT must retain every existing writable or attach-only entry unchanged.
+The claim-time name, mount, and eight-volume limits apply to the desired set.
+An omitted mount defaults to `/volumes/<name>`, except that a held attach-only
+entry is retained by sending it as the claim returned it, without `mount`. To
+change an existing name's
+mount or mode, first remove that name and wait for successful convergence.
+An identical target can be retried. Claims with `on_expire: archive` must change
+that policy before adding a volume.
+
+The sandbox must be running. Paused, archived, and failed sandboxes return 409
+without waking. A failed sandbox holding volumes is release-only after its VMM
+exits; PUT with an empty target cannot restore its processes. Missing, invalid,
+or sandbox bearer tokens return 401. Another tenant's sandbox and an unknown id
+return 404. Invalid, inaccessible, or non-local entries return 400. Removed
+tenant tokens are rejected at authentication. Root can still repair or remove
+held entries, but admitting a new name for a removed tenant returns 403. A conflicting
+writer or dirty image returns 409.
+
+## PATCH /v1/sandboxes/{id}/volumes
+
+Changes the desired set by name, then converges synchronously. Auth, validation,
+and responses match PUT. Omitted `add` or `remove` arrays leave that part
+unchanged. An empty patch retries the current desired set.
+
+```json
+{"add": [{"name": "imagenet", "mount": "/datasets/training"}], "remove": ["old-data"]}
+```
+
+Adding an identical entry or removing an absent valid name is harmless.
+Duplicate names, a name in both arrays, or changes to writable or attach-only
+entries return 400. Removal refuses busy or foreign mounts and uses neither
+lazy nor forced unmount. A foreign mount, another bind mount, or a completed
+ordinary unmount with nonzero exit returns 409 with the guest mount path and
+a safe per-volume reason. Guest transport failures remain server errors;
+timeouts return 504 and cancellation returns 408.
+
+### Partial failure and recovery
+
+PUT and PATCH return the GET response shape on success. A convergence error
+returns a non-2xx status with the same view, so callers can inspect desired,
+applied, and per-volume state; semantic validation 400s carry it too. A
+malformed body, a missing `volumes` array, authentication and an unknown sandbox
+carry only `error`. A failed request may already have persisted its target and
+completed some device changes. GET reads the last confirmed set, rather than
+probing the devices. Resolve guest mount references or runtime failures before
+retrying. Retry the same PUT to converge, or send a new
+target to abandon unfinished additions. Concurrent requests serialize per
+sandbox without blocking renewals or new exec, file, and port connections.
+
+Before applying a target, sandboxd journals `VolumesWant` beside `Volumes`,
+storing it only while it differs from `Volumes`. A
+superseding request may first clean up an older desired-only attachment while
+the old target stays recorded. If that cleanup fails, the response retains the
+old view; retry the same PUT to finish accepting the new target. Admission
+holds cover the deduplicated union of both sets, including during restart and
+teardown; a teardown keeps them until the VM is confirmed gone. Failed or
+uncertain operations retain their holds until successful cleanup. Startup
+reserves those holds without replaying attachments. Release cancels convergence and transfers the held union to teardown without waiting
+for a blocked device call. Capture operations (hibernate, fork, checkpoint, and
+promote) are refused while either set is nonempty or a convergence request
+still owns admitted volume holds.
+
+Device calls honor cancellation and have a 30-second ceiling per volume; a
+caller that disconnects cancels the run, leaving interrupted states to retry. Journal writes
+are atomic. Host-crash durability follows `sync_claims`. An older daemon
+ignores `VolumesWant`; finish pending volume changes before downgrading.
+
+Live convergence requires Cocoon v0.7.2 or newer for idempotent matching
+attach and absent detach. PUT and PATCH reject an older release, or a runtime
+whose version cannot be queried, with 503 before changing intent, holds, or
+devices; a dev build is assumed current. sandboxd caches the first successful
+version read, so restart it after upgrading cocoon. GET and release remain
+available.
+
+Catalog names must keep the same backing image while held, including pending
+holds. Publish changes under versioned names such as `dataset-v1` and
+`dataset-v2`. The journal cannot detect a catalog name repointed to a new
+image.
+
+A replacement first unmounts the old disk, then mounts the new disk. Pause reads
+and close files, mmap, cwd, and bind mount references during this gap. Existing
+independent guest processes and sessions continue. See the
+[same-sandbox replacement tutorial](live-volumes.md).
+
 ## POST /v1/sandboxes/{id}/release
 
 Auth: the sandbox's own token, or the root token for operator cleanup by id.
@@ -669,7 +795,7 @@ an SDK caller should set.
 Auth: node API token. Root sees every live claim; a tenant sees only its own.
 The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated,
 archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes,
-volumes?: [{name, mount, mode?}], restarts?, restarted_at?, failed?}]}` — `cpu_count` and `mem_total_bytes` are the
+volumes?: [{name, mount?, mode?}], volumes_want?: [{name, mount?, mode?}], restarts?, restarted_at?, failed?}]}` — `cpu_count` and `mem_total_bytes` are the
 size tier's allocation; `restarts` counts the cold boots that replaced an
 exited VMM and `restarted_at` stamps the last one (each loses guest memory and
 the instance-metadata document, so a caller re-establishes guest state when

docs/sdk-python.md

@@ -161,6 +161,26 @@ peer advertising both the template and every volume; when that intersection is
 empty, one volume holder may self-verify access to a shared template store
 before provisioning.
 
+## Changing read-only datasets
+
+Use `sb.get_volumes()`, `sb.set_volumes([...])`, and
+`sb.patch_volumes(add=[...], remove=[...])` on the same running sandbox. These
+methods use the client's API token on the owning node; a sandbox token cannot
+manage datasets. `set_volumes([])` clears mutable entries; `None` is rejected.
+PUT must retain writable and attach-only entries. PATCH retains omitted names.
+Each method takes `timeout=` (default: the client timeout, 120 s); a large
+replacement can need longer, and a call that times out cancels the server run,
+so retry the same target.
+
+Responses include `volumes_want`, `volumes`, `states`, and `converging`.
+`VolumePhase` and `VOLUME_ATTACHED`, `VOLUME_PENDING`, and `VOLUME_DETACHING`
+name the three states. A pending state with `converging=False` needs an explicit
+retry. `VolumeError`, exported from `cocoonsandbox` and `cocoonsandbox.errors`,
+retains a complete failure view in `.response`. Complete replies update both
+public sets; incomplete errors preserve the last view. Serialize calls on one
+handle. See the [replacement tutorial](live-volumes.md) and
+[API recovery contract](sandboxd-api.md#partial-failure-and-recovery).
+
 ## Renewing
 
 ```python