diff --git a/docs/deploy.md b/docs/deploy.md index f8020f0e..cbf2fdb7 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -112,6 +112,7 @@ gets the intersection of the two. | `cocoon_bin` | `cocoon` | cocoon CLI binary | | `restore_mode` | unset | clone and wake-restore memory mode: `copy`, `ondemand`, or `mmap`; use `mmap` for dense pools | | `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention | +| `managed_volume_hibernate` | absent | Enable explicit hibernate for one mounted managed writable volume with `hook_argv` and `hook_timeout_seconds` from 1 through 300. Requires `managed_volume_dir` and a guest-installed hook. Changes require restart. See [managed volumes](managed-volumes.md#hibernate-one-mounted-managed-volume). | | `sync_claims` | false | fsync `claims.json` and its directory on every commit (claim, release, renew, hibernate, wake, fork, archive, reap), about 0.3 ms each on NVMe; without it a host power loss can leave an empty journal, which fails the next start's reconcile until the file is removed, after which hibernate snapshots are reaped as orphans and archive checkpoints stay stranded. Off by default; turn it on when running claims must come back from a power loss (see `vmm_restart`) | | `cocoond_socket` | unset | `cocoon daemon`'s read-only API socket (by default `/cocoond.sock`); set, the node detects VMM exits from its event stream and polls only as a 60 s backstop. See [dead VMMs](#dead-vmms-and-host-restarts) | | `vmm_restart` | `cold` | what the node does with a running claim whose VMM exited outside a transition (a VMM crash, or a host restart that took it): `cold` cold-boots the claim's own VM from its disk under the same sandbox id and token (guest memory, processes, and the instance-metadata document are lost), `none` keeps the claim listed as `failed` for the caller to release or wake. Egress-lane and volume claims, a VM record gone from cocoon, and a VMM that exits more than 3 times in 10 minutes are marked `failed` under either value. See [dead VMMs](#dead-vmms-and-host-restarts) | diff --git a/docs/index.md b/docs/index.md index 0c0e15e5..a169cb21 100644 --- a/docs/index.md +++ b/docs/index.md @@ -29,6 +29,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. online pool reconfigure, metrics, the usage journal - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a read-only catalog volume on a running claim with the volume API +- [Tenant managed persistent volumes](managed-volumes.md) — node-local tenant + resources, warm attachment, hibernate, offline growth, and recovery limits - [Go SDK](sdk.md) — connecting (single node and clusters), every option, the full sandbox surface, error handling - [Python SDK](sdk-python.md) — the guest and data-plane surface for the diff --git a/docs/managed-volumes.md b/docs/managed-volumes.md new file mode 100644 index 00000000..7654fc3c --- /dev/null +++ b/docs/managed-volumes.md @@ -0,0 +1,319 @@ +# Tenant managed persistent volumes + +Managed volumes keep a tenant's data disk when a sandbox claim is released or +replaced with a different template. The node-local API supports create, inspect, +list, and conditional delete. Claims can attach managed disks to warm VMs with +durable single-writer exclusion. This builds on the existing +[read-only live volume API](live-volumes.md). + +This implements [issue #300](https://github.com/cocoonstack/sandbox/issues/300) +with these limits: + +- Explicit hibernate and wake support one mounted writable managed volume when + the node enables a workload hook. Idle hibernation skips volume claims, and + attach-only or multi-volume claims cannot sleep. +- Growth extends a volume that no sandbox holds. Growing a held volume answers + 409 naming the holder; online growth waits for cocoon to wrap Cloud + Hypervisor's disk resize. +- A claim naming a volume on another mesh node is redirected to that node. +- Archive, checkpoint, fork and promote refuse claims that hold volumes. + +## Configure one node + +Set a nonempty `api_token` and an absolute, clean `managed_volume_dir` in the +sandboxd configuration. The managed directory must be private to the daemon, +on persistent storage, and outside disposable VM working directories. Every +existing ancestor must be owned by root or the daemon user. The daemon creates +private `disks/`, `registry.json` and `lock` entries; it exclusively locks the +registry directory. + +Startup rejects managed stores inside `data_dir/readers`, `data_dir/goldens` +and top-level `data_dir/fork-*` trees before creating files. The check includes +existing symbolic links at `data_dir`, `readers` and `goldens`. A store such as +`data_dir/managed-volumes` remains valid. Keep these directories under trusted +host control; validation does not detect bind-mount aliases or prevent a +concurrent pathname swap. + +```json +{ + "api_token": "replace-with-your-root-token", + "managed_volume_dir": "/var/lib/sandboxd/managed-volumes", + "sync_claims": true +} +``` + +Volumes are raw sparse files. Put `managed_volume_dir` on xfs or btrfs where +possible, so later copies can use reflinks. `disk_reserve` does not watch this +directory when it is on another filesystem. Set `sync_claims`: the claim +journal is what keeps a writer's lock across a power loss. + +Keep the registry and backing files together. Do not resize, rename, replace or +hardlink a backing file by hand. Each managed resource records its immutable +ID, owner, node, capacity and backing device/inode/size. An identity mismatch +requires operator recovery and fails closed. The volume then reports state +`recovery_required`, and PUT and claims return `503` with code +`volume_recovery_required` and no `Retry-After`. If the backing file is gone, +DELETE with the identity `If-Match` drops the record of the unheld volume. A backing that +is present but replaced keeps refusing DELETE, so the daemon never removes a +file it did not create. + +The existing static catalog remains operator-owned. Its files and every path +component must be controlled by trusted operator or daemon principals. Static +catalog paths must not alias private managed storage, and a held name must +continue to identify the same backing. Stable symlink aliases and replacements +before attachment are checked. These checks precede the runtime's later path +open; they cannot defeat a concurrent host pathname swap. An untrusted actor +must not be able to change a catalog path component. + +## Manage a resource + +Use an explicit root or tenant management token. A sandbox token alone cannot +manage resources or call the existing live volume endpoints. Tenant tokens can +create and attach only their own managed resources; root must name an existing +tenant owner when creating one. + +| Request | Success | Refusal or retry behavior | +| --- | --- | --- | +| `PUT /v1/volumes/work` with `{"size_gib":20}` | `201`, `volume`, identity `ETag` | Identical retry returns `200` and the same identity; changed owner or capacity returns `409`. | +| `GET /v1/volumes` | Catalog and managed summaries visible to caller | Managed summaries include identity and owner; use individual GET for holders. | +| `GET /v1/volumes/work` | Managed detail with public holder IDs, modes and phases | Foreign and unknown names share `404`; host paths and process IDs stay private. | +| `POST /v1/volumes/work/grow` with identity `If-Match` and `{"size_gib":40}` | `200` with `volume` at the new size | The current size returns `200` unchanged; a smaller size or a held volume returns `409`; wrong identity returns `412`. | +| `DELETE /v1/volumes/work` with `If-Match: "vol_…"` | `204` | Missing precondition `428`; wrong identity `412`; held resource `409`. | + +Capacity is an integer from 1 through 1024 GiB. The backing is a raw sparse file; +logical capacity is not reserved physical space. Storage exhaustion can still +occur while guests write. Creation does not format or mount a filesystem. +Deletion keeps no record. Replaying a completed deletion returns `404`. +Replaying an old identity after a same-name resource is recreated returns `412` +and leaves the new resource in place. + +The Go SDK exposes `CreateVolume`, `GetVolume`, `GrowVolume`, and `DeleteVolume`. Managed detail +uses `ManagedVolumeInfo`; existing `VolumeInfo` remains comparable. Structured +`ManagedVolumeError` unwraps to `APIError`. Python exposes `create_volume`, +`get_volume`, `grow_volume`, and `delete_volume`; structured fields remain on `APIError`. +Both SDKs send names, sizes and IDs as given. The daemon is the only validator, +and its `400` comes back as `APIError`. + +## Format once, reuse across templates + +The initial raw disk requires explicit filesystem setup. This Python example +uses an attach-only claim, waits for the disk serial, formats it once and +unmounts it before release. Rerunning the example against an already formatted +resource deliberately fails the blank-disk assertion. + +```python +from cocoonsandbox import Client + +client = Client("127.0.0.1:7777", api_token="tenant-management-token") +volume = client.create_volume("work", 20) +with client.new("rt:24.04", volumes=[{"name": "work", "mode": "rw"}], mount=False) as sb: + device = sb.exec( + "sh", "-c", + 'for i in $(seq 1 200); do for p in /sys/block/*/serial; do ' + '[ "$(cat "$p")" = work ] && { echo "/dev/$(basename "$(dirname "$p")")"; exit 0; }; ' + 'done; sleep 0.01; done; exit 1', + ).strip() + sb.exec("sh", "-c", 'blkid "$1" >/dev/null 2>&1; [ "$?" = 2 ]', "probe", device) + sb.exec("mkfs.ext4", "-q", "-F", device) + sb.exec("mkdir", "-p", "/workspace") + sb.exec("mount", device, "/workspace") + sb.write_file("/workspace/marker", b"persistent") + sb.exec("sync", "-f", "/workspace/marker") + sb.exec("umount", "/workspace") + +with client.new("new-template:local", volumes=[{"name": "work", "mode": "rw", "mount": "/workspace"}]) as sb: + assert sb.read_file("/workspace/marker") == b"persistent" + +client.delete_volume("work", volume["id"]) +``` + +A reader excludes a writer, and a writer excludes both readers and other +writers. Multiple readers may share a ready disk. Live additions and removal +of managed entries are deferred, including mounted read-only managed entries. +A running managed claim can still change mounted read-only catalog entries +through the existing `GET/PUT/PATCH /v1/sandboxes/{id}/volumes` interface. +Attach-only managed entries retain their empty mount across those changes and +across daemon restart. + +## Grow an unheld volume + +Growth extends the raw backing file of a volume that no sandbox holds. Release +every claim of the volume first; a held volume returns `409` with code +`volume_busy` and its holder list. The target is an absolute size from 1 +through 1024 GiB. + +```python +volume = client.get_volume("work") +client.grow_volume("work", volume["id"], 40) +``` + +The equivalent HTTP request is: + +```http +POST /v1/volumes/work/grow +Authorization: Bearer tenant-management-token +If-Match: "vol_resource_identity" +Content-Type: application/json + +{"size_gib":40} +``` + +`200` returns the `volume` envelope at the new `size_bytes`. The identity ETag +does not change. Repeating the current size returns `200` and changes nothing, +so the same request is safe to retry after a lost response. A smaller size +returns `409`, and so does a volume whose growth is still in progress. A wrong +resource ID returns `412`, a missing `If-Match` returns `428`, and invalid JSON +or size returns `400`. Foreign and unknown volumes both return `404`. + +Growth does not extend a partition or filesystem. Extend the guest filesystem +after the next claim attaches the disk. + +The daemon records the target in the registry before it extends the file, and +commits the new size once the file is extended and synced. A daemon that stops +in between settles the volume at next start to whichever size the file has. A +failed final registry write returns `503` with `Retry-After`; the daemon keeps +the new size and retries the write. + +## Hibernate one mounted managed volume + +The node can enable explicit Hibernate and Wake for one managed writable disk +on a `net: none` claim. The applied, desired, and held volume sets must match. +Attach-only volumes, additional disks, catalog mixtures, egress, replication, +and archive remain unsupported. Idle hibernation continues to skip all volume +claims. Existing API and SDK methods take no new arguments. + +Install the same workload hook at an absolute path in every supported guest +image, together with GNU `timeout` and util-linux `flock` at `/usr/bin`. +Enable the hook in node configuration and restart sandboxd. + +```json +{ + "managed_volume_dir": "/var/lib/sandboxd/managed-volumes", + "managed_volume_hibernate": { + "hook_argv": ["/usr/local/bin/workload-volume"], + "hook_timeout_seconds": 10 + } +} +``` + +`hook_timeout_seconds` accepts 1 through 300. The daemon appends five separate +arguments to `hook_argv`. + +```text +prepare|resume operation_id volume_id volume_name exact_mount_path +``` + +`prepare` must stop workload writers and close their filesystem references +before returning success. The daemon then verifies the mount, unmounts without +force, detaches the disk, and captures VM memory. `resume` restarts writers +after the exact backing is attached and mounted. It also compensates for a +failed preparation. Both actions must be idempotent for the same operation ID. +The hook must not detach from its supervised process group or close inherited +lock descriptors while descendants continue working. + +Hook calls execute as structured argv under a guest lock at +`/run/sandboxd-volume-hook.lock`. GNU `timeout` kills the hook's process group +at its execution limit. Lock acquisition has the same time limit, and the host +RPC allows both limits plus a drain interval. A failed `prepare` fails the +hibernate: the daemon reattaches and remounts the disk if needed, runs `resume` +once, and the claim keeps running. Once the host has verified the mount, a +failed `resume` is logged and not retried, so the guest stays awake with its +writers stopped. A hook that deliberately escapes supervision is outside this +contract. + +A successful hibernate keeps the managed hold and backing file. Another claim +or a volume deletion receives 409 while the owner sleeps. Wake verifies the +backing's path, device, inode, and size before restoring memory. A missing or +replaced backing leaves the claim paused. Restore the original backing before +retrying wake. A restore that fails before any guest runs leaves the claim +hibernated, and the next wake restores from the same snapshot. A partial wake +keeps the restored VM alive and retries attach, mount, and resume on that same +VM, which cocoon identifies by VM ID and a transition generation past the one +recorded at hibernate. It never restores old memory over a running VM. If that +VM dies after its snapshot was consumed, recovery refuses rather than replaying +stale memory against changed disk contents. + +Existing data streams end when capture stops the VM. Explicit and data-plane +wake publish access only after remount and resume. Passive operator access +stays paused during incomplete transitions. Readers never serve the detached +snapshot. + +An effective outbound policy, including a proxy on the `none` lane, blocks managed +hibernation before it changes the running guest. Wake never consults outbound +policy, so a policy reload while the claim sleeps cannot strand it. Release and +lease expiry never wait for a hibernate or wake in flight: they remove the VM +and snapshot, the in-flight step fails at its next VM call, and the hold is +freed once cocoon confirms the VM gone. The managed disk is preserved. A claim +released or expired after prepare detached its disk leaves the disk clean, so +read-only claims can attach it at once; one torn down mid-prepare or mid-wake +needs a writable claim to replay it first. + +The sleep record, including the selected hook and operation ID, lives on the +claim's `claims.json` row. Retries use them even after a configuration change +or disabling new managed sleeps. Each step's intent lands in `claims.json` +before its effect, so a daemon crash resumes from the last step with either +value of `sync_claims`. With `sync_claims` off, host power loss can leave a +stale record; the claim may then be release-only. Once wake has verified the +remount, a failed final write is retried in the background and wake still +succeeds; a crash before it lands runs `resume` once more on the same VM. + +## Placement + +A volume lives on one node. Each node advertises a tenant-scoped key for each +of its managed volumes, never the name, so peers cannot list or probe another +tenant's volumes. A tenant claim that names a volume on a peer is redirected to +that peer, even from an entry node without `managed_volume_dir`. Another tenant +naming it gets the same 400 as an unknown name. When +the volume is local but no node holds both the volume and the requested +template, the claim answers 409. Root claims are not redirected to managed +volumes; claim through the owner tenant or on the owner node. + +PUT refuses with 409 when another mesh node already advertises the same owner's +name, and the message names that node's client address. The check reads the +gossiped view, so two PUTs for one name on different nodes within about one +gossip interval can both succeed. Create each volume from one place, and +delete the duplicate if that happens. + +## Ownership and recovery + +`claims.json` is the only journal of who holds a volume. The registry records +resources only: per volume its ID, owner, node, size, pending growth, backing +identity and state. A claim holds its volumes from admission until cocoon +confirms its VM gone, and restart derives the hold from the claim row, as it +does for catalog volumes. Release, lease expiry and egress quarantine follow +the ordinary claim paths and never write the registry. A VM that survives +removal stays queued for the reap tick and keeps its volumes held; a refused +claim reports that holder with state `releasing`. + +At startup, an unclaimed running `sbx-` VM that cannot be removed is queued the +same way, holding the disks cocoon reports attached. If its disks cannot be +inspected, it holds every managed name until a later tick inspects or removes +it. A claim whose VM is gone is dropped and its volumes freed. A failed VM +listing prevents startup and leaves claims, VMs and files untouched. + +A registry that cannot be parsed, or that disagrees with the backing +directory, prevents startup. A failed registry write keeps the new state in +memory and retries in the background. Until it lands, create, delete, growth +and new managed claims answer 503 with `Retry-After`, while running claims keep +their sockets and release normally. GET, listing, and placement checks do not +persist recovery state, so a temporary backing inspection error cannot +permanently disable a healthy volume. + +An interrupted allocation with no recorded backing inode stays `creating`. No +VM has seen that file, so repeating the same PUT replaces it with a fresh one, +and DELETE removes it. The registry has no format version. +Unknown keys are ignored and dropped on the next write; unknown lifecycle states +prevent startup. + +## Upgrade + +The registry and claim formats changed during development and have no +migration. Drain nodes that ran earlier builds of this branch (release every +claim that holds a managed volume) before upgrading. + +## Follow-up order + +1. Online growth of a held volume, once cocoon wraps Cloud Hypervisor's disk resize. +2. Archive export and restore of external data disks. +3. Idle hibernation and attach-only sleep of volume claims. +4. Kubernetes lifecycle and status ownership once the daemon contracts are stable. diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 31a255cd..11a6e14d 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -19,8 +19,10 @@ All bodies are JSON. Three token kinds: Endpoints below that say "node API token" accept root or tenant unless marked root-only. Errors are `{"error": "message"}` with the status codes -listed per endpoint. Request bodies are capped at 1 MiB; a larger one is -rejected with `400 {"error": "invalid request body"}`. +listed per endpoint. Managed volume routes also return a `code` field and +may return a `volume` name and `holders` array. Request bodies are capped +at 1 MiB; a larger one is rejected with 400. Managed routes use their +endpoint-specific error bodies. ## POST /v1/claim @@ -233,7 +235,15 @@ unclean `rw` release (needs recovery — one `rw` claim must replay and cleanly release before `ro` claims resume; see [deploy](deploy.md#writable-dataset-volumes)); 429 node at `max_claims`, the calling tenant at its own `max_claims`, or the node draining; 500 -provisioning failed. +provisioning failed. A tenant's managed volume on another mesh node redirects +the claim to that node, like a catalog volume; another tenant's managed volume +answers the same 400 as an unknown name. A local volume whose template no node +holds alongside it answers 409. Managed volumes use the managed volume error +body: a volume that is growing returns 409 `volume_busy`; +a missing or replaced backing returns 503 `volume_recovery_required` without +`Retry-After`, except while a hibernate or a growth owns the volume, when it +is a storage error; unavailable managed storage returns 503 +`volume_storage_unavailable` with `Retry-After`. ## GET /v1/volumes @@ -255,6 +265,115 @@ list; the field is emitted (as `true`) only for a writable entry this node declares and omitted otherwise (peer-only rows never carry it), so a read-only entry's response is byte-identical to v1. +When `managed_volume_dir` is enabled, this list also includes caller-visible +managed summaries. Individual +managed GET responses carry holder details. Catalog summaries retain their +existing shape. + +## PUT /v1/volumes/{name} + +Creates a node-local managed volume. Auth requires an explicit root or tenant +bearer token. A tenant token makes that tenant the owner. A root token must +name an existing tenant in `tenant`. A tenant cannot name another owner. +Every `/v1/volumes/{name}` route authenticates like the other client routes: +a missing, invalid, or sandbox token gets the same `401` body, and a tenant +store outage gets `503` with `Retry-After`. + +The strict JSON body requires `size_gib`, an integer from 1 through 1024. +`tenant` is optional for a tenant token. The name must start with a lowercase +letter and contain at most 19 more lowercase letters, digits, underscores, or +hyphens. It must not start with `cocoon-`. + +```json +{"size_gib":20} +``` + +`201` returns the `volume` envelope and an `ETag` containing the quoted, +immutable `volume.id`. For example: + +```json +{"volume":{"kind":"managed","id":"vol_1234567890abcdef1234567890abcdef", + "tenant":"acme","node":"node-1","state":"ready", + "created_at":"2026-10-20T00:00:00Z","name":"work", + "default_mount":"/volumes/work","size_bytes":21474836480, + "available":true,"nodes":1,"writable":true}} +``` + +An identical request for the same owner and size returns `200` with the same +ID and ETag. The volume is a sparse raw disk. Creation does not format a +filesystem. `holders` is present when sandboxes hold the volume. + +Other errors use `{"error":"message","code":"volume_code"}`. They include `400` +for an invalid body, name, size, or owner; `403` when a tenant selects another owner; `404` for another tenant's existing +volume or on a node without `managed_volume_dir`; `409` for a catalog name, different size or owner, pending deletion, +or a name the same owner already holds on another mesh node (the message names +that node's client address; gossip lag leaves a window of about one gossip +interval in which two nodes can both accept it); +and `503` for unavailable storage or tenant lookup. A create interrupted after +allocation stays `creating`; repeat the same PUT to finish it, or DELETE it. A +volume whose backing identity changed settles in state `recovery_required`, and +PUT and claims of it return `503` with code `volume_recovery_required`. +`volume_backing_mismatch` and `volume_recovery_required` omit `Retry-After`. Transient storage failures +include a retry hint. + +## GET /v1/volumes/{name} + +Auth requires an explicit root or tenant bearer token. Root can inspect any +managed volume. A tenant can inspect only its own. An unknown or foreign name +returns the same `404` body with code `volume_not_found`. + +`200` returns `{"volume": {...}}` and the quoted immutable ID in `ETag`. +`holders` contains public `sandbox_id`, `mode`, and `state` values when the +volume is held. Host backing paths and process IDs are absent. + +Other errors include `401` for a missing or invalid token and `503` when the +managed store or backing inspection is unavailable. A `503` includes +`Retry-After`. + +## DELETE /v1/volumes/{name} + +Auth requires an explicit root or tenant bearer token. Send the quoted ID +from the volume's `ETag` as `If-Match`. Root can delete any managed volume; +a tenant can delete only its own. + +`204` has no body. Deletion keeps no record, so repeating a completed deletion +returns `404`. A new resource with the same name has a new ID, so replaying the +old deletion returns `412` and does not remove it. + +Errors use the managed volume error body. They include `401` for a missing or +invalid token; `404` for an unknown, foreign, or already deleted name; `409` for a +held or growing volume; `412` for a malformed or mismatched `If-Match`; +`428` for a missing `If-Match`; and `503` for unavailable storage or backing +identity that requires operator recovery. A busy response has code +`volume_busy` and includes the volume name and public holder list. An unheld +volume whose backing file is gone is deleted with `204`, which is the API exit +from `recovery_required`. A backing that is present but replaced returns +`volume_backing_mismatch` without `Retry-After` and keeps the record; +transient storage failures include a retry hint. + +## POST /v1/volumes/{name}/grow + +Auth requires an explicit root or tenant management token. The node owning the +managed volume handles the request. `If-Match` contains its quoted immutable +resource ID. The strict JSON body requires `size_gib`, the absolute target from +1 through 1024. + +```json +{"size_gib":40} +``` + +`200` returns the `volume` envelope at the new `size_bytes`; the ETag remains +the resource ID. Growth applies only to a volume no sandbox holds. The current +size returns `200` without a change, so a lost response is safe to retry. +Growth does not extend a filesystem. + +Errors are `400` invalid body, name, or size; `401` missing or invalid token; +`404` foreign or unknown volume; `409` a smaller size, a held volume (code +`volume_busy` with the public holder list), a growth in progress, or a volume +being created or deleted; `412` wrong identity; `428` missing `If-Match`; and +`503` unavailable storage with `Retry-After`. See +[Grow an unheld volume](managed-volumes.md#grow-an-unheld-volume). + ## GET /v1/sandboxes/{id}/volumes Returns the persisted desired set (`volumes_want`), confirmed applied set @@ -396,12 +515,21 @@ freeing its memory; the next agent access restores it transparently snapshot point and the stop coincident), except a read-only file call that a [reader](#get-v1sandboxesidagent) serves without waking it. Idempotent on an already-hibernated sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and -snapshot) at its deadline, unless it claimed `on_expire: archive` or its pool -archives. When to hibernate is the caller's policy — the -node only provides the transition. 204 on success, 404 unknown id or wrong -token, 409 on the egress lane or when volumes are attached (neither kind of -sandbox hibernates; see [egress](egress.md)), 429 while the disk is under -[`disk_reserve`](deploy.md). +snapshot) at its deadline. Claims without volume holds are excepted when they +claimed `on_expire: archive` or their pool archives. Hibernated managed-volume +claims expire through release, which preserves the backing disk. When to hibernate +is the caller's policy. The node only provides the transition. 204 on success, 404 unknown id or wrong +token, 409 on the egress lane or unsupported volume claims, 429 while the disk is under +[`disk_reserve`](deploy.md). An opt-in node hook supports one mounted writable +managed volume on the `none` lane. See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume) +for hook installation, exclusive ownership, and failure recovery. Other volume +capture operations remain blocked. A managed hibernate also returns 409 when a +guest mount blocks the unmount (close guest references and retry), 409 +`volume_hook_failed` with the hook's own exit code and output when the guest +`prepare` hook exits non-zero (a host failure to reach the guest is a plain +500), 409 +`managed_sleep_recovery_required` when the disk could not be reattached and +remounted after a failed preparation, and 503 without `Retry-After` below Cocoon v0.7.2. ## POST /v1/sandboxes/{id}/wake @@ -416,6 +544,8 @@ while stored, and waking it starts a fresh lease of the length the claim was gra its own disk (memory is lost) and its restart budget resets. 204 on success, 404 unknown id or wrong token, 409 a failed egress-lane or volume sandbox, or one whose VM record is gone, 500 when the cold boot itself fails. +A managed wake also returns 409 when a guest mount conflicts with the remount +and 503 without `Retry-After` below Cocoon v0.7.2. ## POST /v1/sandboxes/{id}/renew diff --git a/docs/sdk-python.md b/docs/sdk-python.md index d726d366..12d6cdd2 100644 --- a/docs/sdk-python.md +++ b/docs/sdk-python.md @@ -21,6 +21,9 @@ wire is chunked, the caller's copy is not). Wire fidelity is pinned by the shared protocol fixture corpus that the Rust guest, Go, and Python all round-trip in CI. +For node-local `create_volume`, `get_volume`, `delete_volume` and the format-once +workflow, see [managed volumes](managed-volumes.md). + ## A complete example Claim a sandbox, push a project in, run a build, then freeze the built state @@ -123,7 +126,9 @@ branched). `sb.template_digest` is the exact content identity when the claim cloned a promoted template; it is empty for other sources. `Sandbox` is a context manager; `sb.close()` releases it (releasing one already gone is not an error — double-release and reap races stay silent). -Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Checkpoint +Volume sandboxes cannot fork, checkpoint, or promote. Explicit hibernate supports +one mounted managed writable volume when the node enables a workload hook. +See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume). Checkpoint branches do not accept volumes of either mode in this version. `mount=False` on `client.new` or `template.new` claims the same volumes diff --git a/docs/sdk.md b/docs/sdk.md index 2d66f2aa..f951e28a 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -9,6 +9,9 @@ module). One `Client` talks to one entry node; sandbox handles dial their owning node directly, so a client works unchanged against a single node or a cluster. +For node-local `CreateVolume`, `GetVolume`, `DeleteVolume`, managed detail +types and structured errors, see [managed volumes](managed-volumes.md). + ## A complete example Claim a sandbox, push a project into it, run a build, then freeze the built @@ -227,7 +230,9 @@ promoted template; it is empty for other sources. `Owner()` names the owning node, and `Token()` returns the per-sandbox bearer to persist with `ID` for a later `Lookup`; `Close()` releases the sandbox (releasing one already gone is not an error, and `Close` is bounded internally so it stays defer-friendly). -Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Passing +Volume sandboxes cannot fork, checkpoint, or promote. Explicit hibernate supports +one mounted managed writable volume when the node enables a workload hook. +See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume). Passing `WithVolumes` or `WithClaimRef` to `Checkpoint.New` returns a local error: checkpoint branches support neither volumes nor a claim reference in this version. `WithMetadata`, `WithArchiveOnExpire` and `WithEnv` apply to `New`, `Template.New`, and diff --git a/docs/security.md b/docs/security.md index 6c519f49..06207441 100644 --- a/docs/security.md +++ b/docs/security.md @@ -130,6 +130,28 @@ to every other tenant's next read. A dataset that genuinely needs multiple writers needs an out-of-band process for who writes when, which the catalog ACL does not provide. +Managed volumes are node-local tenant resources, separate from the operator's +catalog. Management requires an explicit root or tenant bearer token; a +sandbox token cannot create, inspect, grow, or delete a resource. Tenant +listings omit other tenants' resources. Individual reads and mutations return +the same 404 for a foreign or unknown volume. Root can manage resources for +any tenant, but must name an existing tenant when creating one. API responses +expose resource IDs and public holder IDs, modes, and states. They do not +expose backing paths or process IDs. + +The managed directory and its ancestors must stay under trusted host control. +The daemon records the backing file's device, inode, and size; an identity +mismatch blocks use and requires operator recovery. Deletion requires the +quoted immutable resource ID in `If-Match`. A replay of a completed deletion +cannot delete a new resource created under the same name. A held volume +cannot be deleted, and only its tenant or root can request deletion. + +Growth refuses a held volume, so a running guest never sees its disk change +size. The registry records the target before the file is extended, and a +restart settles an interrupted growth from the file's size. Repeating the +current size is a no-op, so a lost growth response is safe to retry. See +[Tenant managed persistent volumes](managed-volumes.md#grow-an-unheld-volume). + ## Known limitations Facts to plan around, stated so the boundary is honest: diff --git a/e2e/fakeengine_test.go b/e2e/fakeengine_test.go index 9bbc9113..74c767d7 100644 --- a/e2e/fakeengine_test.go +++ b/e2e/fakeengine_test.go @@ -47,6 +47,18 @@ func newFakeEngine(dir string) *fakeEngine { func (f *fakeEngine) CheckLiveVolumeSupport(context.Context) error { return nil } +func (f *fakeEngine) RunVolumeHook(context.Context, string, engine.VolumeHookCall) error { + return errors.New("fakeEngine has no volume hook") +} + +func (f *fakeEngine) VerifyMountedVolume(context.Context, string, string, string) error { + return errors.New("fakeEngine cannot verify mounted volumes") +} + +func (f *fakeEngine) InspectHibernateSource(context.Context, string) (engine.HibernateSource, bool, error) { + return engine.HibernateSource{}, false, errors.New("fakeEngine cannot inspect hibernate sources") +} + func (f *fakeEngine) Clone(_ context.Context, _, name string, _ types.PoolKey) (types.VMRecord, error) { return f.createRecord(name) } diff --git a/e2e/managed_volume_acceptance_test.go b/e2e/managed_volume_acceptance_test.go new file mode 100644 index 00000000..cd4cdb52 --- /dev/null +++ b/e2e/managed_volume_acceptance_test.go @@ -0,0 +1,86 @@ +package e2e + +import ( + "encoding/json" + "fmt" + "net/http" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/server" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func warmTotal(st *stack) int { + total := 0 + for _, n := range st.mgr.WarmCounts() { + total += n + } + return total +} + +// TestManagedVolumeIssue300Acceptance walks the acceptance line of issue #300 over HTTP. +func TestManagedVolumeIssue300Acceptance(t *testing.T) { + other := types.PoolKey{Template: "rt:22.04", Net: types.NetNone, Size: types.SizeSmall} + cfg := &config.Config{ + ManagedVolumeDir: filepath.Join(t.TempDir(), "managed"), + Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 1}, {PoolKey: other, Warm: 1}}, + } + st := startConfigStack(t, "root-token", cfg) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}}); err != nil { + t.Fatal(err) + } + if c := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":20}`, ""); c.status != http.StatusCreated { + t.Fatalf("create 20 GiB for t1: %d %s", c.status, c.body) + } + deadline := time.Now().Add(5 * time.Second) + for warmTotal(st) < 2 && time.Now().Before(deadline) { + time.Sleep(10 * time.Millisecond) + } + if warmTotal(st) < 2 { + t.Fatalf("warm pools never filled: %v", st.mgr.WarmCounts()) + } + claimBody := func(template string) string { + return fmt.Sprintf(`{"template":%q,"net":"none","size":"small","volumes":[{"name":"work","mode":"rw","attach_only":true}]}`, template) + } + warmBefore := st.mgr.Counters().ClaimsWarm + first := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t1-token", claimBody("rt:24.04"), "") + var sb server.ClaimResponse + if err := json.Unmarshal(first.body, &sb); first.status != http.StatusOK || err != nil || sb.ID == "" { + t.Fatalf("warm rw claim: %d %s", first.status, first.body) + } + if st.mgr.Counters().ClaimsWarm != warmBefore+1 { + t.Fatalf("rw claim with a managed volume was not served from the warm pool: counters=%+v", st.mgr.Counters()) + } + second := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t1-token", claimBody("rt:22.04"), "") + if second.status != http.StatusConflict || second.retryAfter != "" || !strings.Contains(string(second.body), `"sandbox_id":"`+sb.ID+`"`) { + t.Fatalf("second rw claim: %d Retry-After=%q %s, want 409 naming %s without Retry-After", second.status, second.retryAfter, second.body, sb.ID) + } + if r := requestManagedVolume(t, st, http.MethodPost, "/v1/sandboxes/"+sb.ID+"/release", sb.Token, "", ""); r.status != http.StatusOK && r.status != http.StatusNoContent { + t.Fatalf("release: %d %s", r.status, r.body) + } + if g := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", ""); g.status != http.StatusOK { + t.Fatalf("volume after release: %d %s, want it kept", g.status, g.body) + } + again := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t1-token", claimBody("rt:22.04"), "") + if again.status != http.StatusOK { + t.Fatalf("claim of another template with the same volume: %d %s", again.status, again.body) + } + foreignGet := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t2-token", "", "") + unknownGet := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/nope", "t2-token", "", "") + if foreignGet.status != http.StatusNotFound || string(foreignGet.body) != string(unknownGet.body) { + t.Fatalf("t2 GET: %d %s; unknown: %d %s; want identical 404", foreignGet.status, foreignGet.body, unknownGet.status, unknownGet.body) + } + if list := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes", "t2-token", "", ""); strings.Contains(string(list.body), `"work"`) { + t.Fatalf("t2 list shows t1's volume: %s", list.body) + } + foreignClaim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", claimBody("rt:24.04"), "") + unknownClaim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", strings.Replace(claimBody("rt:24.04"), `"work"`, `"nope"`, 1), "") + if foreignClaim.status != http.StatusBadRequest || string(foreignClaim.body) != string(unknownClaim.body) { + t.Fatalf("t2 claim: %d %s; unknown: %d %s; want identical 400", foreignClaim.status, foreignClaim.body, unknownClaim.status, unknownClaim.body) + } +} diff --git a/e2e/managed_volume_http_test.go b/e2e/managed_volume_http_test.go new file mode 100644 index 00000000..d59e0b69 --- /dev/null +++ b/e2e/managed_volume_http_test.go @@ -0,0 +1,434 @@ +package e2e + +import ( + "encoding/json/v2" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/tenants" +) + +func TestManagedVolumeHTTPIdentityAndConditionalDelete(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":20}`, "") + if created.status != http.StatusCreated { + t.Fatalf("PV-M01 create status=%d body=%s, want 201", created.status, created.body) + } + volume := decodeManagedVolume(t, created.body) + if volume.ID == "" || volume.Name != "work" || volume.Tenant != "t1" || volume.Kind != "managed" || + volume.SizeBytes != 20<<30 || volume.State != "ready" { + t.Fatalf("PV-M01 unexpected resource projection: %+v", volume) + } + if created.etag != `"`+volume.ID+`"` { + t.Fatalf("PV-M01 etag=%q, want resource identity", created.etag) + } + for _, body := range []string{`{"size_gib":20}`, `{"tenant":"t1","size_gib":20}`} { + repeated := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", body, "") + if repeated.status != http.StatusOK || decodeManagedVolume(t, repeated.body).ID != volume.ID { + t.Fatalf("PV-M02 create retry status=%d body=%s", repeated.status, repeated.body) + } + } + changed := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":21}`, "") + if changed.status != http.StatusConflict { + t.Fatalf("PV-M03 changed capacity status=%d, want 409", changed.status) + } + got := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if got.status != http.StatusOK || got.etag != created.etag || decodeManagedVolume(t, got.body).ID != volume.ID { + t.Fatalf("PV-M02 inspect status=%d body=%s", got.status, got.body) + } + for _, tc := range []struct { + etag string + want int + }{ + {want: http.StatusPreconditionRequired}, + {etag: `"vol_00000000000000000000000000000000"`, want: http.StatusPreconditionFailed}, + } { + refused := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", tc.etag) + if refused.status != tc.want { + t.Fatalf("PV-M10 delete precondition status=%d, want %d", refused.status, tc.want) + } + } + deleted := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", created.etag) + if deleted.status != http.StatusNoContent { + t.Fatalf("PV-M09 delete status=%d body=%s", deleted.status, deleted.body) + } + replay := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", created.etag) + if replay.status != http.StatusNotFound { + t.Fatalf("PV-M10 delete replay status=%d body=%s, want 404", replay.status, replay.body) + } + replacement := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":20}`, "") + if replacement.status != http.StatusCreated || decodeManagedVolume(t, replacement.body).ID == volume.ID { + t.Fatalf("PV-M10 replacement lacks a fresh identity: status=%d body=%s", replacement.status, replacement.body) + } + stale := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", created.etag) + if stale.status != http.StatusPreconditionFailed { + t.Fatalf("PV-M10 old deletion replay over a replacement status=%d, want 412", stale.status) + } + survivor := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if survivor.status != http.StatusOK || decodeManagedVolume(t, survivor.body).ID != decodeManagedVolume(t, replacement.body).ID { + t.Fatalf("PV-M10 stale delete changed replacement: status=%d body=%s", survivor.status, survivor.body) + } +} + +func TestManagedVolumeHTTPGrowOffline(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("create status=%d body=%s", created.status, created.body) + } + grown := requestManagedVolume(t, st, http.MethodPost, "/v1/volumes/work/grow", "t1-token", `{"size_gib":2}`, created.etag) + if grown.status != http.StatusOK || grown.etag != created.etag || decodeManagedVolume(t, grown.body).SizeBytes != 2<<30 { + t.Fatalf("grow status=%d etag=%q body=%s, want 200 at 2 GiB", grown.status, grown.etag, grown.body) + } + if missing := requestManagedVolume(t, st, http.MethodPost, "/v1/volumes/work/grow", "t1-token", `{"size_gib":3}`, ""); missing.status != http.StatusPreconditionRequired { + t.Fatalf("grow without If-Match status=%d, want 428", missing.status) + } + if stale := requestManagedVolume(t, st, http.MethodPost, "/v1/volumes/work/grow", "t1-token", `{"size_gib":3}`, `"vol_00000000000000000000000000000000"`); stale.status != http.StatusPreconditionFailed { + t.Fatalf("grow with a stale ID status=%d, want 412", stale.status) + } + if got := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", ""); decodeManagedVolume(t, got.body).SizeBytes != 2<<30 { + t.Fatalf("refused grows changed capacity: %s", got.body) + } +} + +func TestManagedVolumeHTTPAuthorizationAndPrivacy(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/private", "t1-token", `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("PV-M04 fixture create status=%d body=%s", created.status, created.body) + } + for _, method := range []string{http.MethodGet, http.MethodPut, http.MethodDelete} { + foreign := requestManagedVolume(t, st, method, "/v1/volumes/private", "t2-token", `{"size_gib":1}`, created.etag) + if foreign.status != http.StatusNotFound { + t.Errorf("PV-M04/PV-M08 foreign %s status=%d, want 404", method, foreign.status) + } + if method == http.MethodGet { + missing := requestManagedVolume(t, st, method, "/v1/volumes/unknown", "t2-token", "", "") + if foreign.status != missing.status || string(foreign.body) != string(missing.body) { + t.Errorf("PV-M04 foreign and unknown inspection differ: %d/%d", foreign.status, missing.status) + } + } + assertManagedReplyPrivate(t, foreign.body) + } + listing := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes", "t2-token", "", "") + if listing.status != http.StatusOK || strings.Contains(string(listing.body), `"private"`) { + t.Fatalf("PV-M04 foreign listing status=%d body=%s", listing.status, listing.body) + } + root := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/private", "root-token", "", "") + if root.status != http.StatusOK || decodeManagedVolume(t, root.body).Tenant != "t1" { + t.Fatalf("PV-M04 root inspection status=%d body=%s", root.status, root.body) + } + for _, token := range []string{"", "invalid-token"} { + for _, method := range []string{http.MethodGet, http.MethodPut, http.MethodDelete} { + reply := requestManagedVolume(t, st, method, "/v1/volumes/private", token, `{"size_gib":1}`, created.etag) + if reply.status != http.StatusUnauthorized { + t.Errorf("PV-M05 invalid credential %s status=%d, want 401", method, reply.status) + } + assertManagedReplyPrivate(t, reply.body) + } + } + claimStatus, claim := rawClaim(t, st, "") + if claimStatus != http.StatusOK { + t.Fatalf("PV-M05 sandbox credential fixture status=%d", claimStatus) + } + sandboxAttempt := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/private", claim.Token, "", "") + if sandboxAttempt.status != http.StatusUnauthorized { + t.Errorf("PV-M05 sandbox credential grants volume management: status=%d", sandboxAttempt.status) + } + released := requestManagedVolume(t, st, http.MethodPost, "/v1/sandboxes/"+claim.ID+"/release", claim.Token, "", "") + if released.status != http.StatusNoContent { + t.Fatalf("PV-M05 release credential fixture status=%d", released.status) + } + override := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/override", "t1-token", `{"size_gib":1,"tenant":"t2"}`, "") + if override.status != http.StatusForbidden { + t.Errorf("PV-M06 tenant override status=%d, want 403", override.status) + } + invalidRoot := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/root-owned", "root-token", `{"size_gib":1}`, "") + if invalidRoot.status != http.StatusBadRequest { + t.Errorf("PV-M06 root missing tenant status=%d, want 400", invalidRoot.status) + } + rootCreate := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/two", "root-token", `{"size_gib":1,"tenant":"t2"}`, "") + if rootCreate.status != http.StatusCreated || decodeManagedVolume(t, rootCreate.body).Tenant != "t2" { + t.Fatalf("PV-M06 root explicit tenant status=%d body=%s", rootCreate.status, rootCreate.body) + } + deleted := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/private", "t1-token", "", created.etag) + if deleted.status != http.StatusNoContent { + t.Fatalf("PV-M09 fixture deletion status=%d body=%s", deleted.status, deleted.body) + } + foreignReplay := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/private", "t2-token", "", created.etag) + if foreignReplay.status != http.StatusNotFound { + t.Fatalf("PV-M05/PV-M10 deletion ID incorrectly grants authority: status=%d", foreignReplay.status) + } +} + +func TestManagedVolumeHTTPRejectsInputAndSerializesCreation(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + for _, body := range []string{ + `{}`, `{"size_gib":0}`, `{"size_gib":-1}`, `{"size_gib":1025}`, + `{"size_gib":1.5}`, `{"size_gib":9223372036854775808}`, + `{"size_gib":1,"unknown":true}`, `{"size_gib":1} {}`, `{"size_gib":`, + } { + reply := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/invalid", "t1-token", body, "") + if reply.status != http.StatusBadRequest { + t.Errorf("PV-M07 input %q status=%d, want 400", body, reply.status) + } + } + oversized := `{"size_gib":1,"padding":"` + strings.Repeat("x", 1<<20) + `"}` + tooLarge := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/oversized", "t1-token", oversized, "") + if tooLarge.status != http.StatusBadRequest { + t.Errorf("PV-M07 oversized body status=%d, want 400", tooLarge.status) + } + for _, name := range []string{".bad", "cocoon-root"} { + reply := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/"+name, "t1-token", `{"size_gib":1}`, "") + if reply.status != http.StatusBadRequest { + t.Errorf("PV-M07/PV-M08 name %q status=%d, want 400", name, reply.status) + } + } + const callers = 8 + replies := make([]managedHTTPReply, callers) + var group sync.WaitGroup + for i := range replies { + group.Go(func() { + replies[i] = requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/race", "t1-token", `{"size_gib":1}`, "") + }) + } + group.Wait() + var id string + var created int + for _, reply := range replies { + if reply.status != http.StatusOK && reply.status != http.StatusCreated { + t.Fatalf("PV-M12 concurrent create status=%d body=%s", reply.status, reply.body) + } + volume := decodeManagedVolume(t, reply.body) + if id != "" && volume.ID != id { + t.Fatalf("PV-M12 competing creates returned different identities") + } + id = volume.ID + if reply.status == http.StatusCreated { + created++ + } + } + if created != 1 { + t.Fatalf("PV-M12 got %d creations, want exactly one", created) + } +} + +func TestManagedVolumeHTTPRequiresExplicitCredential(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + missing := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/implicit", "", `{"size_gib":1,"tenant":"t1"}`, "") + if missing.status != http.StatusUnauthorized { + t.Fatalf("PV-M05 legacy empty root configuration grants management without credentials: status=%d", missing.status) + } + for _, route := range []string{"/v1/volumes", "/v1/volumes/implicit"} { + for _, token := range []string{"", "invalid-token"} { + got := requestManagedVolume(t, st, http.MethodGet, route, token, "", "") + if got.status != http.StatusUnauthorized { + t.Fatalf("PV-M05 open legacy root configuration grants discovery: status=%d", got.status) + } + } + } +} + +func TestManagedVolumeHTTPRecoveryRequiredOmitsRetryAfter(t *testing.T) { + dir := filepath.Join(t.TempDir(), "managed") + var cfg config.Config + if err := json.Unmarshal(fmt.Appendf(nil, `{"managed_volume_dir":%q}`, dir), &cfg); err != nil { + t.Fatal(err) + } + st := startConfigStack(t, "root-token", &cfg) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}}); err != nil { + t.Fatal(err) + } + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("create status=%d body=%s", created.status, created.body) + } + backing := filepath.Join(dir, "disks", decodeManagedVolume(t, created.body).ID+".raw") + // build the replacement while the original still exists, so the filesystem cannot reuse its inode + replacement := backing + ".new" + if err := os.WriteFile(replacement, nil, 0o600); err != nil { + t.Fatal(err) + } + if err := os.Truncate(replacement, 1<<30); err != nil { + t.Fatal(err) + } + if err := os.Rename(replacement, backing); err != nil { + t.Fatal(err) + } + claim := `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"work","mode":"rw"}]}` + for _, step := range []struct{ method, route, body string }{ + {http.MethodPost, "/v1/claim", claim}, + {http.MethodPut, "/v1/volumes/work", `{"size_gib":1}`}, + {http.MethodPut, "/v1/volumes/work", `{"size_gib":1}`}, + {http.MethodPost, "/v1/claim", claim}, + } { + reply := requestManagedVolume(t, st, step.method, step.route, "t1-token", step.body, "") + var failure struct { + Code string `json:"code"` + } + if err := json.Unmarshal(reply.body, &failure); err != nil { + t.Fatal(err) + } + if reply.status != http.StatusServiceUnavailable || failure.Code != "volume_recovery_required" || reply.retryAfter != "" { + t.Fatalf("%s %s after backing replacement: status=%d Retry-After=%q body=%s, want 503 volume_recovery_required without a retry hint", step.method, step.route, reply.status, reply.retryAfter, reply.body) + } + } + got := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if state := decodeManagedVolume(t, got.body).State; state != "recovery_required" { + t.Fatalf("GET state %q, want recovery_required", state) + } +} + +func TestManagedVolumeHTTPMissingDirectoryIsTransient(t *testing.T) { + dir := filepath.Join(t.TempDir(), "managed") + var cfg config.Config + if err := json.Unmarshal(fmt.Appendf(nil, `{"managed_volume_dir":%q}`, dir), &cfg); err != nil { + t.Fatal(err) + } + st := startConfigStack(t, "root-token", &cfg) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}}); err != nil { + t.Fatal(err) + } + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("create status=%d body=%s", created.status, created.body) + } + claim := `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"work","mode":"ro"}]}` + for _, moved := range []string{filepath.Join(dir, "disks"), dir} { + if err := os.Rename(moved, moved+".aside"); err != nil { + t.Fatal(err) + } + for _, step := range []struct{ method, route, body string }{ + {http.MethodPost, "/v1/claim", claim}, + {http.MethodPut, "/v1/volumes/work", `{"size_gib":1}`}, + {http.MethodPost, "/v1/claim", claim}, + } { + reply := requestManagedVolume(t, st, step.method, step.route, "t1-token", step.body, "") + var failure struct { + Code string `json:"code"` + } + if err := json.Unmarshal(reply.body, &failure); err != nil { + t.Fatal(err) + } + if reply.status != http.StatusServiceUnavailable || failure.Code != "volume_storage_unavailable" || reply.retryAfter == "" { + t.Fatalf("%s %s with %s moved aside: status=%d Retry-After=%q body=%s, want 503 volume_storage_unavailable with a retry hint", step.method, step.route, filepath.Base(moved), reply.status, reply.retryAfter, reply.body) + } + } + if err := os.Rename(moved+".aside", moved); err != nil { + t.Fatal(err) + } + replay := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":1}`, "") + if replay.status != http.StatusOK || decodeManagedVolume(t, replay.body).State != "ready" { + t.Fatalf("PUT after restoring %s: status=%d body=%s, want 200 ready", filepath.Base(moved), replay.status, replay.body) + } + } +} + +func TestManagedVolumeHTTPWithoutManagedStorageAnswersNotFound(t *testing.T) { + st := startConfigStack(t, "root-token", &config.Config{}) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}}); err != nil { + t.Fatal(err) + } + unknown := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if unknown.status != http.StatusNotFound { + t.Fatalf("GET on a node without managed storage: %d %s, want 404", unknown.status, unknown.body) + } + for _, step := range []struct{ method, route, body string }{ + {http.MethodPut, "/v1/volumes/work", `{"size_gib":1}`}, + {http.MethodDelete, "/v1/volumes/work", ""}, + {http.MethodPost, "/v1/volumes/work/grow", `{"size_gib":2}`}, + } { + reply := requestManagedVolume(t, st, step.method, step.route, "t1-token", step.body, `"vol_0123456789abcdef0123456789abcdef"`) + if reply.status != http.StatusNotFound || reply.retryAfter != "" || string(reply.body) != string(unknown.body) { + t.Errorf("%s %s on a node without managed storage: %d Retry-After=%q %s, want the GET 404 %s", step.method, step.route, reply.status, reply.retryAfter, reply.body, unknown.body) + } + } +} + +type managedHTTPReply struct { + status int + body []byte + etag string + retryAfter string +} + +type managedHTTPVolume struct { + ID string `json:"id"` + Name string `json:"name"` + Tenant string `json:"tenant"` + Kind string `json:"kind"` + State string `json:"state"` + SizeBytes int64 `json:"size_bytes"` +} + +func startManagedHTTPStack(t *testing.T, rootToken string) *stack { + t.Helper() + var cfg config.Config + body := fmt.Sprintf(`{"managed_volume_dir":%q}`, filepath.Join(t.TempDir(), "managed")) + if err := json.Unmarshal([]byte(body), &cfg); err != nil { + t.Fatal(err) + } + st := startConfigStack(t, rootToken, &cfg) + if rootToken == "" { + return st + } + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{ + {Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}, + }); err != nil { + t.Fatalf("configure tenants: %v", err) + } + return st +} + +func requestManagedVolume(t *testing.T, st *stack, method, route, token, body, etag string) managedHTTPReply { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), method, "http://"+st.addr+route, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + req.Header.Set("Content-Type", "application/json") + if etag != "" { + req.Header.Set("If-Match", etag) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("%s %s: %v", method, route, err) + } + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + assertManagedReplyPrivate(t, data) + return managedHTTPReply{status: resp.StatusCode, body: data, etag: resp.Header.Get("ETag"), retryAfter: resp.Header.Get("Retry-After")} +} + +func decodeManagedVolume(t *testing.T, body []byte) managedHTTPVolume { + t.Helper() + var reply struct { + Volume managedHTTPVolume `json:"volume"` + } + if err := json.Unmarshal(body, &reply); err != nil { + t.Fatalf("decode managed volume: %v", err) + } + return reply.Volume +} + +func assertManagedReplyPrivate(t *testing.T, body []byte) { + t.Helper() + for _, secret := range []string{"root-token", "t1-token", "t2-token", "/tmp/", "disk.raw", "registry.json"} { + if strings.Contains(string(body), secret) { + t.Errorf("managed API exposed private host or credential detail") + } + } +} diff --git a/e2e/managed_volume_placement_test.go b/e2e/managed_volume_placement_test.go new file mode 100644 index 00000000..026a3f7d --- /dev/null +++ b/e2e/managed_volume_placement_test.go @@ -0,0 +1,127 @@ +package e2e + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "net/http" + "net/http/httptest" + "slices" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/server" + "github.com/cocoonstack/sandbox/sandboxd/tenants" +) + +func managedGossipKey(tenant, name string) string { + sum := sha256.Sum256([]byte("volume|" + tenant + "|" + name)) + return "managed/" + hex.EncodeToString(sum[:16]) +} + +type managedOwnerPlacer struct { + edgePlacer + key string + template bool +} + +func (p *managedOwnerPlacer) owners(names []string) []string { + if slices.Contains(names, p.key) { + return []string{"peer:7777"} + } + return nil +} + +func (p *managedOwnerPlacer) VolumeCandidates(_ string, names []string) []string { + return p.owners(names) +} + +func (p *managedOwnerPlacer) VolumeOwners(names []string) []string { return p.owners(names) } + +func (p *managedOwnerPlacer) TemplateVolumeOwners(_ string, names []string) []string { + return p.owners(names) +} + +func (p *managedOwnerPlacer) TemplateOwners(string) []string { + if p.template { + return []string{"peer:7777"} + } + return nil +} + +func placedManagedStack(t *testing.T, placer server.Placer) *stack { + t.Helper() + return placeStack(t, startManagedHTTPStack(t, "root-token"), placer) +} + +func placeStack(t *testing.T, st *stack, placer server.Placer) *stack { + t.Helper() + srv := server.New(st.token, "", st.mgr, st.eng.real, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(ts.Close) + t.Cleanup(srv.CloseRelays) + probe := *st + probe.addr = strings.TrimPrefix(ts.URL, "http://") + return &probe +} + +func TestManagedVolumeClaimRedirectsToOwnerNode(t *testing.T) { + entries := map[string]func(*testing.T) *stack{ + "entry with managed storage": func(t *testing.T) *stack { return startManagedHTTPStack(t, "root-token") }, + "entry without managed storage": func(t *testing.T) *stack { + st := startConfigStack(t, "root-token", &config.Config{}) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}}); err != nil { + t.Fatal(err) + } + return st + }, + } + for name, entry := range entries { + t.Run(name, func(t *testing.T) { + st := placeStack(t, entry(t), &managedOwnerPlacer{key: managedGossipKey("t1", "work")}) + claim := `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"%s","mode":"rw"}]}` + reply := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t1-token", strings.Replace(claim, "%s", "work", 1), "") + var resp server.ClaimResponse + _ = json.Unmarshal(reply.body, &resp) + if reply.status != http.StatusOK || len(resp.Redirect) == 0 { + t.Fatalf("owner tenant claim of a peer's managed volume: %d %s, want a redirect to the owner node", reply.status, reply.body) + } + foreign := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", strings.Replace(claim, "%s", "work", 1), "") + unknown := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", strings.Replace(claim, "%s", "nope", 1), "") + if foreign.status != http.StatusBadRequest || string(foreign.body) != string(unknown.body) { + t.Fatalf("other tenant: %d %s; unknown: %d %s; want identical 400 replies", foreign.status, foreign.body, unknown.status, unknown.body) + } + }) + } +} + +func TestManagedVolumeCreateRefusesANamePeerAlreadyHolds(t *testing.T) { + st := placedManagedStack(t, &managedOwnerPlacer{internal: "peer:7777", public: "https://peer.example", key: managedGossipKey("t1", "work")}) + for _, c := range []struct{ token, body string }{ + {"t1-token", `{"size_gib":1}`}, + {"root-token", `{"size_gib":1,"tenant":"t1"}`}, + } { + reply := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", c.token, c.body, "") + if reply.status != http.StatusConflict || reply.retryAfter != "" || !strings.Contains(string(reply.body), "https://peer.example") { + t.Fatalf("%s create of t1's work held on peer:7777: %d Retry-After=%q %s, want 409 naming https://peer.example", c.token, reply.status, reply.retryAfter, reply.body) + } + } + if got := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "root-token", "", ""); got.status != http.StatusNotFound { + t.Fatalf("refused create left a local volume: %d %s", got.status, got.body) + } + if other := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t2-token", `{"size_gib":1}`, ""); other.status != http.StatusCreated { + t.Fatalf("t2 create of a name only t1 holds on a peer: %d %s, want 201", other.status, other.body) + } +} + +func TestManagedVolumeClaimFailsClearlyWhenNoNodeHasTemplateAndVolume(t *testing.T) { + st := placedManagedStack(t, &managedOwnerPlacer{template: true}) + if c := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":1}`, ""); c.status != http.StatusCreated { + t.Fatalf("create: %d %s", c.status, c.body) + } + reply := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t1-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"work","mode":"rw"}]}`, "") + if reply.status != http.StatusConflict || !strings.Contains(string(reply.body), "no node holds both") { + t.Fatalf("volume local, template only on a peer: %d %s, want 409 saying no node holds both", reply.status, reply.body) + } +} diff --git a/e2e/managed_volume_privacy_test.go b/e2e/managed_volume_privacy_test.go new file mode 100644 index 00000000..0a1f671d --- /dev/null +++ b/e2e/managed_volume_privacy_test.go @@ -0,0 +1,127 @@ +package e2e + +import ( + "encoding/json/v2" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/server" + "github.com/cocoonstack/sandbox/sandboxd/tenants" +) + +func TestLiveAttachManagedVolumePrivacy(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + for name, token := range map[string]string{"private": "t1-token", "owned": "t2-token"} { + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/"+name, token, `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("create %s: %d %s", name, created.status, created.body) + } + } + claim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small"}`, "") + if claim.status != http.StatusOK { + t.Fatalf("claim: %d %s", claim.status, claim.body) + } + var sb server.ClaimResponse + if err := json.Unmarshal(claim.body, &sb); err != nil { + t.Fatal(err) + } + route := "/v1/sandboxes/" + sb.ID + "/volumes" + foreign := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"private","mount":"/volumes/probe"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"unknown","mount":"/volumes/probe"}]}`, "") + if foreign.status != http.StatusBadRequest || unknown.status != http.StatusBadRequest || string(foreign.body) != string(unknown.body) { + t.Errorf("foreign=%d %s; unknown=%d %s; want identical 400 replies", foreign.status, foreign.body, unknown.status, unknown.body) + } + owned := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"owned","mount":"/volumes/probe"}]}`, "") + if owned.status != http.StatusBadRequest || !strings.Contains(string(owned.body), "claim-time attachment only") { + t.Errorf("owned=%d %s; want unsupported live attach", owned.status, owned.body) + } +} + +type catalogPeerPlacer struct{ edgePlacer } + +func (*catalogPeerPlacer) VolumeCandidates(string, []string) []string { return []string{"peer:7777"} } + +func (*catalogPeerPlacer) VolumeOwners([]string) []string { return []string{"peer:7777"} } + +func (*catalogPeerPlacer) VolumeHolders() map[string]int { return map[string]int{"peer-only": 1} } + +func TestRootPeerCatalogRedirectWithManagedStore(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(fmt.Sprint(enabled), func(t *testing.T) { + var st *stack + if enabled { + st = startManagedHTTPStack(t, "root-token") + } else { + st = startConfigStack(t, "root-token", &config.Config{}) + } + srv := server.New(st.token, "", st.mgr, st.eng.real, &catalogPeerPlacer{}, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + defer ts.Close() + defer srv.CloseRelays() + probe := *st + probe.addr = strings.TrimPrefix(ts.URL, "http://") + response := requestManagedVolume(t, &probe, http.MethodPost, "/v1/claim", "root-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"peer-only"}]}`, "") + if response.status != http.StatusOK { + t.Fatalf("claim: %d %s", response.status, response.body) + } + var reply server.ClaimResponse + if err := json.Unmarshal(response.body, &reply); err != nil { + t.Fatal(err) + } + if !slices.Equal(reply.Redirect, []string{"peer:7777"}) || reply.ID != "" { + t.Fatalf("redirect reply = %+v", reply) + } + }) + } +} + +func TestStaticVolumePrivacyWithManagedStore(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(fmt.Sprint(enabled), func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "catalog.raw") + if err := os.WriteFile(path, []byte("catalog"), 0o600); err != nil { + t.Fatal(err) + } + cfg := &config.Config{Volumes: []config.VolumeSpec{{Name: "private-static", Path: path, Tenants: []string{"t1"}}}} + expected := http.StatusBadRequest + if enabled { + cfg.ManagedVolumeDir = filepath.Join(dir, "managed") + } + st := startConfigStack(t, "root-token", cfg) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}}); err != nil { + t.Fatal(err) + } + t.Run("claim", func(t *testing.T) { + foreign := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"private-static"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"unknown"}]}`, "") + if foreign.status != expected || unknown.status != expected || string(foreign.body) != string(unknown.body) { + t.Fatalf("foreign=%d %s; unknown=%d %s; want identical %d replies", foreign.status, foreign.body, unknown.status, unknown.body, expected) + } + }) + t.Run("live-attach", func(t *testing.T) { + claim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small"}`, "") + if claim.status != http.StatusOK { + t.Fatalf("claim: %d %s", claim.status, claim.body) + } + var sb server.ClaimResponse + if err := json.Unmarshal(claim.body, &sb); err != nil { + t.Fatal(err) + } + route := "/v1/sandboxes/" + sb.ID + "/volumes" + foreign := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"private-static","mount":"/volumes/probe"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"unknown","mount":"/volumes/probe"}]}`, "") + if foreign.status != expected || unknown.status != expected || string(foreign.body) != string(unknown.body) { + t.Fatalf("foreign=%d %s; unknown=%d %s; want identical %d replies", foreign.status, foreign.body, unknown.status, unknown.body, expected) + } + }) + }) + } +} diff --git a/e2e/wire_parity_test.go b/e2e/wire_parity_test.go index 012f172e..1dc2eb15 100644 --- a/e2e/wire_parity_test.go +++ b/e2e/wire_parity_test.go @@ -44,6 +44,10 @@ var wirePairs = []struct { {"checkpointRecord", false, types.Checkpoint{}, []string{"node", "key", "tenant", "policy_source", "no_egress", "guest_env", "archive"}}, {"volumeListResponse", false, server.VolumeListResponse{}, nil}, {"VolumeInfo", false, types.VolumeInfo{}, nil}, + {"ManagedVolumeInfo", false, types.ManagedVolumeInfo{}, nil}, + {"VolumeHolder", false, types.VolumeHolder{}, nil}, + {"VolumeCreate", true, types.VolumeCreate{}, nil}, + {"volumeResponse", false, server.VolumeResponse{}, nil}, {"VolumeState", false, types.VolumeState{}, nil}, {"SandboxVolumesResponse", false, types.SandboxVolumesResponse{}, nil}, {"VolumePatch", true, types.VolumePatch{}, nil}, @@ -117,6 +121,7 @@ func parseSDKFields(t *testing.T) map[string]map[string]string { t.Fatal(err) } out := map[string]map[string]string{} + embedded := map[string][]string{} for _, path := range paths { if strings.HasSuffix(path, "_test.go") { continue @@ -130,21 +135,38 @@ func parseSDKFields(t *testing.T) map[string]map[string]string { if !ok { return true } - if st, ok := spec.Type.(*ast.StructType); ok { - out[spec.Name.Name] = map[string]string{} - for _, f := range st.Fields.List { - if f.Tag == nil { - continue - } - tag, _ := strconv.Unquote(f.Tag.Value) - if name, opts := splitJSONTag(reflect.StructTag(tag).Get("json")); name != "" && name != "-" { - out[spec.Name.Name][name] = opts + st, ok := spec.Type.(*ast.StructType) + if !ok { + return true + } + out[spec.Name.Name] = map[string]string{} + for _, f := range st.Fields.List { + if len(f.Names) == 0 { + if id, ok := f.Type.(*ast.Ident); ok { + embedded[spec.Name.Name] = append(embedded[spec.Name.Name], id.Name) } } + if f.Tag == nil { + continue + } + tag, _ := strconv.Unquote(f.Tag.Value) + if name, opts := splitJSONTag(reflect.StructTag(tag).Get("json")); name != "" && name != "-" { + out[spec.Name.Name][name] = opts + } } return true }) } + var flatten func(string) map[string]string + flatten = func(name string) map[string]string { + for _, base := range embedded[name] { + maps.Copy(out[name], flatten(base)) + } + return out[name] + } + for name := range out { + flatten(name) + } return out } diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index fba3efba..d14a037f 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -274,9 +274,11 @@ type Config struct { DiskReserve *DiskReserve `json:"disk_reserve,omitempty"` - Volumes []VolumeSpec `json:"volumes,omitempty"` - Mesh *MeshConfig `json:"mesh,omitempty"` - Pools []PoolSpec `json:"pools"` + ManagedVolumeHibernate *ManagedVolumeHibernate `json:"managed_volume_hibernate,omitempty"` + ManagedVolumeDir string `json:"managed_volume_dir,omitempty"` + Volumes []VolumeSpec `json:"volumes,omitempty"` + Mesh *MeshConfig `json:"mesh,omitempty"` + Pools []PoolSpec `json:"pools"` } // HasEgress reports whether the node can attach egress-lane VMs. @@ -398,6 +400,9 @@ func (c *Config) validateReplication() error { } func (c *Config) validateVolumes() error { + if err := c.ValidateManagedVolumeStore(); err != nil { + return err + } names := make(map[string]struct{}, len(c.Volumes)) paths := make(map[string]string, len(c.Volumes)) for _, volume := range c.Volumes { @@ -422,6 +427,66 @@ func (c *Config) validateVolumes() error { return nil } +func (c *Config) ValidateManagedVolumeStore() error { + if c.ManagedVolumeHibernate != nil { + if err := c.ManagedVolumeHibernate.Validate(); err != nil { + return err + } + if c.ManagedVolumeDir == "" { + return fmt.Errorf("managed_volume_hibernate requires managed_volume_dir") + } + } + if c.ManagedVolumeDir == "" { + return nil + } + if !filepath.IsAbs(c.ManagedVolumeDir) || filepath.Clean(c.ManagedVolumeDir) != c.ManagedVolumeDir || c.ManagedVolumeDir == "/" { + return fmt.Errorf("managed_volume_dir must be a clean absolute non-root path") + } + if c.APIToken == "" { + return fmt.Errorf("managed_volume_dir requires a nonempty api_token") + } + dataDir, err := filepath.Abs(c.DataDir) + if err != nil { + return fmt.Errorf("resolve data_dir: %w", err) + } + resolved, err := resolveExistingAncestors(dataDir) + if err != nil { + return fmt.Errorf("resolve data_dir: %w", err) + } + for _, root := range []string{dataDir, resolved} { + relative, err := filepath.Rel(root, c.ManagedVolumeDir) + if err != nil { + return fmt.Errorf("compare managed_volume_dir with data_dir: %w", err) + } + first, _, _ := strings.Cut(relative, string(filepath.Separator)) + if first == "readers" || first == "goldens" || strings.HasPrefix(first, "fork-") { + return fmt.Errorf("managed_volume_dir must be outside data_dir startup cleanup paths (readers, goldens, fork-*)") + } + } + for _, namespace := range []string{"readers", "goldens"} { + swept, err := resolveExistingAncestors(filepath.Join(dataDir, namespace)) + if err != nil { + return fmt.Errorf("resolve data_dir cleanup path: %w", err) + } + relative, err := filepath.Rel(swept, c.ManagedVolumeDir) + if err != nil { + return fmt.Errorf("compare managed_volume_dir with cleanup path: %w", err) + } + if relative != ".." && !strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return fmt.Errorf("managed_volume_dir must be outside data_dir %s cleanup path", namespace) + } + relative, err = filepath.Rel(c.ManagedVolumeDir, swept) + if err != nil { + return fmt.Errorf("compare cleanup path with managed_volume_dir: %w", err) + } + first, _, _ := strings.Cut(relative, string(filepath.Separator)) + if first == "disks" { + return fmt.Errorf("data_dir %s cleanup path must be outside managed volume backing storage", namespace) + } + } + return nil +} + func (c *Config) validatePreview() error { if c.PreviewListen == "" { return nil @@ -663,6 +728,25 @@ func StorageBytes(s string) (int64, error) { return units.RAMInBytes(s) } +func resolveExistingAncestors(path string) (string, error) { + suffix := "" + for { + _, err := os.Lstat(path) + if err == nil { + resolved, resolveErr := filepath.EvalSymlinks(path) + if resolveErr != nil { + return "", resolveErr + } + return filepath.Join(resolved, suffix), nil + } + if !os.IsNotExist(err) || filepath.Dir(path) == path { + return "", err + } + suffix = filepath.Join(filepath.Base(path), suffix) + path = filepath.Dir(path) + } +} + func validatePolicy(p *egress.Policy, secrets map[string]struct{}) error { if p == nil { return nil diff --git a/sandboxd/config/managed_hibernate.go b/sandboxd/config/managed_hibernate.go new file mode 100644 index 00000000..108d196f --- /dev/null +++ b/sandboxd/config/managed_hibernate.go @@ -0,0 +1,24 @@ +package config + +import ( + "errors" + "path/filepath" + "strings" +) + +type ManagedVolumeHibernate struct { + HookArgv []string `json:"hook_argv"` + HookTimeoutSeconds int `json:"hook_timeout_seconds"` +} + +func (h ManagedVolumeHibernate) Validate() error { + if len(h.HookArgv) == 0 || !filepath.IsAbs(h.HookArgv[0]) || filepath.Clean(h.HookArgv[0]) != h.HookArgv[0] || h.HookTimeoutSeconds < 1 || h.HookTimeoutSeconds > 300 { + return errors.New("managed_volume_hibernate requires absolute hook_argv and hook_timeout_seconds from 1 through 300") + } + for _, arg := range h.HookArgv { + if strings.ContainsRune(arg, 0) { + return errors.New("managed_volume_hibernate hook_argv contains NUL") + } + } + return nil +} diff --git a/sandboxd/config/managed_hibernate_test.go b/sandboxd/config/managed_hibernate_test.go new file mode 100644 index 00000000..ea1151a3 --- /dev/null +++ b/sandboxd/config/managed_hibernate_test.go @@ -0,0 +1,28 @@ +package config + +import "testing" + +func TestManagedHibernateHookValidation(t *testing.T) { + for _, tc := range []struct { + name string + hook ManagedVolumeHibernate + valid bool + }{ + {"valid", ManagedVolumeHibernate{[]string{"/usr/local/bin/hook", "literal $(arg)"}, 10}, true}, + {"missing command", ManagedVolumeHibernate{nil, 10}, false}, + {"relative command", ManagedVolumeHibernate{[]string{"hook"}, 10}, false}, + {"NUL argument", ManagedVolumeHibernate{[]string{"/hook", "bad\x00arg"}, 10}, false}, + {"zero timeout", ManagedVolumeHibernate{[]string{"/hook"}, 0}, false}, + {"excess timeout", ManagedVolumeHibernate{[]string{"/hook"}, 301}, false}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := tc.hook.Validate(); (err == nil) != tc.valid { + t.Fatalf("valid=%v error=%v", tc.valid, err) + } + }) + } + cfg := Config{ManagedVolumeHibernate: &ManagedVolumeHibernate{[]string{"/hook"}, 1}} + if err := cfg.ValidateManagedVolumeStore(); err == nil { + t.Fatal("enabled hibernate without managed storage") + } +} diff --git a/sandboxd/config/managed_volume_test.go b/sandboxd/config/managed_volume_test.go new file mode 100644 index 00000000..5bd02a80 --- /dev/null +++ b/sandboxd/config/managed_volume_test.go @@ -0,0 +1,137 @@ +package config + +import ( + "os" + "path/filepath" + "testing" +) + +func TestManagedVolumeStoreExcludesStartupCleanupPaths(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + for _, tc := range []struct { + path string + allow bool + }{ + {filepath.Join(data, "managed"), true}, + {filepath.Join(root, "outside"), true}, + {filepath.Join(data, "readers-backup", "managed"), true}, + {filepath.Join(data, "forks", "managed"), true}, + {filepath.Join(data, "readers"), false}, + {filepath.Join(data, "readers", "managed"), false}, + {filepath.Join(data, "fork-state"), false}, + {filepath.Join(data, "fork-state", "managed"), false}, + {filepath.Join(data, "goldens"), false}, + {filepath.Join(data, "goldens", "managed"), false}, + } { + t.Run(tc.path, func(t *testing.T) { + cfg := Config{DataDir: data, ManagedVolumeDir: tc.path, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Fatalf("path %q accepted=%v, want %v: %v", tc.path, err == nil, tc.allow, err) + } + }) + } +} + +func TestManagedVolumeStoreResolvesDataDirectoryAliases(t *testing.T) { + root := t.TempDir() + realDir := filepath.Join(root, "real") + alias := filepath.Join(root, "alias") + if err := os.Mkdir(realDir, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Symlink(realDir, alias); err != nil { + t.Fatal(err) + } + t.Chdir(root) + for _, tc := range []struct { + name, data, managed string + allow bool + }{ + {"alias-readers", alias, filepath.Join(realDir, "readers", "managed"), false}, + {"alias-goldens", alias, filepath.Join(realDir, "goldens"), false}, + {"alias-fork", alias, filepath.Join(realDir, "fork-persistent"), false}, + {"alias-new-suffix", filepath.Join(alias, "new", "data"), filepath.Join(realDir, "new", "data", "readers", "managed"), false}, + {"alias-safe", alias, filepath.Join(realDir, "managed"), true}, + {"relative", "real", filepath.Join(realDir, "readers", "managed"), false}, + {"relative-safe", "real", filepath.Join(realDir, "readers-safe"), true}, + {"empty", "", filepath.Join(root, "fork-store"), false}, + {"empty-safe", "", filepath.Join(root, "managed"), true}, + {"ancestor", filepath.Join(realDir, "data"), realDir, true}, + {"same-root", realDir, realDir, true}, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := Config{DataDir: tc.data, ManagedVolumeDir: tc.managed, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Fatalf("data=%q managed=%q accepted=%v, want %v: %v", tc.data, tc.managed, err == nil, tc.allow, err) + } + }) + } + if _, err := os.Stat(filepath.Join(realDir, "new")); !os.IsNotExist(err) { + t.Fatalf("validation created data directories: %v", err) + } +} + +func TestManagedVolumeStoreRejectsCleanupDirectoryAliases(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + t.Run(namespace, func(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + external := filepath.Join(root, "external") + for _, dir := range []string{data, external} { + if err := os.Mkdir(dir, 0o700); err != nil { + t.Fatal(err) + } + } + if err := os.Symlink(external, filepath.Join(data, namespace)); err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + path string + allow bool + }{ + {external, false}, + {filepath.Join(external, "managed"), false}, + {filepath.Join(data, "managed"), true}, + {root, true}, + } { + cfg := Config{DataDir: data, ManagedVolumeDir: tc.path, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Errorf("%s alias target %q accepted=%v, want %v: %v", namespace, tc.path, err == nil, tc.allow, err) + } + } + if _, err := os.Stat(filepath.Join(external, "managed")); !os.IsNotExist(err) { + t.Fatalf("validation created managed directory: %v", err) + } + }) + } +} + +func TestManagedVolumeStoreProtectsBackingDirectoryFromCleanupAliases(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + for _, target := range []string{"disks", "disks/nested", "disks-backup", "work/readers"} { + t.Run(namespace+"/"+target, func(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + managed := filepath.Join(root, "managed") + aliasTarget := filepath.Join(managed, target) + for _, dir := range []string{data, aliasTarget} { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + } + if err := os.Symlink(aliasTarget, filepath.Join(data, namespace)); err != nil { + t.Fatal(err) + } + cfg := Config{DataDir: data, ManagedVolumeDir: managed, APIToken: "root"} + allow := target == "disks-backup" || target == "work/readers" + if err := cfg.ValidateManagedVolumeStore(); (err == nil) != allow { + t.Fatalf("cleanup alias %q accepted=%v, want %v: %v", target, err == nil, allow, err) + } + }) + } + } +} diff --git a/sandboxd/engine/cocoond_test.go b/sandboxd/engine/cocoond_test.go index 053dc205..77044695 100644 --- a/sandboxd/engine/cocoond_test.go +++ b/sandboxd/engine/cocoond_test.go @@ -48,7 +48,7 @@ func TestInspectReadsOneVMAndReportsAMissingOne(t *testing.T) { "echo \"Error: inspect: vm $3: vm not found\" >&2; exit 1\n") e := New("cocoon", nil, nil, false, false, "") rec, ok, err := e.Inspect(t.Context(), "sbx-1") - wantDisks := []types.VMDisk{{Name: "imagenet", ReadOnly: true}, {Name: "scratch"}} + wantDisks := []types.VMDisk{{Name: "imagenet", Path: "/v/a.img", ReadOnly: true}, {Name: "scratch", Path: "/v/b.img"}} if err != nil || !ok || rec.State != "running" || rec.Config.Name != "sbx-1" || !slices.Equal(rec.AttachedDevices.Disks, wantDisks) { t.Errorf("Inspect sbx-1 = %+v, %v, %v", rec, ok, err) } diff --git a/sandboxd/engine/managed_sleep.go b/sandboxd/engine/managed_sleep.go new file mode 100644 index 00000000..230ccba4 --- /dev/null +++ b/sandboxd/engine/managed_sleep.go @@ -0,0 +1,98 @@ +package engine + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "slices" + "strconv" + "strings" + "time" +) + +type VolumeHookCall struct { + Argv []string + Action, Operation, VolumeID, VolumeName, Mount string + Timeout time.Duration +} + +// VolumeHookExit is a guest volume hook that ran and exited non-zero; Output is the guest's own. +type VolumeHookExit struct { + Code int32 + Output string +} + +func (e *VolumeHookExit) Error() string { return fmt.Sprintf("exit code %d: %s", e.Code, e.Output) } + +type HibernateSource struct { + ID string `json:"id"` + Name string `json:"name"` +} + +func (e *Engine) RunVolumeHook(ctx context.Context, socket string, call VolumeHookCall) error { + seconds := strconv.FormatFloat(call.Timeout.Seconds(), 'f', -1, 64) + argv := []string{"/usr/bin/flock", "--timeout", seconds, "/run/sandboxd-volume-hook.lock", "/usr/bin/timeout", "--signal=KILL", "--", seconds + "s"} + argv = append(argv, call.Argv...) + argv = append(argv, call.Action, call.Operation, call.VolumeID, call.VolumeName, call.Mount) + ctx, cancel := context.WithTimeout(ctx, 2*call.Timeout+10*time.Second) + defer cancel() + _, err := e.silkdOutput(ctx, socket, argv...) + if exit, ok := errors.AsType[*guestExitError](err); ok { + return &VolumeHookExit{Code: exit.code, Output: exit.output} + } + return err +} + +func (e *Engine) InspectHibernateSource(ctx context.Context, name string) (HibernateSource, bool, error) { + names, err := e.SnapshotList(ctx) + if err != nil { + return HibernateSource{}, false, err + } + if !slices.Contains(names, name) { + return HibernateSource{}, false, nil + } + raw, err := e.run(ctx, "snapshot", "inspect", name) + if err != nil { + return HibernateSource{}, false, err + } + var source HibernateSource + if err = json.Unmarshal(raw, &source); err != nil { + return source, false, fmt.Errorf("parse snapshot inspect: %w", err) + } + if source.ID == "" || source.Name != name { + return source, false, errors.New("snapshot identity unavailable") + } + return source, true, nil +} + +func (e *Engine) VerifyMountedVolume(ctx context.Context, socket, name, mount string) error { + ctx, cancel := context.WithTimeout(ctx, volumeSetupTimeout) + defer cancel() + device, found, err := e.findVolumeDevice(ctx, socket, name) + if err != nil { + return err + } + if !found { + return &VolumeMountConflict{Message: "managed volume device is absent"} + } + devNum, mounts, err := e.volumeMounts(ctx, socket, device) + if err != nil { + return err + } + target := e.resolveMount(ctx, socket, mount, mounts) + matched := false + for _, entry := range mounts { + if entry.path != target && entry.devNum != devNum { + continue + } + if matched || entry.path != target || entry.devNum != devNum || entry.root != "/" || !slices.Contains(strings.Split(entry.options, ","), "rw") { + return &VolumeMountConflict{Message: fmt.Sprintf("managed volume must have one exact writable root mount at %q", target)} + } + matched = true + } + if !matched { + return &VolumeMountConflict{Message: fmt.Sprintf("managed volume mount %q is absent", target)} + } + return nil +} diff --git a/sandboxd/engine/managed_sleep_test.go b/sandboxd/engine/managed_sleep_test.go new file mode 100644 index 00000000..07218490 --- /dev/null +++ b/sandboxd/engine/managed_sleep_test.go @@ -0,0 +1,124 @@ +package engine + +import ( + "errors" + "slices" + "strings" + "testing" + "time" +) + +func TestManagedHookUsesBoundedSerializedArgv(t *testing.T) { + sock := sockPath(t) + fake := serveFakeSilkd(t, sock) + e := New("cocoon", nil, nil, false, false, "") + call := VolumeHookCall{Argv: []string{"/opt/hook", "$(touch /tmp/escape); literal"}, Action: "prepare", Operation: "op", VolumeID: "vol", VolumeName: "data", Mount: "/data with spaces", Timeout: 10 * time.Second} + if err := e.RunVolumeHook(t.Context(), sock, call); err != nil { + t.Fatal(err) + } + fake.mu.Lock() + got := slices.Clone(fake.execCalls[0]) + fake.execCode = 137 + fake.mu.Unlock() + want := []string{"/usr/bin/flock", "--timeout", "10", "/run/sandboxd-volume-hook.lock", "/usr/bin/timeout", "--signal=KILL", "--", "10s", "/opt/hook", "$(touch /tmp/escape); literal", "prepare", "op", "vol", "data", "/data with spaces"} + if !slices.Equal(got, want) { + t.Fatalf("guest argv %q", got) + } + if exit, ok := errors.AsType[*VolumeHookExit](e.RunVolumeHook(t.Context(), sock, call)); !ok || exit.Code != 137 { + t.Fatalf("hook termination = %v, want a guest exit with code 137", exit) + } + err := e.RunVolumeHook(t.Context(), sockPath(t), call) + if _, ok := errors.AsType[*VolumeHookExit](err); ok || err == nil { + t.Fatalf("unreachable guest = %v, want a transport error, not a guest exit", err) + } +} + +func TestManagedMountRequiresLiteralWritableRoot(t *testing.T) { + for _, tc := range []struct { + name, mountinfo string + valid bool + }{ + {"exact", "2 1 8:32 / /data rw - ext4 /dev/vdc rw\n", true}, + {"absent", "", false}, + {"read only", "2 1 8:32 / /data ro - ext4 /dev/vdc ro\n", false}, + {"wrong device", "2 1 8:48 / /data rw - ext4 /dev/vdd rw\n", false}, + {"subdirectory bind", "2 1 8:32 /sub /data rw - ext4 /dev/vdc rw\n", false}, + {"alias", "2 1 8:32 / /elsewhere rw - ext4 /dev/vdc rw\n", false}, + {"extra mount", "2 1 8:32 / /data rw - ext4 /dev/vdc rw\n3 1 8:32 / /alias rw - ext4 /dev/vdc rw\n", false}, + } { + t.Run(tc.name, func(t *testing.T) { + sock := sockPath(t) + fake := serveFakeSilkd(t, sock) + configureVolumeDevices(fake) + fake.mu.Lock() + fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") + fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + tc.mountinfo) + fake.mu.Unlock() + err := New("cocoon", nil, nil, false, false, "").VerifyMountedVolume(t.Context(), sock, "imagenet", "/data") + if tc.valid && err != nil { + t.Fatal(err) + } + if !tc.valid { + if _, ok := errors.AsType[*VolumeMountConflict](err); !ok { + t.Fatalf("mount accepted or error wrong: %v", err) + } + } + fake.mu.Lock() + defer fake.mu.Unlock() + if len(guestChanges(fake.execCalls)) != 0 { + t.Fatal("mount preflight changed guest") + } + }) + } +} + +func TestManagedMountResolvesASymlinkedParent(t *testing.T) { + for _, tc := range []struct { + name, mountinfo string + valid bool + }{ + {"mounted", "2 1 8:32 / /home/u/ws/data rw - ext4 /dev/vdc rw\n", true}, + {"read only", "2 1 8:32 / /home/u/ws/data ro - ext4 /dev/vdc ro\n", false}, + {"absent", "", false}, + } { + t.Run(tc.name, func(t *testing.T) { + sock := sockPath(t) + fake := serveFakeSilkd(t, sock) + configureVolumeDevices(fake) + fake.mu.Lock() + fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") + fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + tc.mountinfo) + fake.realpath = map[string]string{"/workspace/data": "/home/u/ws/data"} + fake.mu.Unlock() + err := New("cocoon", nil, nil, false, false, "").VerifyMountedVolume(t.Context(), sock, "imagenet", "/workspace/data") + if tc.valid && err != nil { + t.Fatal(err) + } + if !tc.valid { + if _, ok := errors.AsType[*VolumeMountConflict](err); !ok || !strings.Contains(err.Error(), `"/home/u/ws/data"`) { + t.Fatalf("mount accepted or conflict hid the resolved guest path: %v", err) + } + } + }) + } +} + +func TestHibernateSourceRequiresExactInspectMetadata(t *testing.T) { + for _, tc := range []struct { + name, listed, metadata string + found, fail bool + }{ + {"present", `[{"name":"sleep"}]`, `{"name":"sleep","id":"immutable-source"}`, true, false}, + {"missing", `[]`, `{}`, false, false}, + {"wrong name", `[{"name":"sleep"}]`, `{"name":"other","id":"immutable-source"}`, false, true}, + {"missing id", `[{"name":"sleep"}]`, `{"name":"sleep"}`, false, true}, + } { + t.Run(tc.name, func(t *testing.T) { + fakeCocoon(t, "#!/bin/sh\ncase \"$1 $2\" in\n'snapshot list') echo '"+tc.listed+"';;\n'snapshot inspect') echo '"+tc.metadata+"';;\n*) exit 1;;\nesac\n") + source, found, err := New("cocoon", nil, nil, false, false, "").InspectHibernateSource(t.Context(), "sleep") + if found != tc.found || (err != nil) != tc.fail { + t.Fatalf("source=%+v found=%v err=%v", source, found, err) + } + }) + } +} diff --git a/sandboxd/main.go b/sandboxd/main.go index 1c6298a9..a40e468b 100644 --- a/sandboxd/main.go +++ b/sandboxd/main.go @@ -90,6 +90,7 @@ func main() { if err != nil { logger.Fatalf(ctx, err, "init pool manager") } + defer func() { _ = mgr.Close() }() ctx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) defer stop() diff --git a/sandboxd/pool/archive.go b/sandboxd/pool/archive.go index af26aa17..d7f87e05 100644 --- a/sandboxd/pool/archive.go +++ b/sandboxd/pool/archive.go @@ -49,7 +49,7 @@ func (m *Manager) archiveOnce(ctx context.Context) { var victims []*types.Sandbox m.mu.Lock() for _, sb := range m.claimed { - if sb.HibernateSnap == "" || sb.ArchiveCk != "" || m.tenantGone(sb.Tenant) { + if m.hasVolumeHolds(sb) || sb.HibernateSnap == "" || sb.ArchiveCk != "" || m.tenantGone(sb.Tenant) { continue } if _, busy := m.archiving[sb.ID]; busy { diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 683074ee..85c4ea73 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -217,6 +217,9 @@ func (m *Manager) AgentSocket(id, token string) (string, error) { return "", ErrLeaseLost } // no activity stamp: a control-plane poll must not keep an idle sandbox awake + if sb.VolumeSleep != nil { + return "", ErrPaused + } return sb.VsockSocket, nil } @@ -368,13 +371,17 @@ func (m *Manager) overQuota(extra int, tenant string) error { return m.quotaErr(extra, tenant) } -// admitVolumes takes one claim's holds; a refusal on either check leaves nothing held. +// admitVolumes takes one claim's holds; a refusal on any check leaves nothing held. func (m *Manager) admitVolumes(tenant string, volumes []resolvedVolume) ([]types.Volume, error) { applied := appliedVolumes(volumes) if err := m.admitClaim(tenant, applied); err != nil { - return nil, err + return nil, m.managedBusy(err, volumes) + } + err := m.admitManaged(volumes) + if err == nil { + err = m.confirmVolumesClean(volumes) } - if err := m.confirmVolumesClean(volumes); err != nil { + if err != nil { m.unreserveVolumes(applied) return nil, err } @@ -535,6 +542,7 @@ func (m *Manager) abortVolumeClaim(ctx context.Context, vmName string, reserved } func (m *Manager) reapOnce(ctx context.Context) { + m.retryManagedSnapshot() now := time.Now() type victim struct { action reapAction @@ -674,7 +682,7 @@ func (m *Manager) reapOnce(ctx context.Context) { } func (m *Manager) archivesAtDeadline(sb *types.Sandbox) bool { - return sb.Failed == "" && !m.tenantGone(sb.Tenant) && (sb.OnExpire == types.ExpireArchive || sb.HibernateSnap != "" && m.archiveAfterFor(sb.Key) > 0) + return !m.hasVolumeHolds(sb) && sb.Failed == "" && !m.tenantGone(sb.Tenant) && (sb.OnExpire == types.ExpireArchive || sb.HibernateSnap != "" && m.archiveAfterFor(sb.Key) > 0) } func (m *Manager) pauseExpired(ctx context.Context, sb *types.Sandbox) error { @@ -754,6 +762,11 @@ func (m *Manager) admitOptions(ctx context.Context, key types.PoolKey, o ClaimOp if err != nil { return nil, nil, err } + if slices.ContainsFunc(volumeSpecs, func(v resolvedVolume) bool { return v.managedID != "" }) { + if err = m.tenantRemoved(ctx, o.Tenant); err != nil { + return nil, nil, err + } + } applied, err := m.admitVolumes(o.Tenant, volumeSpecs) return volumeSpecs, applied, err } @@ -801,8 +814,19 @@ func archivable(key types.PoolKey, volumes bool, onExpire types.ExpireAction) er case key.Net == types.NetEgress: return ErrNoEgressHibernate case volumes: - return ErrVolumeCapture + return ErrVolumeArchive default: return nil } } + +// ErrVolumeArchive refuses archive for a claim that holds volumes; it matches ErrVolumeCapture. +var ErrVolumeArchive error = volumeArchiveError{} + +type volumeArchiveError struct{} + +func (volumeArchiveError) Error() string { + return "archive does not export volumes yet; release keeps the volume" +} + +func (volumeArchiveError) Is(target error) bool { return target == ErrVolumeCapture } diff --git a/sandboxd/pool/claim_socket_test.go b/sandboxd/pool/claim_socket_test.go new file mode 100644 index 00000000..8c895acd --- /dev/null +++ b/sandboxd/pool/claim_socket_test.go @@ -0,0 +1,51 @@ +package pool + +import ( + "context" + "sync" + "testing" + "time" +) + +type metadataWriteGate struct { + *fakeEngine + entered chan struct{} + resume chan struct{} +} + +func (e *metadataWriteGate) WriteInstanceMetadata(ctx context.Context, socket string, doc []byte) error { + close(e.entered) + <-e.resume + return e.fakeEngine.WriteInstanceMetadata(ctx, socket, doc) +} + +func TestAgentSocketAvailableDuringMetadataUpdate(t *testing.T) { + e := newFakeEngine() + m := newTestManager(t, e) + t.Cleanup(func() { _ = m.Close() }) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{}) + if err != nil { + t.Fatal(err) + } + gate := &metadataWriteGate{fakeEngine: e, entered: make(chan struct{}), resume: make(chan struct{})} + m.eng = gate + done := make(chan error, 1) + go func() { done <- m.SetInstanceMetadata(t.Context(), sb.ID, []byte(`{"role":"test"}`)) }() + finish := sync.OnceFunc(func() { + close(gate.resume) + if updateErr := <-done; updateErr != nil { + t.Errorf("metadata update: %v", updateErr) + } + }) + t.Cleanup(finish) + select { + case <-gate.entered: + case <-time.After(time.Second): + t.Fatal("metadata update did not enter engine") + } + socket, err := m.AgentSocket(sb.ID, sb.Token) + finish() + if err != nil || socket != sb.VsockSocket { + t.Fatalf("running claim unavailable during metadata update: socket=%q err=%v", socket, err) + } +} diff --git a/sandboxd/pool/claims.go b/sandboxd/pool/claims.go index 5d7111df..10edc410 100644 --- a/sandboxd/pool/claims.go +++ b/sandboxd/pool/claims.go @@ -60,6 +60,7 @@ type claimDTO struct { ArchivePending string `json:"archive_pending,omitempty"` ReplicaCk string `json:"replica_ck,omitempty"` ReplicaPending string `json:"replica_pending,omitempty"` + VolumeSleep *types.VolumeSleep `json:"volume_sleep,omitempty"` FromCheckpoint string `json:"from_checkpoint,omitempty"` Restarts int `json:"restarts,omitzero"` RestartedAt time.Time `json:"restarted_at,omitzero"` @@ -267,7 +268,7 @@ func dtoOf(sb *types.Sandbox) claimDTO { Tenant: sb.Tenant, EgressClass: sb.EgressClass, ClaimRef: sb.ClaimRef, Metadata: sb.Metadata, OnExpire: sb.OnExpire, Volumes: slices.Clone(sb.Volumes), VolumesWant: volumesWantDTO(sb), Env: sb.Env, VsockSocket: sb.VsockSocket, TAP: sb.TAP, HibernateSnap: sb.HibernateSnap, PendingSnap: sb.PendingSnap, ReaderSafe: sb.ReaderSafe, - ArchiveCk: sb.ArchiveCk, ArchiveEpoch: sb.ArchiveEpoch, ArchivePending: sb.ArchivePending, ReplicaCk: sb.ReplicaCk, ReplicaPending: sb.ReplicaPending, FromCheckpoint: sb.FromCheckpoint, + ArchiveCk: sb.ArchiveCk, ArchiveEpoch: sb.ArchiveEpoch, ArchivePending: sb.ArchivePending, ReplicaCk: sb.ReplicaCk, ReplicaPending: sb.ReplicaPending, VolumeSleep: sb.VolumeSleep, FromCheckpoint: sb.FromCheckpoint, Restarts: sb.Restarts, RestartedAt: sb.RestartedAt, Failed: sb.Failed, } } diff --git a/sandboxd/pool/expiry_test.go b/sandboxd/pool/expiry_test.go index a0bbee2f..9b6fe9fa 100644 --- a/sandboxd/pool/expiry_test.go +++ b/sandboxd/pool/expiry_test.go @@ -3,6 +3,7 @@ package pool import ( "context" "errors" + "strings" "sync/atomic" "testing" "time" @@ -319,3 +320,10 @@ func (s *failingPublishStore) Publish(ctx context.Context, staging, id, lineage } return s.Store.Publish(ctx, staging, id, lineage) } + +func TestArchiveRefusalOfVolumeClaimNamesArchive(t *testing.T) { + err := archivable(testKey, true, types.ExpireArchive) + if !errors.Is(err, ErrVolumeCapture) || !strings.Contains(err.Error(), "archive") || strings.Contains(err.Error(), "hibernate") { + t.Fatalf("archive refusal = %q, want a volume-capture refusal that names archive and does not claim hibernate is impossible", err) + } +} diff --git a/sandboxd/pool/hibernate.go b/sandboxd/pool/hibernate.go index cac827ed..70fbb06a 100644 --- a/sandboxd/pool/hibernate.go +++ b/sandboxd/pool/hibernate.go @@ -69,6 +69,9 @@ func (m *Manager) hibernateLocked(ctx context.Context, sb *types.Sandbox) error volumeBlocked := m.hasVolumeHolds(sb) m.mu.Unlock() if volumeBlocked { + if sb.VolumeSleep != nil || m.managedSleepHook != nil { + return m.driveManagedSleepLocked(ctx, sb, managedSleeping) + } return ErrVolumeCapture } if sb.Key.Net == types.NetEgress { @@ -159,6 +162,12 @@ func (m *Manager) wakeOrRead(ctx context.Context, sb *types.Sandbox, readerOK bo if sb.Failed != "" { return "", fmt.Errorf("%w: %s", ErrFailed, sb.Failed) } + if sb.VolumeSleep != nil { + if err := m.driveManagedSleepLocked(ctx, sb, managedRunning); err != nil { + return "", err + } + return sb.VsockSocket, nil + } // settle a dangling intent before choosing running vs hibernated if _, err := m.settlePendingSnap(ctx, sb); err != nil { return "", fmt.Errorf("wake %s: %w", sb.ID, err) diff --git a/sandboxd/pool/managed_attachment.go b/sandboxd/pool/managed_attachment.go new file mode 100644 index 00000000..79e128f8 --- /dev/null +++ b/sandboxd/pool/managed_attachment.go @@ -0,0 +1,41 @@ +package pool + +import "errors" + +// admitManaged re-checks each managed volume after its name is reserved; a delete or grow that won the name first has already changed the record. +func (m *Manager) admitManaged(volumes []resolvedVolume) error { + r := m.managed + for _, volume := range volumes { + if volume.managedID == "" { + continue + } + r.mu.Lock() + v, ok := r.data.Volumes[volume.applied.Name] + r.mu.Unlock() + switch { + case !ok || v.ID != volume.managedID: + return ErrVolumeUnavailable + case v.GrowTo != 0: + return growInProgress(v.Name) + case v.State != volumeReady: + return managedStorageError() + } + } + return nil +} + +// managedBusy names the holders of a refused managed volume; any other refusal passes through. +func (m *Manager) managedBusy(err error, volumes []resolvedVolume) error { + if !errors.Is(err, ErrVolumeBusy) { + return err + } + m.mu.Lock() + defer m.mu.Unlock() + for _, volume := range volumes { + held := m.volumeAdmission[volume.applied.Name] + if volume.managedID != "" && (held.writers > 0 || volume.applied.RW() && held.readers > 0) { + return &VolumeError{Status: 409, Code: volumeBusyCode, Message: volumeBusyMessage, Volume: volume.applied.Name, Holders: m.volumeHoldersLocked(volume.applied.Name)} + } + } + return err +} diff --git a/sandboxd/pool/managed_foundation_test.go b/sandboxd/pool/managed_foundation_test.go new file mode 100644 index 00000000..c7172a56 --- /dev/null +++ b/sandboxd/pool/managed_foundation_test.go @@ -0,0 +1,291 @@ +package pool + +import ( + "bytes" + "errors" + "os" + "os/exec" + "path/filepath" + "syscall" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func foundationConfig(t *testing.T) *config.Config { + t.Helper() + dir := t.TempDir() + return &config.Config{DataDir: filepath.Join(dir, "data"), ManagedVolumeDir: filepath.Join(dir, "managed"), APIToken: "root", Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 1}}} +} + +func foundationClaim(t *testing.T, m *Manager, names ...string) *types.Sandbox { + t.Helper() + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions(names...)) + if err != nil { + t.Fatal(err) + } + return sb +} + +func TestFoundationCreateCommitAllowsConcurrentVolumeRun(t *testing.T) { + m := openManagedTest(t, foundationConfig(t), newManagedEngine()) + createManagedTest(t, m, "data") + sb := foundationClaim(t, m, "data") + unrelated := foundationClaim(t, m) + entered, resume := make(chan struct{}), make(chan struct{}) + m.managed.commitFile = func(path string, raw []byte) error { close(entered); <-resume; return writeVolumeSnapshot(path, raw) } + created := make(chan error, 1) + go func() { + _, _, err := m.CreateVolume(t.Context(), "other", types.VolumeCreate{SizeGiB: 1}, "t1") + created <- err + }() + select { + case <-entered: + case <-time.After(3 * time.Second): + t.Fatal("create never reached registry commit") + } + if !m.mu.TryLock() { + close(resume) + t.Fatal("create unexpectedly holds manager mutex before indirect waiter") + } + m.mu.Unlock() + runDone := make(chan error, 1) + go func() { + _, run, err := m.beginVolumeRun(t.Context(), sb) + if err == nil { + m.finishVolumeRun(sb, run) + } + runDone <- err + }() + managerLocked := false + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if !m.mu.TryLock() { + managerLocked = true + break + } + m.mu.Unlock() + time.Sleep(time.Millisecond) + } + renewed := make(chan error, 1) + go func() { + _, err := m.Renew(t.Context(), unrelated.ID, Cred{Operator: true}, time.Minute, "") + renewed <- err + }() + blocked := false + select { + case err := <-renewed: + if err != nil { + t.Fatal(err) + } + case <-time.After(150 * time.Millisecond): + blocked = true + } + t.Logf("create commit initially manager unlocked; volume-run waiter manager_locked=%t unrelated_renew_blocked=%t", managerLocked, blocked) + m.managed.commitFile = writeVolumeSnapshot + close(resume) + if err := <-created; err != nil { + t.Fatal(err) + } + if err := <-runDone; err != nil { + t.Fatal(err) + } + if blocked { + if err := <-renewed; err != nil { + t.Fatal(err) + } + } + if managerLocked || blocked { + t.Error("registry I/O blocks global manager through volume-run lock ordering") + } +} + +func TestFoundationReleaseRollbackKeepsLiveClaim(t *testing.T) { + cfg := foundationConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb := foundationClaim(t, m, "data") + if err := os.Mkdir(m.store.path+".tmp", 0o700); err != nil { + t.Fatal(err) + } + releaseErr := m.Release(t.Context(), sb.ID, Cred{Operator: true}) + if releaseErr == nil { + t.Fatal("injected claim persistence failure did not fail release") + } + _, live := m.Sandbox(sb.ID) + _, patchErr := m.PatchVolumes(t.Context(), sb.ID, "t1", types.VolumePatch{}) + if err := os.Remove(m.store.path + ".tmp"); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + if err := m.Close(); err != nil { + t.Fatal(err) + } + next := openManagedTest(t, cfg, e) + if err := next.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + next.retryRemovals(t.Context()).Wait() + _, adopted := next.Sandbox(sb.ID) + _, present, inspectErr := e.Inspect(t.Context(), sb.VMName) + if !live || errors.Is(patchErr, ErrFailed) || !adopted || !present { + t.Errorf("failed release: claim_restored=%t patch_error=%v after restart claim_present=%t VM_present=%t err=%v", live, patchErr, adopted, present, inspectErr) + } +} + +func TestManagedRegistryCloseWaitsForWriter(t *testing.T) { + cfg := managedConfig(t) + m := openManagedTest(t, cfg, newManagedEngine()) + entered, resume := make(chan struct{}), make(chan struct{}) + m.managed.commitFile = func(path string, raw []byte) error { + close(entered) + <-resume + return writeVolumeSnapshot(path, raw) + } + created := make(chan error, 1) + go func() { + _, _, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + created <- err + }() + <-entered + closed := make(chan error, 1) + go func() { closed <- m.Close() }() + select { + case <-closed: + t.Fatal("closed the daemon lock during an active registry write") + case <-time.After(20 * time.Millisecond): + } + if other, err := openVolumeRegistry(cfg.ManagedVolumeDir, "node"); err == nil { + _ = other.close() + t.Fatal("another daemon acquired the lock while a writer was active") + } + close(resume) + <-created + if err := <-closed; err != nil { + t.Fatal(err) + } +} + +func TestManagedIdleRetryRecoversWriteFailure(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + m.managed.commitFile = func(string, []byte) error { return syscall.ENOSPC } + _, _, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + requireVolumeStatus(t, err, 503) + m.managed.commitFile = writeVolumeSnapshot + m.reapOnce(t.Context()) + waitFor(t, func() bool { + _, _, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + return err == nil + }) + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if _, err := m.GetVolume("data", "t1"); err != nil { + t.Fatalf("maintenance did not persist the retained registry state: %v", err) + } +} + +func TestManagedRetiredCatalogNameRemainsReserved(t *testing.T) { + cfg := managedConfig(t) + path := filepath.Join(filepath.Dir(cfg.DataDir), "catalog.img") + if err := os.WriteFile(path, []byte("catalog"), 0o600); err != nil { + t.Fatal(err) + } + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: path, Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + sb := foundationClaim(t, m, "catalog") + m.Close() + cfg.Volumes = nil + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + _, _, err := m.CreateVolume(t.Context(), "catalog", types.VolumeCreate{SizeGiB: 1}, "t1") + requireVolumeStatus(t, err, 409) + createManagedTest(t, m, "other") + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + createManagedTest(t, m, "catalog") +} + +func TestManagedReadFailureDoesNotChangeDurableLifecycle(t *testing.T) { + if os.Getenv("MANAGED_EMFILE_CHILD") == "" { + cmd := exec.Command(os.Args[0], "-test.run=^TestManagedReadFailureDoesNotChangeDurableLifecycle$") + cmd.Env = append(os.Environ(), "MANAGED_EMFILE_CHILD=1") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("isolated EMFILE test: %v\n%s", err, out) + } + return + } + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + createManagedTest(t, m, "data") + path := filepath.Join(m.managed.dir, "registry.json") + before, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + var previous syscall.Rlimit + if err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &previous); err != nil { + t.Fatal(err) + } + limit := previous + limit.Cur = 64 + if err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil { + t.Fatal(err) + } + var files []*os.File + for { + f, openErr := os.Open("/dev/null") + if errors.Is(openErr, syscall.EMFILE) { + break + } + if openErr != nil { + t.Fatal(openErr) + } + files = append(files, f) + } + _, getErr := m.GetVolume("data", "t1") + _, _, placementErr := m.VolumePlacement(testKey, "t1", []string{"data"}) + for _, f := range files { + _ = f.Close() + } + if err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &previous); err != nil { + t.Fatal(err) + } + requireVolumeStatus(t, getErr, 503) + requireVolumeStatus(t, placementErr, 503) + after, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(before, after) { + t.Fatal("read-only lookup changed the durable registry") + } + info, err := m.GetVolume("data", "t1") + if err != nil || info.State != "ready" { + t.Fatalf("transient open error persisted: %+v %v", info, err) + } +} + +func TestManagedRegistryParseFailureFailsStartup(t *testing.T) { + cfg := managedConfig(t) + m := openManagedTest(t, cfg, newManagedEngine()) + createManagedTest(t, m, "data") + if err := m.Close(); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json"), []byte("{"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := NewManager(t.Context(), cfg, newManagedEngine(), testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("a corrupt registry opened") + } +} diff --git a/sandboxd/pool/managed_grow.go b/sandboxd/pool/managed_grow.go new file mode 100644 index 00000000..c25cc09e --- /dev/null +++ b/sandboxd/pool/managed_grow.go @@ -0,0 +1,124 @@ +package pool + +import ( + "context" + "syscall" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func growInProgress(name string) error { + return &VolumeError{Status: 409, Code: volumeBusyCode, Message: "volume growth is in progress", Volume: name} +} + +// GrowVolume grows an unheld managed volume to an absolute size; repeating the current size is a no-op. +func (m *Manager) GrowVolume(ctx context.Context, name, expectedID string, sizeGiB int64, tenant string) (types.ManagedVolumeInfo, error) { + if !types.ValidVolumeName(name) || sizeGiB < 1 || sizeGiB > 1024 { + return types.ManagedVolumeInfo{}, volumeError(400, "invalid_volume_grow", "invalid volume name or size_gib (1 through 1024)") + } + r := m.managed + if r == nil { + return types.ManagedVolumeInfo{}, managedNotFound() + } + if err := ctx.Err(); err != nil { + return types.ManagedVolumeInfo{}, err + } + target := sizeGiB << 30 + m.mu.Lock() + r.mu.Lock() + defer r.mu.Unlock() + v, err := r.admitGrow(name, expectedID, tenant, target) + if err == nil && target > v.Size { + err = m.volumeRefusalLocked(v) + } + if err == nil && target > v.Size { + // recorded under m.mu, so a claim that reserves the name afterwards refuses on grow_to + v.GrowTo = target + r.data.Volumes[name] = v + } + m.mu.Unlock() + if err != nil { + return types.ManagedVolumeInfo{}, err + } + if v.Size == target { + return r.info(v), nil + } + if err := r.commitAndRelock(); err != nil { + current := r.data.Volumes[name] + current.GrowTo = 0 + r.data.Volumes[name] = current + return types.ManagedVolumeInfo{}, err + } + r.mu.Unlock() + actual, growErr := r.growBacking(v) + r.mu.Lock() + v = settleGrowth(r.data.Volumes[name], actual) + r.data.Volumes[name] = v + if err := r.commitAndRelock(); err != nil { + return types.ManagedVolumeInfo{}, err + } + if growErr != nil { + return types.ManagedVolumeInfo{}, managedStorageError() + } + return r.info(v), nil +} + +func (r *volumeRegistry) admitGrow(name, expectedID, tenant string, target int64) (managedVolume, error) { + if r.closed || r.writeFault != nil { + return managedVolume{}, managedStorageError() + } + v, ok := r.data.Volumes[name] + if !ok || tenant != "" && tenant != v.Tenant { + return managedVolume{}, managedNotFound() + } + switch { + case expectedID == "": + return v, volumeError(428, "volume_precondition_required", "If-Match must identify the volume") + case string(v.ID) != expectedID: + return v, volumeError(412, "volume_precondition_failed", "volume identity does not match") + case v.GrowTo != 0: + return v, growInProgress(name) + case v.State == volumeRecovery: + return v, managedRecoveryError() + case v.State != volumeReady: + return v, volumeError(409, "volume_not_ready", "volume is being created or deleted") + case target < v.Size: + return v, volumeError(409, "volume_grow_conflict", "target is below the volume's capacity; volumes cannot shrink") + } + return v, nil +} + +// growBacking extends the recorded backing to v.GrowTo and reports the size it left behind. +func (r *volumeRegistry) growBacking(v managedVolume) (backingIdentity, error) { + if err := r.verifyDirectories(); err != nil { + return backingIdentity{}, err + } + f, err := openPrivateFile(r.path(v.ID), syscall.O_RDWR) + if err != nil { + return backingIdentity{}, err + } + defer func() { _ = f.Close() }() + st, err := f.Stat() + if err != nil { + return backingIdentity{}, err + } + raw := st.Sys().(*syscall.Stat_t) + actual := backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()} + if actual.Device != v.Backing.Device || actual.Inode != v.Backing.Inode { + return actual, errBackingMismatch + } + if err := f.Truncate(v.GrowTo); err != nil { + return actual, err + } + actual.Size = v.GrowTo + return actual, f.Sync() +} + +// settleGrowth adopts GrowTo only when the backing already has that size, then clears the intent. +func settleGrowth(v managedVolume, actual backingIdentity) managedVolume { + if v.GrowTo != 0 && actual.Device == v.Backing.Device && actual.Inode == v.Backing.Inode && actual.Size == v.GrowTo { + v.Size, v.Backing.Size = v.GrowTo, v.GrowTo + } + v.GrowTo = 0 + return v +} diff --git a/sandboxd/pool/managed_grow_test.go b/sandboxd/pool/managed_grow_test.go new file mode 100644 index 00000000..e30a61a6 --- /dev/null +++ b/sandboxd/pool/managed_grow_test.go @@ -0,0 +1,130 @@ +package pool + +import ( + "encoding/json/v2" + "errors" + "os" + "path/filepath" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func growManagedTest(t *testing.T, m *Manager, name, id string, sizeGiB int64) (types.ManagedVolumeInfo, error) { + t.Helper() + return m.GrowVolume(t.Context(), name, id, sizeGiB, "t1") +} + +func requireBackingSize(t *testing.T, path string, want int64) { + t.Helper() + st, err := os.Stat(path) + if err != nil || st.Size() != want { + t.Fatalf("backing file: %v %v, want %d bytes", st, err, want) + } +} + +func TestManagedGrowUnheld(t *testing.T) { + cfg := managedConfig(t) + m := openManagedTest(t, cfg, newManagedEngine()) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + grown, err := growManagedTest(t, m, "data", v.ID, 2) + if err != nil || grown.SizeBytes != 2<<30 || grown.State != "ready" { + t.Fatalf("grow 1->2 GiB: %+v %v", grown, err) + } + requireBackingSize(t, path, 2<<30) + if got, getErr := m.GetVolume("data", "t1"); getErr != nil || got.SizeBytes != 2<<30 || got.State != "ready" { + t.Fatalf("GET after grow: %+v %v, want ready at 2 GiB", got, getErr) + } + + registry := filepath.Join(cfg.ManagedVolumeDir, "registry.json") + before, err := os.ReadFile(registry) + if err != nil { + t.Fatal(err) + } + if replay, replayErr := growManagedTest(t, m, "data", v.ID, 2); replayErr != nil || replay.SizeBytes != 2<<30 { + t.Fatalf("same target again: %+v %v, want 2 GiB unchanged", replay, replayErr) + } + if after, readErr := os.ReadFile(registry); readErr != nil || string(after) != string(before) { + t.Fatalf("same target rewrote the registry: %v", readErr) + } + requireBackingSize(t, path, 2<<30) + + _, err = growManagedTest(t, m, "data", v.ID, 1) + requireVolumeStatus(t, err, 409) + requireBackingSize(t, path, 2<<30) + + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + _, err = growManagedTest(t, m, "data", v.ID, 3) + if busy := requireVolumeStatus(t, err, 409); len(busy.Holders) != 1 || busy.Holders[0].SandboxID != sb.ID { + t.Fatalf("grow of a held volume named %+v, want holder %s", busy.Holders, sb.ID) + } + requireBackingSize(t, path, 2<<30) +} + +func TestManagedGrowIntentSettles(t *testing.T) { + t.Run("grown file", func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + createManagedTest(t, m, "other") + path := m.managed.path(volumeID(v.ID)) + writes := 0 + m.managed.commitFile = func(path string, raw []byte) error { + writes++ + if writes == 2 { + return errors.New("registry write after the truncate failed") + } + return writeVolumeSnapshot(path, raw) + } + _, err := growManagedTest(t, m, "data", v.ID, 2) + if ve := requireVolumeStatus(t, err, 503); ve.Code != "volume_storage_unavailable" { + t.Fatalf("failed grow commit: %+v, want the retryable storage error", ve) + } + requireBackingSize(t, path, 2<<30) + m.Close() + m = openManagedTest(t, cfg, e) + if got, err := m.GetVolume("data", "t1"); err != nil || got.State != "ready" || got.SizeBytes != 2<<30 { + t.Fatalf("reopen after a lost grow commit: %+v %v, want ready at 2 GiB", got, err) + } + if _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("other")); err != nil { + t.Fatalf("claim of another volume after reopen: %v", err) + } + }) + t.Run("file not grown", func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + m.Close() + registry := filepath.Join(cfg.ManagedVolumeDir, "registry.json") + raw, err := os.ReadFile(registry) + if err != nil { + t.Fatal(err) + } + var doc map[string]any + if err = json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + doc["volumes"].(map[string]any)["data"].(map[string]any)["grow_to"] = 2 << 30 + if raw, err = json.Marshal(doc); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(registry, raw, 0o600); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if got, err := m.GetVolume("data", "t1"); err != nil || got.State != "ready" || got.SizeBytes != 1<<30 { + t.Fatalf("reopen with an unapplied grow intent: %+v %v, want ready at 1 GiB", got, err) + } + requireBackingSize(t, path, 1<<30) + if _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")); err != nil { + t.Fatalf("claim after the intent settled: %v", err) + } + }) +} diff --git a/sandboxd/pool/managed_integration_test.go b/sandboxd/pool/managed_integration_test.go new file mode 100644 index 00000000..fd9061eb --- /dev/null +++ b/sandboxd/pool/managed_integration_test.go @@ -0,0 +1,215 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestManagedStoreRequiresAuthenticatedRootConfiguration(t *testing.T) { + cfg := managedConfig(t) + cfg.APIToken = "" + _, err := NewManager(t.Context(), cfg, newManagedEngine(), testSecrets(t), Backends{}) + if err == nil || !strings.Contains(err.Error(), "api_token") { + t.Fatalf("anonymous managed store opened: %v", err) + } +} + +func TestManagedNamesAndCatalogSymlinksSurviveRestart(t *testing.T) { + cfg := managedConfig(t) + image := writeVolumeImage(t, "catalog.img", "catalog") + link := filepath.Join(t.TempDir(), "catalog-link") + if err := os.Symlink(image, link); err != nil { + t.Fatal(err) + } + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: link, Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "kb-data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("kb-data", "catalog")) + if err != nil { + t.Fatal(err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if got, err := m.GetVolume("kb-data", "t1"); err != nil || got.ID != v.ID { + t.Fatalf("reopened name: %+v %v", got, err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } +} + +func TestManagedRejectsCatalogSymlinkToManagedBacking(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + link := filepath.Join(t.TempDir(), "catalog-link") + if err := os.Symlink(m.managed.path(volumeID(v.ID)), link); err != nil { + t.Fatal(err) + } + m.Close() + cfg.Volumes = []config.VolumeSpec{{Name: "alias", Path: link, Writable: true}} + if next, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + next.Close() + t.Fatal("managed backing alias accepted as an independent catalog writer") + } +} + +type managedCancelEngine struct { + *managedTestEngine + entered chan struct{} + vm string +} + +func (e *managedCancelEngine) DiskAttach(ctx context.Context, name string, spec engine.VolumeSpec) error { + if spec.Name == "catalog" && (e.vm == "" || e.vm == name) { + close(e.entered) + <-ctx.Done() + return ctx.Err() + } + return e.managedTestEngine.DiskAttach(ctx, name, spec) +} + +func TestManagedReleaseCancelsExistingVolumeRunAndRetainsFullUnion(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog"), Writable: true}} + e := &managedCancelEngine{managedTestEngine: newManagedEngine(), entered: make(chan struct{})} + m, openErr := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if openErr != nil { + t.Fatal(openErr) + } + t.Cleanup(func() { m.Close() }) + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + createManagedTest(t, m, "data") + sb, openErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if openErr != nil { + t.Fatal(openErr) + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, err := m.PatchVolumes(ctx, sb.ID, "", types.VolumePatch{Add: []types.Volume{{Name: "catalog"}}}) + done <- err + }() + <-e.entered + e.stuckRemove.Store(true) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err == nil { + t.Fatal("surviving VMM reported removed") + } + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("mutation cancellation: %v", err) + } + case <-time.After(time.Second): + t.Fatal("release did not cancel the live volume run") + } + requireManagedBusy(t, m, "data", holderReleasing) + _, openErr = m.ClaimProvision(t.Context(), testKey, managedOptions("catalog")) + if !errors.Is(openErr, ErrVolumeBusy) { + t.Fatalf("uncertain catalog writer admitted: %v", openErr) + } + e.stuckRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + requireManagedClaimable(t, m, "data") +} + +func TestManagedExpiryCancelsExistingVolumeRun(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog"), Writable: true}} + e := &managedCancelEngine{managedTestEngine: newManagedEngine(), entered: make(chan struct{})} + m, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { m.Close() }) + if err = m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + createManagedTest(t, m, "data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + e.vm = sb.VMName + catalogReader(t, m) + done := make(chan error, 1) + go func() { + _, err := m.PatchVolumes(t.Context(), sb.ID, "t1", types.VolumePatch{Add: []types.Volume{{Name: "catalog"}}}) + done <- err + }() + <-e.entered + m.mu.Lock() + sb.Deadline = time.Now().Add(-time.Second) + m.mu.Unlock() + m.reapOnce(t.Context()) + waitFor(t, func() bool { return volumeHoldersOf(m, "data") == (volumeHolders{}) }) + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("mutation cancellation: %v", err) + } + case <-time.After(time.Second): + t.Fatal("expiry did not cancel the live volume run") + } + if got := volumeHoldersOf(m, "catalog"); got != (volumeHolders{readers: 1}) { + t.Fatalf("expired claim left catalog admission %+v, want only the bystander reader", got) + } +} + +func catalogReader(t *testing.T, m *Manager) *types.Sandbox { + t.Helper() + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "t1", Volumes: []types.Volume{{Name: "catalog", Mode: types.VolumeModeRO, AttachOnly: true}}}) + if err != nil { + t.Fatalf("bystander catalog reader: %v", err) + } + return sb +} + +func TestManagedExpiryDropsVolumeDiagnostics(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog")}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb := foundationClaim(t, m, "data") + if _, err := m.PatchVolumes(t.Context(), sb.ID, "t1", types.VolumePatch{Add: []types.Volume{{Name: "catalog"}}}); err != nil { + t.Fatal(err) + } + e.detachVolumeErr = &engine.VolumeMountConflict{Message: `cannot unmount "/volumes/catalog"; close guest references and retry`} + if _, err := m.PatchVolumes(t.Context(), sb.ID, "t1", types.VolumePatch{Remove: []string{"catalog"}}); err == nil { + t.Fatal("busy detach succeeded") + } + e.detachVolumeErr = nil + m.mu.Lock() + recorded := len(m.volumeErrors[sb]) + sb.Deadline = time.Now().Add(-time.Second) + m.mu.Unlock() + if recorded == 0 { + t.Fatal("busy detach recorded no diagnostic") + } + m.reapOnce(t.Context()) + waitFor(t, func() bool { _, live := m.Sandbox(sb.ID); return !live }) + m.mu.Lock() + defer m.mu.Unlock() + if leaked := m.volumeErrors[sb]; leaked != nil { + t.Fatalf("expired claim kept its diagnostics: %v", leaked) + } +} diff --git a/sandboxd/pool/managed_journal_test.go b/sandboxd/pool/managed_journal_test.go new file mode 100644 index 00000000..2d918c55 --- /dev/null +++ b/sandboxd/pool/managed_journal_test.go @@ -0,0 +1,252 @@ +package pool + +import ( + "context" + "errors" + "os" + "sync" + "sync/atomic" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/outbound" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +// staleVMEngine fails `vm list`, `vm rm` or `vm inspect` on demand. +type staleVMEngine struct { + *managedTestEngine + mu sync.Mutex + listErr error + stuck, uninspectable string +} + +func (e *staleVMEngine) List(ctx context.Context) ([]types.VMRecord, error) { + e.mu.Lock() + err := e.listErr + e.mu.Unlock() + if err != nil { + return nil, err + } + return e.managedTestEngine.List(ctx) +} + +func (e *staleVMEngine) Remove(ctx context.Context, name string) error { + e.mu.Lock() + stuck := e.stuck == name + e.mu.Unlock() + if stuck { + return errors.New("vm rm: device busy") + } + return e.managedTestEngine.Remove(ctx, name) +} + +func (e *staleVMEngine) Inspect(ctx context.Context, name string) (types.VMRecord, bool, error) { + e.mu.Lock() + failed := e.uninspectable == name + e.mu.Unlock() + if failed { + return types.VMRecord{}, false, errors.New("vm inspect: transient failure") + } + return e.managedTestEngine.Inspect(ctx, name) +} + +func (e *staleVMEngine) set(edit func(*staleVMEngine)) { + e.mu.Lock() + defer e.mu.Unlock() + edit(e) +} + +func requireManagedBusy(t *testing.T, m *Manager, name, state string) { + t.Helper() + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions(name)) + v := requireVolumeStatus(t, err, 409) + if len(v.Holders) == 0 || v.Holders[0].State != state { + t.Fatalf("claim of %s refused with holders %+v, want state %q", name, v.Holders, state) + } +} + +func TestManagedListFailureLeavesClaimsAndVMs(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + sb := &types.Sandbox{ID: "sb_ordinary", VMName: "sbx-ordinary", Key: egKey, VsockSocket: "/fake/vsock", Token: "token"} + e.vms[sb.VMName] = sb.VsockSocket + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + withOutbound(t, m, func(o *outbound.Options) { o.LockNIC = true; o.Sweep = func(map[string]bool) error { return nil } }) + inventoryErr := errors.New("cocoon vm list: transient failure") + m.eng = &staleVMEngine{managedTestEngine: e, listErr: inventoryErr} + if err := m.Reconcile(t.Context()); !errors.Is(err, inventoryErr) { + t.Fatalf("startup accepted an unknown inventory: %v", err) + } + durable, err := newClaimStore(cfg.DataDir, false).load() + if err != nil { + t.Fatal(err) + } + if _, kept := durable[sb.ID]; !kept { + t.Error("a transient list failure deleted a claim row") + } + if rec, found, err := e.Inspect(t.Context(), sb.VMName); err != nil || !found || rec.State != vmStateRunning { + t.Fatalf("a transient list failure touched a claim's VM: found=%v state=%s err=%v", found, rec.State, err) + } +} + +func TestManagedRestartAfterVMLossConverges(t *testing.T) { + cfg := managedConfig(t) + base := newManagedEngine() + m := openManagedTest(t, cfg, base) + createManagedTest(t, m, "data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + if err = m.Close(); err != nil { + t.Fatal(err) + } + if err = base.fakeEngine.Remove(t.Context(), sb.VMName); err != nil { + t.Fatal(err) + } + if _, err = base.RunCold(t.Context(), "sbx-orphan", testKey, ""); err != nil { + t.Fatal(err) + } + e := &staleVMEngine{managedTestEngine: base, stuck: "sbx-orphan"} + for restart := range 2 { + m = openManagedTest(t, cfg, base) + m.eng = e + if err = m.Reconcile(t.Context()); err != nil { + t.Fatalf("restart %d: %v", restart, err) + } + if _, live := m.Sandbox(sb.ID); live { + t.Fatalf("restart %d re-adopted a claim whose VM is gone", restart) + } + requireManagedClaimable(t, m, "data") + if err = m.Close(); err != nil { + t.Fatal(err) + } + } +} + +func TestRegistryWriteFaultIsLocal(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog"), Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + managed := foundationClaim(t, m, "data") + catalog := foundationClaim(t, m, "catalog") + m.managed.commitFile = func(string, []byte) error { return errors.New("disk full") } + _, _, err := m.CreateVolume(t.Context(), "other", types.VolumeCreate{SizeGiB: 1}, "t1") + requireVolumeStatus(t, err, 503) + for _, sb := range []*types.Sandbox{managed, catalog} { + if _, err := m.AgentSocket(sb.ID, sb.Token); err != nil { + t.Errorf("a registry write fault took the socket of %v: %v", sb.Volumes, err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Errorf("a registry write fault blocked release of %v: %v", sb.Volumes, err) + } + if e.vmAlive(sb.VMName) { + t.Errorf("release of %v left its VM", sb.Volumes) + } + } +} + +func TestManagedStartupQueuesUnremovableStaleVM(t *testing.T) { + for _, inspectFails := range []bool{false, true} { + t.Run(map[bool]string{false: "inventory known", true: "inspect fails"}[inspectFails], func(t *testing.T) { + cfg := managedConfig(t) + base := newManagedEngine() + m := openManagedTest(t, cfg, base) + work := createManagedTest(t, m, "work") + createManagedTest(t, m, "other") + if err := m.Close(); err != nil { + t.Fatal(err) + } + if _, err := base.RunCold(t.Context(), "sbx-stale", testKey, ""); err != nil { + t.Fatal(err) + } + base.mu.Lock() + base.disks = map[string][]types.VMDisk{"sbx-stale": {{Name: "work", Path: m.managed.path(volumeID(work.ID))}}} + base.mu.Unlock() + e := &staleVMEngine{managedTestEngine: base, stuck: "sbx-stale"} + if inspectFails { + e.uninspectable = "sbx-stale" + } + m = openManagedTest(t, cfg, base) + m.eng = e + if err := m.Reconcile(t.Context()); err != nil { + t.Fatalf("an unremovable stale VM failed startup: %v", err) + } + requireManagedBusy(t, m, "work", "releasing") + if inspectFails { + requireManagedBusy(t, m, "other", "releasing") + e.set(func(e *staleVMEngine) { e.uninspectable = "" }) + m.retryRemovals(t.Context()).Wait() + requireManagedBusy(t, m, "work", "releasing") + } + requireManagedClaimable(t, m, "other") + e.set(func(e *staleVMEngine) { e.stuck = "" }) + m.retryRemovals(t.Context()).Wait() + requireManagedClaimable(t, m, "work") + }) + } +} + +func TestManagedReleaseClaimsWriteFailureRecommits(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + createManagedTest(t, m, "data") + sb := foundationClaim(t, m, "data") + tmp := m.store.path + ".tmp" + if err := os.Mkdir(tmp, 0o700); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("a release whose claims write failed reported success") + } + if _, live := m.Sandbox(sb.ID); !live { + t.Fatal("a failed release dropped the claim") + } + if err := os.Remove(tmp); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + createManagedTest(t, m, "other") + requireManagedClaimable(t, m, "other") + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("retried release: %v", err) + } + requireManagedClaimable(t, m, "data") +} + +func TestWarmManagedClaimWritesNoRegistry(t *testing.T) { + e := newManagedEngine() + m := openManagedTest(t, managedConfig(t), e) + createManagedTest(t, m, "data") + createManagedTest(t, m, "spare") + warm := refillWarmVM(t, m) + e.mu.Lock() + e.pids[warm.VMName] = 4242 + e.mu.Unlock() + var writes atomic.Int32 + m.managed.commitFile = func(path string, raw []byte) error { + writes.Add(1) + return writeVolumeSnapshot(path, raw) + } + if _, err := m.ClaimWarm(t.Context(), testKey, managedOptions("data")); err != nil { + t.Fatal(err) + } + if _, err := m.ClaimWarm(t.Context(), testKey, managedOptions("spare")); !errors.Is(err, ErrNoWarm) { + t.Fatalf("second warm claim: %v, want a warm miss", err) + } + if _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("spare")); err != nil { + t.Fatal(err) + } + if n := writes.Load(); n != 0 { + t.Fatalf("a warm managed claim and a warm-miss fallback wrote registry.json %d times", n) + } +} diff --git a/sandboxd/pool/managed_live.go b/sandboxd/pool/managed_live.go new file mode 100644 index 00000000..48f9b254 --- /dev/null +++ b/sandboxd/pool/managed_live.go @@ -0,0 +1,17 @@ +package pool + +import ( + "fmt" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +// validateManagedTarget refuses a live change that would detach a managed volume; managed volumes attach at claim time only. +func (m *Manager) validateManagedTarget(held, target []types.Volume) error { + for _, volume := range held { + if _, managed := m.managedPath(volume.Name); managed && volumeIndex(target, volume.Name) < 0 { + return fmt.Errorf("%w: managed volumes cannot be removed live", ErrBadVolume) + } + } + return nil +} diff --git a/sandboxd/pool/managed_recovery_regression_test.go b/sandboxd/pool/managed_recovery_regression_test.go new file mode 100644 index 00000000..12ab532d --- /dev/null +++ b/sandboxd/pool/managed_recovery_regression_test.go @@ -0,0 +1,250 @@ +package pool + +import ( + "errors" + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestManagedQuarantineRemovalSurvivesRestart(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + t.Run(fmt.Sprint(syncClaims), func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + sb.Key.Net = types.NetEgress + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + e.stuckRemove.Store(true) + if m.quarantineClaim(t.Context(), sb) { + t.Fatal("ineffective removal reported success") + } + waitFor(t, m.store.synced) + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, claimErr, 409) + e.stuckRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || present { + t.Fatalf("quarantined VM survived removal retry after restart: present=%v err=%v", present, err) + } + requireManagedClaimable(t, m, "data") + }) + } +} + +func TestManagedOrdinaryFailedClaimRetainsOwnershipOnRestart(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + sb.Failed = "ordinary failure requiring explicit release" + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + m.retryRemovals(t.Context()).Wait() + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || !present { + t.Fatalf("ordinary failed claim removed: %v %v", present, err) + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, claimErr, 409) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } +} + +func TestManagedStoreRejectsSweepOwnedPathsBeforeCreatingFiles(t *testing.T) { + for _, rel := range []string{"readers", "readers/managed", "fork-persistent", "fork-persistent/managed", "goldens", "goldens/managed"} { + t.Run(rel, func(t *testing.T) { + cfg := managedConfig(t) + cfg.ManagedVolumeDir = filepath.Join(cfg.DataDir, rel) + m, err := NewManager(t.Context(), cfg, newManagedEngine(), testSecrets(t), testBackends(t, cfg)) + if err == nil { + m.Close() + t.Fatal("accepted managed store inside startup cleanup namespace") + } + if _, err := os.Stat(cfg.ManagedVolumeDir); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("unsafe store modified before rejection: %v", err) + } + }) + } +} + +func TestManagedStorePreservesBackingWhenCleanupAliasesDisks(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + t.Run(namespace, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + v := createManagedTest(t, m, "data") + backing := m.managed.path(volumeID(v.ID)) + f, openErr := os.OpenFile(backing, os.O_WRONLY, 0) + if openErr != nil { + t.Fatal(openErr) + } + _, writeErr := f.WriteAt([]byte("preserved"), 4096) + if err := errors.Join(writeErr, f.Close(), m.Close()); err != nil { + t.Fatal(err) + } + cleanup := filepath.Join(cfg.DataDir, namespace) + if err := os.Remove(cleanup); err != nil && !errors.Is(err, os.ErrNotExist) { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(cfg.ManagedVolumeDir, "disks"), cleanup); err != nil { + t.Fatal(err) + } + next, openErr := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if openErr == nil { + t.Error("constructor accepted cleanup alias into managed backing directory") + if err := next.Reconcile(t.Context()); err != nil { + t.Errorf("reconcile: %v", err) + } + next.Close() + } + f, openErr = os.Open(backing) + if openErr != nil { + t.Fatalf("managed backing lost: %v", openErr) + } + defer f.Close() + marker := make([]byte, len("preserved")) + if _, err := f.ReadAt(marker, 4096); err != nil || string(marker) != "preserved" { + t.Fatalf("managed bytes changed: %q %v", marker, err) + } + }) + } +} + +func TestManagedStoppedClaimRetainsOwnershipOnRestart(t *testing.T) { + for _, failed := range []bool{false, true} { + name := "new-failure-release" + if failed { + name = "previous-failure-expiry" + } + t.Run(name, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + sb := foundationClaim(t, m, "data") + backing := m.managed.path(volumeID(v.ID)) + identity, err := inspectBacking(backing) + if err != nil { + t.Fatal(err) + } + if failed { + sb.Failed = "ordinary failure requiring explicit release" + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + } + if err := m.Close(); err != nil { + t.Fatal(err) + } + e.mu.Lock() + e.stopped[sb.VMName] = true + e.pids[sb.VMName] = 0 + e.mu.Unlock() + for restart := range 2 { + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if _, live := m.Sandbox(sb.ID); !live { + t.Fatalf("restart %d discarded the durable stopped claim", restart) + } + runLiveness(t, m) + listed := m.Sandboxes("t1", "", nil) + if len(listed) != 1 || listed[0].ID != sb.ID || listed[0].Failed == "" { + t.Fatalf("restart %d did not list the failed owner: %+v", restart, listed) + } + if rec, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || !present || rec.State != vmStateStopped || rec.PID != 0 { + t.Fatalf("restart %d changed stopped VM record: %+v present=%t err=%v", restart, rec, present, err) + } + if current, err := inspectBacking(backing); err != nil || current != identity { + t.Fatalf("restart %d changed backing identity: %+v %v", restart, current, err) + } + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 409) + if _, _, _, wakeErr := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); !errors.Is(wakeErr, ErrFailed) { + t.Fatalf("new data plane request returned %v, want ErrFailed", wakeErr) + } + if wakeErr := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(wakeErr, ErrFailed) { + t.Fatalf("explicit wake returned %v, want ErrFailed", wakeErr) + } + e.mu.Lock() + changed := len(e.removes) != 0 || len(e.starts) != 0 + e.mu.Unlock() + if changed { + t.Fatal("recovery cold-booted or removed the stopped owner") + } + persisted, err := m.store.load() + if err != nil || persisted[sb.ID] == nil || persisted[sb.ID].Failed == "" || persisted[sb.ID].VMName != sb.VMName { + t.Fatalf("failed claim identity not durable: %+v %v", persisted[sb.ID], err) + } + if !failed { + if events := usageEventsOf(t, m, sb.ID); events != "claim,vmm_exit,vmm_failed" { + t.Fatalf("restart %d lost or repeated VMM exit audit: %s", restart, events) + } + } + if restart == 0 { + if err := m.Close(); err != nil { + t.Fatal(err) + } + } + } + if failed { + m.mu.Lock() + m.claimed[sb.ID].Deadline = time.Now().Add(-time.Minute) + m.mu.Unlock() + m.reapOnce(t.Context()) + } else if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return volumeHoldersOf(m, "data") == (volumeHolders{}) }) + if _, live := m.Sandbox(sb.ID); live { + t.Fatal("release or expiry retained the failed claim") + } + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || present { + t.Fatalf("release or expiry retained VM record: present=%t err=%v", present, err) + } + if current, err := inspectBacking(backing); err != nil || current != identity { + t.Fatalf("release or expiry changed backing identity: %+v %v", current, err) + } + if _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")); err != nil { + t.Fatalf("release or expiry did not free writer admission: %v", err) + } + }) + } +} diff --git a/sandboxd/pool/managed_release_test.go b/sandboxd/pool/managed_release_test.go new file mode 100644 index 00000000..cd454939 --- /dev/null +++ b/sandboxd/pool/managed_release_test.go @@ -0,0 +1,282 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/archives/archivestest" + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func (f *fakeEngine) vmAlive(name string) bool { + f.mu.Lock() + defer f.mu.Unlock() + _, alive := f.vms[name] + return alive +} + +func (f *fakeEngine) captures() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.hibernates) +} + +func requireManagedClaimable(t *testing.T, m *Manager, name string) { + t.Helper() + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions(name)) + if err != nil { + t.Fatalf("volume %s is not claimable: %v", name, err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } +} + +func reopenSleepTest(t *testing.T, cfg *config.Config, e *sleepTestEngine) *Manager { + t.Helper() + m, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = m.Close() }) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + return m +} + +func TestManagedReleaseOnMetastoreNodeReturns(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + m.archives = archivestest.NewMem() + done := make(chan error, 1) + go func() { done <- m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}) }() + select { + case err := <-done: + if err != nil { + t.Fatalf("release: %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("release of a sleep-capable managed claim on a metastore node did not return") + } + if e.vmAlive(sb.VMName) { + t.Fatal("release returned with the VM still present") + } + requireManagedClaimable(t, m, "data") +} + +func TestManagedReleasePreemptsInFlightSleep(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + e.hookEntered = make(chan struct{}, 4) + e.hookContinue = make(chan struct{}) + hibernated := make(chan error, 1) + go func() { hibernated <- m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) }() + select { + case <-e.hookEntered: + case <-time.After(3 * time.Second): + t.Fatal("hibernate did not reach the prepare hook") + } + released := make(chan error, 1) + go func() { released <- m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}) }() + select { + case err := <-released: + if err != nil { + close(e.hookContinue) + t.Fatalf("release: %v", err) + } + case <-time.After(3 * time.Second): + close(e.hookContinue) + t.Fatal("release waited for the in-flight prepare hook") + } + alive := e.vmAlive(sb.VMName) + close(e.hookContinue) + if alive { + t.Fatal("release returned with the VM still present") + } + if err := <-hibernated; err == nil { + t.Fatal("a hibernate preempted by release reported success") + } + if n := e.captures(); n != 0 { + t.Fatalf("%d captures followed a preempting release", n) + } + if _, err := m.ClaimProvision(t.Context(), testKey, managedReaderOptions("data")); !errors.Is(err, ErrVolumeNeedsRecovery) { + t.Fatalf("ro claim after a release that preempted prepare = %v, want the dirty marker kept", err) + } + requireManagedClaimable(t, m, "data") +} + +func managedReaderOptions(name string) ClaimOptions { + return ClaimOptions{Tenant: "t1", Volumes: []types.Volume{{Name: name, Mode: types.VolumeModeRO, Mount: "/" + name}}} +} + +func TestManagedSleepingClaimTeardownLeavesVolumeClean(t *testing.T) { + for _, expiry := range []bool{false, true} { + t.Run(map[bool]string{false: "release", true: "expiry"}[expiry], func(t *testing.T) { + m, _, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + if expiry { + m.mu.Lock() + sb.Deadline = time.Now().Add(-time.Minute) + m.mu.Unlock() + m.reapOnce(t.Context()) + } else if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return volumeHoldersOf(m, "data") == (volumeHolders{}) }) + reader, err := m.ClaimProvision(t.Context(), testKey, managedReaderOptions("data")) + if err != nil { + t.Fatalf("ro claim after tearing down a cleanly detached sleeping writer: %v", err) + } + if err := m.Release(t.Context(), reader.ID, Cred{Token: reader.Token}); err != nil { + t.Fatal(err) + } + }) + } +} + +func TestManagedExpiryScheduledOnce(t *testing.T) { + e := newManagedEngine() + m := openManagedTest(t, managedConfig(t), e) + createManagedTest(t, m, "data") + managed, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + plain, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "t1"}) + if err != nil { + t.Fatal(err) + } + m.mu.Lock() + managed.Deadline = time.Now().Add(-time.Minute) + m.mu.Unlock() + // a wake, sleep or stuck device call owns the managed claim's Transition + managed.Transition.Lock() + for range cap(m.maintSem) { + m.reapOnce(t.Context()) + } + waitFor(t, func() bool { return len(m.maintSem) == cap(m.maintSem) || !e.vmAlive(managed.VMName) }) + m.mu.Lock() + plain.Deadline = time.Now().Add(-time.Minute) + m.mu.Unlock() + m.reapOnce(t.Context()) + reaped := false + for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline) && !reaped; time.Sleep(20 * time.Millisecond) { + reaped = !e.vmAlive(plain.VMName) + } + managed.Transition.Unlock() + waitFor(t, func() bool { return !e.vmAlive(managed.VMName) && len(m.maintSem) == 0 }) + if !reaped { + t.Fatal("an ordinary expired claim was not reaped while one expired managed claim's Transition was held") + } + if c := m.Counters(); c.Reaps != 2 { + t.Fatalf("reaps = %d over one expired managed and one expired ordinary claim, want 2", c.Reaps) + } + requireManagedClaimable(t, m, "data") +} + +func TestFailedManagedReleaseKeepsStreamsUsable(t *testing.T) { + m, _, sb, _ := newSleepTest(t) + if err := os.Mkdir(m.store.path+".tmp", 0o700); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("a release whose claims write failed reported success") + } + if err := os.Remove(m.store.path + ".tmp"); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + if _, err := m.AgentSocket(sb.ID, sb.Token); err != nil { + t.Fatalf("rolled-back release refused the passive socket: %v", err) + } + sock, fenced, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + defer release() + } + if err != nil || sock == "" { + t.Fatalf("rolled-back release refused exec: socket=%q err=%v", sock, err) + } + select { + case <-fenced: + t.Fatal("rolled-back release handed out a pre-fenced stream") + default: + } +} + +type resumeHookEngine struct { + *sleepTestEngine + onResume func() +} + +func (e *resumeHookEngine) RunVolumeHook(ctx context.Context, sock string, call engine.VolumeHookCall) error { + if call.Action == "resume" && e.onResume != nil { + e.onResume() + } + return e.sleepTestEngine.RunVolumeHook(ctx, sock, call) +} + +func TestManagedWakeFinalWriteFailureStaysAwake(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + requireSleep(t, m, sb) + tmp := m.store.path + ".tmp" + m.eng = &resumeHookEngine{sleepTestEngine: e, onResume: func() { _ = os.Mkdir(tmp, 0o700) }} + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("a failed final claims write after a verified mount failed the wake: %v", err) + } + requireExec(t, m, sb) + if n := e.hookCalls("resume"); n != 1 { + t.Fatalf("resume hook ran %d times, want 1", n) + } + if err := os.Remove(tmp); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + if err := m.Close(); err != nil { + t.Fatal(err) + } + recovered := reopenSleepTest(t, cfg, e) + requireExec(t, recovered, sb) + if n := e.hookCalls("resume"); n != 1 { + t.Fatalf("claims.json did not converge to awake: resume hook ran %d times after restart, want 1", n) + } +} + +func TestManagedWakeCrashBeforeClearReplaysResumeOnce(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + requireSleep(t, m, sb) + files := []string{m.store.path, filepath.Join(cfg.ManagedVolumeDir, "registry.json")} + saved := map[string][]byte{} + m.eng = &resumeHookEngine{sleepTestEngine: e, onResume: func() { + for _, path := range files { + raw, err := os.ReadFile(path) + if err != nil { + t.Error(err) + } + saved[path] = raw + } + }} + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if err := m.Close(); err != nil { + t.Fatal(err) + } + for path, raw := range saved { + if err := os.WriteFile(path, raw, 0o600); err != nil { + t.Fatal(err) + } + } + recovered := reopenSleepTest(t, cfg, e) + requireExec(t, recovered, sb) + if n := e.hookCalls("resume"); n != 2 { + t.Fatalf("resume hook ran %d times across the crash, want one replay (2)", n) + } + if len(e.restores) != 1 { + t.Fatalf("restores = %v, want the original restore only", e.restores) + } +} diff --git a/sandboxd/pool/managed_sleep.go b/sandboxd/pool/managed_sleep.go new file mode 100644 index 00000000..6e237261 --- /dev/null +++ b/sandboxd/pool/managed_sleep.go @@ -0,0 +1,417 @@ +package pool + +import ( + "cmp" + "context" + "errors" + "fmt" + "slices" + "strings" + "time" + + "github.com/projecteru2/core/log" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type managedSleepTarget bool + +const ( + managedRunning managedSleepTarget = false + managedSleeping managedSleepTarget = true +) + +func sleepRecoveryError() error { + return volumeError(409, "managed_sleep_recovery_required", "managed sleep ownership or VM state is uncertain; retain the backing and retry or release the sandbox") +} + +// managedSleepVolume resolves the one mounted rw managed volume of a claim whose shape can sleep. +func (m *Manager) managedSleepVolume(sb *types.Sandbox) (resolvedVolume, bool) { + r := m.managed + if r == nil { + return resolvedVolume{}, false + } + m.mu.Lock() + volumes, want := slices.Clone(sb.Volumes), sb.VolumesWant + shaped := len(volumes) == 1 && slices.Equal(want, volumes) && volumes[0].RW() && volumes[0].Mount != "" && sb.ArchiveCk == "" && sb.ArchivePending == "" && !sb.HasReplica() && !sb.Replicated() && sb.Key.Net == types.NetNone + m.mu.Unlock() + if !shaped { + return resolvedVolume{}, false + } + r.mu.Lock() + v, ok := r.data.Volumes[volumes[0].Name] + r.mu.Unlock() + if !ok { + return resolvedVolume{}, false + } + return resolvedVolume{disk: engine.VolumeSpec{Name: v.Name, Path: r.path(v.ID), RW: true}, applied: volumes[0], managedID: v.ID}, true +} + +// sleepHolds reports whether a sleeping claim holds name; its wake reconciles the backing. +func (m *Manager) sleepHolds(name string) bool { + m.mu.Lock() + defer m.mu.Unlock() + for _, sb := range m.claimed { + if sb.VolumeSleep != nil && volumeIndex(sb.Volumes, name) >= 0 { + return true + } + } + return false +} + +func (m *Manager) verifySleepBacking(sb *types.Sandbox, vol resolvedVolume) error { + r := m.managed + r.mu.Lock() + defer r.mu.Unlock() + v, ok := r.data.Volumes[vol.applied.Name] + if !ok || v.ID != vol.managedID || v.State != volumeReady || (sb.Tenant != "" && v.Tenant != sb.Tenant) { + return managedStorageError() + } + r.mu.Unlock() + err := r.verify(v) + r.mu.Lock() + if errors.Is(err, errBackingMismatch) { + return managedRecoveryError() + } + if err != nil { + return managedStorageError() + } + return nil +} + +func (m *Manager) managedSleepPreflight(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) (types.VMRecord, error) { + m.mu.Lock() + live := m.claimed[sb.ID] == sb + busy := m.volumeRuns[sb] != nil + _, outbound := m.view.Load().out.Resolve(sb, m.claimPooledLocked) + m.mu.Unlock() + if m.managedSleepHook == nil || busy || outbound { + return types.VMRecord{}, ErrVolumeCapture + } + if !live { + return types.VMRecord{}, ErrUnknownSandbox + } + if err := m.diskErr(); err != nil { + return types.VMRecord{}, err + } + if err := m.verifySleepBacking(sb, vol); err != nil { + return types.VMRecord{}, err + } + if err := m.eng.CheckLiveVolumeSupport(ctx); err != nil { + return types.VMRecord{}, err + } + rec, present, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return rec, err + } + if !present || rec.Config.Name != sb.VMName || rec.ID == "" || rec.LastTransitionReason == "" || rec.TransitionGeneration == 0 || rec.State != vmStateRunning || !exactSleepDisk(rec, vol) { + return rec, sleepRecoveryError() + } + if err := m.eng.VerifyMountedVolume(ctx, rec.VsockSocket, vol.applied.Name, vol.applied.Mount); err != nil { + return rec, err + } + return rec, nil +} + +func (m *Manager) driveManagedSleepLocked(ctx context.Context, sb *types.Sandbox, target managedSleepTarget) error { + vol, shaped := m.managedSleepVolume(sb) + m.mu.Lock() + live := m.claimed[sb.ID] == sb + m.mu.Unlock() + switch { + case !live: + return ErrUnknownSandbox + case !shaped && sb.VolumeSleep == nil: + return ErrVolumeCapture + case !shaped: + return sleepRecoveryError() + } + ctx = context.WithoutCancel(ctx) + start := time.Now() + if sb.VolumeSleep == nil { + if err := m.beginManagedSleep(ctx, sb, vol); err != nil { + return err + } + } else if err := m.store.commit(m.store.mark()); err != nil { + // an intent whose write failed must land before this call adds effects + return fmt.Errorf("persist managed sleep: %w", err) + } + for { + s := sb.VolumeSleep + if s == nil { + return nil + } + if err := m.verifySleepBacking(sb, vol); err != nil { + return err + } + switch s.Phase { + case types.SleepPreparing: + if err := m.setSleepPhase(ctx, sb, types.SleepAborting); err != nil { + return err + } + case types.SleepAborting: + if err := m.resumeManagedSleep(ctx, sb, vol, false); err != nil { + return err + } + if target == managedRunning { + return nil + } + return m.driveManagedSleepLocked(ctx, sb, target) + case types.SleepCapturing: + if err := m.captureManagedSleep(ctx, sb, vol); err != nil { + return err + } + case types.SleepSleeping: + if err := m.verifySleepingVM(ctx, sb); err != nil { + return err + } + if target == managedSleeping { + return nil + } + if err := m.setSleepPhase(ctx, sb, types.SleepWaking); err != nil { + return err + } + case types.SleepWaking: + if err := m.wakeManagedSleep(ctx, sb, vol); err != nil { + return err + } + case types.SleepResuming: + if err := m.resumeManagedSleep(ctx, sb, vol, true); err != nil { + return err + } + m.counters.wakes.Add(1) + m.counters.wakeNanos.Add(uint64(time.Since(start))) //nolint:gosec // durations are positive + m.recordUsage(ctx, usageEvent{Event: "wake", ID: sb.ID, VMName: sb.VMName}) + if target == managedRunning { + return nil + } + return m.driveManagedSleepLocked(ctx, sb, target) + default: + return sleepRecoveryError() + } + } +} + +func (m *Manager) beginManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) error { + rec, err := m.managedSleepPreflight(ctx, sb, vol) + if err != nil { + return err + } + hook := types.VolumeHook(*m.managedSleepHook) + hook.HookArgv = slices.Clone(hook.HookArgv) + next := &types.VolumeSleep{Operation: randHex(16), Phase: types.SleepPreparing, Snapshot: hibernatePrefix + strings.TrimPrefix(sb.VMName, vmPrefix) + "-" + randHex(3), Hook: hook, VMID: rec.ID, Generation: rec.TransitionGeneration} + live, err := m.commitIfLive(ctx, sb, func() { + sb.VolumeSleep = next + projectSleepState(sb, next) + }) + if !live { + return ErrUnknownSandbox + } + if err != nil { + // nothing has touched the guest yet, so the intent can still be withdrawn + m.mu.Lock() + sb.VolumeSleep, sb.PendingSnap = nil, "" + m.store.set(sb) + m.mu.Unlock() + return fmt.Errorf("persist managed sleep intent: %w", err) + } + m.out.Disarm(sb.ID, false) + return m.prepareManagedSleep(ctx, sb, vol) +} + +// journalSleep publishes next on the claim row; a failed write keeps memory and recommits but ends this call. +func (m *Manager) journalSleep(ctx context.Context, sb *types.Sandbox, next *types.VolumeSleep, sock string) error { + live, err := m.commitIfLive(ctx, sb, func() { + sb.VolumeSleep = next + projectSleepState(sb, next) + if sock != "" { + sb.VsockSocket = sock + } + }) + if !live { + return ErrUnknownSandbox + } + if err != nil { + return fmt.Errorf("persist managed sleep: %w", err) + } + return nil +} + +func (m *Manager) setSleepPhase(ctx context.Context, sb *types.Sandbox, phase types.SleepPhase) error { + next := *sb.VolumeSleep + next.Phase = phase + return m.journalSleep(ctx, sb, &next, "") +} + +func (m *Manager) prepareManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) error { + rec, err := m.sleepVM(ctx, sb, false) + if err == nil { + err = m.sleepHook(ctx, sb, vol, rec.VsockSocket, "prepare") + } + if err == nil { + err = m.eng.VerifyMountedVolume(ctx, rec.VsockSocket, vol.applied.Name, vol.applied.Mount) + if err == nil { + err = m.eng.DetachVolume(ctx, sb.VMName, rec.VsockSocket, vol.applied.Name, vol.applied.Mount) + } + } + if err == nil { + rec, err = m.sleepVM(ctx, sb, false) + if err == nil && len(rec.AttachedDevices.Disks) != 0 { + err = sleepRecoveryError() + } + } + if err != nil { + if persistErr := m.setSleepPhase(ctx, sb, types.SleepAborting); persistErr != nil { + return errors.Join(err, persistErr) + } + if resumeErr := m.resumeManagedSleep(ctx, sb, vol, false); resumeErr != nil { + return errors.Join(err, sleepRecoveryError(), resumeErr) + } + return err + } + return m.setSleepPhase(ctx, sb, types.SleepCapturing) +} + +func (m *Manager) captureManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) error { + s := sb.VolumeSleep + rec, found, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return err + } + if !found || rec.ID != s.VMID || rec.Config.Name != sb.VMName { + return sleepRecoveryError() + } + source, exists, err := m.eng.InspectHibernateSource(ctx, s.Snapshot) + if err != nil { + return err + } + if rec.State == vmStateRunning { + if exists || len(rec.AttachedDevices.Disks) != 0 { + return sleepRecoveryError() + } + return m.captureRunningManagedSleep(ctx, sb, vol) + } + if rec.State != vmStateStopped || rec.TransitionGeneration <= s.Generation || rec.LastTransitionReason != vmReasonStopUser || !exists { + return sleepRecoveryError() + } + next := *s + next.Phase, next.SourceID = types.SleepSleeping, source.ID + err = m.journalSleep(ctx, sb, &next, "") + if errors.Is(err, ErrUnknownSandbox) { + return err + } + // the VM is captured either way, so the billing window closes here + m.recordHibernate(ctx, sb) + return err +} + +func (m *Manager) captureRunningManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) error { + if _, err := m.sleepVM(ctx, sb, false); err != nil { + return err + } + snapshot := sb.VolumeSleep.Snapshot + effectErr := m.eng.Hibernate(ctx, sb.VMName, snapshot) + rec, found, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return errors.Join(effectErr, err) + } + _, exists, err := m.eng.InspectHibernateSource(ctx, snapshot) + if err != nil { + return errors.Join(effectErr, err) + } + if !found { + return errors.Join(effectErr, sleepRecoveryError()) + } + if rec.State != vmStateRunning { + return m.captureManagedSleep(ctx, sb, vol) + } + if exists { + return errors.Join(effectErr, sleepRecoveryError()) + } + if err = m.setSleepPhase(ctx, sb, types.SleepAborting); err != nil { + return errors.Join(effectErr, err) + } + if err = m.resumeManagedSleep(ctx, sb, vol, false); err != nil { + return errors.Join(effectErr, sleepRecoveryError(), err) + } + return cmp.Or(effectErr, errors.New("managed hibernate did not take effect")) +} + +func (m *Manager) wakeManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume) error { + rec, err := m.restoreManagedSleep(ctx, sb) + if err != nil { + return err + } + rec, err = m.consumeManagedSleepSource(ctx, sb) + if err != nil { + return err + } + sock, err := m.probeReady(ctx, sb.VMName, rec.VsockSocket, claimProbeTimeout) + if err != nil { + return err + } + if err = m.attachManagedSleep(ctx, sb, vol, sock, true); err != nil { + return err + } + next := *sb.VolumeSleep + next.Phase = types.SleepResuming + return m.journalSleep(ctx, sb, &next, sock) +} + +func (m *Manager) resumeManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume, restored bool) error { + var rec types.VMRecord + var err error + if restored { + rec, err = m.consumeManagedSleepSource(ctx, sb) + } else { + rec, err = m.sleepVM(ctx, sb, false) + } + if err != nil { + return err + } + if err = m.attachManagedSleep(ctx, sb, vol, rec.VsockSocket, restored); err != nil { + return err + } + logger := log.WithFunc("pool.resumeManagedSleep") + // the host verified the mount, so a failed hook leaves only the guest's writers paused + if err = m.sleepHook(ctx, sb, vol, rec.VsockSocket, "resume"); err != nil { + logger.Errorf(ctx, err, "resume hook of %s failed; its writers stay paused", sb.ID) + } + sock, err := m.probeReady(ctx, sb.VMName, rec.VsockSocket, claimProbeTimeout) + if err != nil { + return err + } + rec, err = m.sleepVM(ctx, sb, restored) + if err != nil { + return err + } + if !exactSleepDisk(rec, vol) { + return sleepRecoveryError() + } + if err = m.eng.VerifyMountedVolume(ctx, sock, vol.applied.Name, vol.applied.Mount); err != nil { + return err + } + live, err := m.commitIfLive(ctx, sb, func() { + sb.VolumeSleep = nil + sb.VsockSocket, sb.HibernateSnap, sb.PendingSnap, sb.ReaderSafe = sock, "", "", false + }) + if !live { + return ErrUnknownSandbox + } + if err != nil { + // memory never rolls back past a verified mount; the recommit converges the row + logger.Warnf(ctx, "persist resume of %s: %v; retrying", sb.ID, err) + } + return nil +} + +func projectSleepState(sb *types.Sandbox, s *types.VolumeSleep) { + sb.ReaderSafe = false + switch s.Phase { + case types.SleepPreparing, types.SleepAborting, types.SleepCapturing: + sb.PendingSnap, sb.HibernateSnap = s.Snapshot, "" + default: + sb.HibernateSnap, sb.PendingSnap = s.Snapshot, "" + } +} diff --git a/sandboxd/pool/managed_sleep_boundaries_test.go b/sandboxd/pool/managed_sleep_boundaries_test.go new file mode 100644 index 00000000..e6d18bb9 --- /dev/null +++ b/sandboxd/pool/managed_sleep_boundaries_test.go @@ -0,0 +1,226 @@ +package pool + +import ( + "context" + "errors" + "net" + "os" + "path/filepath" + "slices" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/egress" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/outbound/outboundtest" +) + +func TestManagedSleepFailedClaimRefusesDataPlane(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(map[bool]string{false: "disabled", true: "enabled"}[enabled], func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + if !enabled { + m.managedSleepHook = nil + } + e.mu.Lock() + e.stopped[sb.VMName] = true + e.mu.Unlock() + m.recoverVMM(t.Context(), sb) + if sb.Failed == "" { + t.Fatal("missing VMM did not fail the mounted claim") + } + sock, _, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + release() + } + if !errors.Is(err, ErrFailed) || sock != "" { + t.Fatalf("failed claim exposed data plane: socket=%q err=%v", sock, err) + } + }) + } +} + +func TestManagedSleepOptInPreservesRunningEgress(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(map[bool]string{false: "disabled", true: "enabled"}[enabled], func(t *testing.T) { + cfg := managedConfig(t) + cfg.Networks = []string{"test-network"} + if enabled { + cfg.ManagedVolumeHibernate = &config.ManagedVolumeHibernate{HookArgv: []string{"/hook"}, HookTimeoutSeconds: 1} + } + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + opts := managedOptions("data") + opts.Volumes[0].AttachOnly = false + sb, err := m.ClaimProvision(t.Context(), egKey, opts) + if err != nil { + t.Fatal(err) + } + sock, _, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + release() + } + if err != nil || sock != sb.VsockSocket { + t.Fatalf("running egress claim lost data plane: socket=%q err=%v", sock, err) + } + }) + } +} + +func TestManagedSleepRefusesEffectiveOutboundBeforeEffects(t *testing.T) { + for _, armed := range []bool{false, true} { + t.Run(map[bool]string{false: "unbound", true: "working proxy"}[armed], func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + editEgress(t, m, func(cfg *config.Config) { + cfg.Pools[0].Egress = &egress.Policy{Allow: []egress.Rule{{Host: "example.com"}}} + cfg.EgressClasses = []config.EgressClass{{Name: "desk", Egress: cfg.Pools[0].Egress}} + }) + sb.EgressClass = "desk" + sock := filepath.Join(outboundtest.SockRoot(t), "v") + sb.VsockSocket = sock + e.mu.Lock() + e.vms[sb.VMName] = sock + e.mu.Unlock() + if armed { + if err := m.out.ArmProxy(t.Context(), sb); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { m.out.Disarm(sb.ID, false) }) + requireProxyDoor(t, sock) + } + before := slices.Clone(e.volumeOps) + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(err, ErrVolumeCapture) { + t.Fatalf("effective outbound policy accepted for sleep: %v", err) + } + if armed { + requireProxyDoor(t, sock) + } + if len(e.hooks) != 0 || len(e.snapshots) != 0 || !slices.Equal(before, e.volumeOps) { + t.Fatal("outbound refusal had lifecycle effects") + } + got, _, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + release() + } + if err != nil || got != sock { + t.Fatalf("outbound refusal lost running guest: socket=%q err=%v", got, err) + } + }) + } +} + +func requireProxyDoor(t *testing.T, sock string) { + t.Helper() + c, err := net.DialTimeout("unix", engine.EgressSocketPath(sock), time.Second) + if err != nil { + t.Fatalf("outbound proxy unavailable: %v", err) + } + c.Close() +} + +type sleepUnmountGate struct { + *sleepTestEngine + entered, resume chan struct{} +} + +func (e *sleepUnmountGate) UnmountVolume(ctx context.Context, sock, mount string) error { + close(e.entered) + <-e.resume + return e.sleepTestEngine.UnmountVolume(ctx, sock, mount) +} + +func TestManagedSleepTeardownRefusesRenewal(t *testing.T) { + for _, expiry := range []bool{false, true} { + t.Run(map[bool]string{false: "release", true: "expiry"}[expiry], func(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + gate := &sleepUnmountGate{sleepTestEngine: e, entered: make(chan struct{}), resume: make(chan struct{})} + m.eng = gate + done := make(chan error, 1) + if expiry { + m.mu.Lock() + sb.Deadline = time.Now().Add(-time.Minute) + m.mu.Unlock() + m.reapOnce(t.Context()) + } else { + go func() { done <- m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}) }() + } + select { + case <-gate.entered: + case <-time.After(time.Second): + close(gate.resume) + t.Fatal("release did not reach teardown") + } + _, renewErr := m.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, time.Hour, "") + rows, loadErr := newClaimStore(cfg.DataDir, true).load() + heldDuringTeardown := volumeHoldersOf(m, "data") != (volumeHolders{}) + close(gate.resume) + if !expiry { + if err := <-done; err != nil { + t.Fatal(err) + } + } + waitFor(t, func() bool { return volumeHoldersOf(m, "data") == (volumeHolders{}) }) + if !errors.Is(renewErr, ErrUnknownSandbox) { + t.Errorf("teardown accepted renewal it discarded: %v", renewErr) + } + if loadErr != nil || rows[sb.ID] != nil { + t.Errorf("claim still durable during guest teardown: rows=%v err=%v", rows, loadErr) + } + if !heldDuringTeardown { + t.Fatal("backing ownership freed before guest teardown") + } + }) + } +} + +func TestManagedSleepReleasePersistenceFailureKeepsGuestAndHold(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + before := slices.Clone(e.volumeOps) + if err := os.Mkdir(m.store.path+".tmp", 0o700); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("failed durable deletion reported success") + } + if err := os.Remove(m.store.path + ".tmp"); err != nil { + t.Fatal(err) + } + if _, ok := m.Sandbox(sb.ID); !ok || !slices.Equal(before, e.volumeOps) { + t.Fatal("failed durable deletion dropped claim or touched guest") + } + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 409) + m.managedSleepHook = nil + sock, _, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + release() + } + if err != nil || sock == "" { + t.Fatalf("failed pre-rename deletion must leave the active guest available: socket=%q err=%v", sock, err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } +} + +func TestManagedSleepAllowsIneffectiveOutboundPolicy(t *testing.T) { + for _, noEgress := range []bool{false, true} { + t.Run(map[bool]string{false: "missing tenant policy", true: "no egress claim"}[noEgress], func(t *testing.T) { + m, _, sb, _ := newSleepTest(t) + editEgress(t, m, func(cfg *config.Config) { + cfg.Pools[0].Egress = &egress.Policy{Allow: []egress.Rule{{Host: "example.com"}}} + if noEgress { + cfg.EgressClasses = []config.EgressClass{{Name: "desk", Egress: cfg.Pools[0].Egress}} + } + }) + sb.EgressClass, sb.NoEgress = "desk", noEgress + requireSleep(t, m, sb) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + }) + } +} diff --git a/sandboxd/pool/managed_sleep_semantics_test.go b/sandboxd/pool/managed_sleep_semantics_test.go new file mode 100644 index 00000000..233e10b4 --- /dev/null +++ b/sandboxd/pool/managed_sleep_semantics_test.go @@ -0,0 +1,244 @@ +package pool + +import ( + "context" + "errors" + "os" + "slices" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/egress" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type missingHookEngine struct{ *sleepTestEngine } + +func (e *missingHookEngine) RunVolumeHook(ctx context.Context, sock string, call engine.VolumeHookCall) error { + _ = e.sleepTestEngine.RunVolumeHook(ctx, sock, call) + return &engine.VolumeHookExit{Code: 127, Output: "/usr/local/bin/volume-hook: not found"} +} + +func (e *sleepTestEngine) hookCalls(action string) int { + e.sleepMu.Lock() + defer e.sleepMu.Unlock() + n := 0 + for _, call := range e.hooks { + if call.Action == action { + n++ + } + } + return n +} + +func requireExec(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + sock, _, release, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if release != nil { + release() + } + if err != nil || sock == "" { + t.Fatalf("exec refused: socket=%q err=%v", sock, err) + } +} + +func TestManagedHibernatePrepareFailureKeepsClaimRunning(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + m.eng = &missingHookEngine{sleepTestEngine: e} + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("hibernate with a missing guest hook reported success") + } + if len(e.hibernates) != 0 { + t.Fatalf("captured after a failed prepare hook: %v", e.hibernates) + } + resumes := e.hookCalls("resume") + requireExec(t, m, sb) + if got := e.hookCalls("resume"); got != resumes { + t.Fatalf("exec after a failed hibernate re-ran the resume hook (%d -> %d)", resumes, got) + } + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("second hibernate with a missing guest hook reported success") + } + if got := e.hookCalls("prepare"); got != 2 { + t.Fatalf("prepare ran %d times over two hibernates, want 2", got) + } + requireExec(t, m, sb) +} + +func TestManagedWakeSurvivesFailedResumeHook(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.hookErr = "resume" + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("a failed resume hook after a verified mount failed the wake: %v", err) + } + requireExec(t, m, sb) +} + +func TestManagedWakeIgnoresOutboundPolicyChange(t *testing.T) { + m, _, sb, cfg := newSleepTest(t) + sb.EgressClass = "desk" + requireSleep(t, m, sb) + next := *cfg + next.Pools = slices.Clone(cfg.Pools) + next.Pools[0].Egress = &egress.Policy{Allow: []egress.Rule{{Host: "example.com"}}} + next.EgressClasses = []config.EgressClass{{Name: "desk", Egress: next.Pools[0].Egress}} + if _, err := m.ReloadConfig(t.Context(), &next); err != nil { + t.Fatal(err) + } + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("an outbound policy published while asleep stranded the claim: %v", err) + } + requireExec(t, m, sb) +} + +// failedRestoreOnce models cocoon quarantining a partially populated restore as state "error"; a later restore from "error" succeeds. +type failedRestoreOnce struct { + *sleepTestEngine + failed, errState bool +} + +func (e *failedRestoreOnce) Restore(ctx context.Context, name, snap string) (string, error) { + if !e.failed { + e.failed, e.errState = true, true + e.mu.Lock() + e.restores = append(e.restores, name) + e.mu.Unlock() + e.sleepMu.Lock() + e.generation++ + e.reason = "error" + e.sleepMu.Unlock() + return "", errors.New("restore: populate failed") + } + e.errState = false + return e.sleepTestEngine.Restore(ctx, name, snap) +} + +func (e *failedRestoreOnce) Inspect(ctx context.Context, name string) (types.VMRecord, bool, error) { + rec, found, err := e.sleepTestEngine.Inspect(ctx, name) + if e.errState { + rec.State = "error" + } + return rec, found, err +} + +func TestManagedWakeRetriesFailedRestore(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + m.eng = &failedRestoreOnce{sleepTestEngine: e} + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("a failed restore reported success") + } + if _, err := m.AgentSocket(sb.ID, sb.Token); !errors.Is(err, ErrPaused) { + t.Fatalf("a failed restore did not leave the claim hibernated: %v", err) + } + requireExec(t, m, sb) + if len(e.restores) != 2 { + t.Fatalf("restores = %v, want the failed attempt and one retry", e.restores) + } +} + +// generationGapEngine advances cocoon's transition generation by more than one per hibernate and restore. +type generationGapEngine struct{ *sleepTestEngine } + +func (e *generationGapEngine) Hibernate(ctx context.Context, name, snap string) error { + err := e.sleepTestEngine.Hibernate(ctx, name, snap) + e.sleepMu.Lock() + e.generation += 3 + e.sleepMu.Unlock() + return err +} + +func (e *generationGapEngine) Restore(ctx context.Context, name, snap string) (string, error) { + sock, err := e.sleepTestEngine.Restore(ctx, name, snap) + e.sleepMu.Lock() + e.generation += 5 + e.sleepMu.Unlock() + return sock, err +} + +func TestManagedSleepToleratesGenerationGaps(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + m.eng = &generationGapEngine{sleepTestEngine: e} + requireSleep(t, m, sb) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("wake refused a restored VM whose generation skipped values: %v", err) + } + requireExec(t, m, sb) +} + +type liveVolumeFloorEngine struct{ *sleepTestEngine } + +func (e *liveVolumeFloorEngine) CheckLiveVolumeSupport(context.Context) error { + return &engine.LiveVolumeUnsupported{Version: "v0.7.1"} +} + +func TestManagedWakeRefusesBelowLiveVolumeFloor(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + m.eng = &liveVolumeFloorEngine{sleepTestEngine: e} + err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}) + if _, ok := errors.AsType[*engine.LiveVolumeUnsupported](err); !ok || len(e.restores) != 0 { + t.Fatalf("wake below the live volume floor: err=%v restores=%d", err, len(e.restores)) + } +} + +func TestManagedWakeRecordsWakeLatency(t *testing.T) { + m, _, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if c := m.Counters(); c.Wakes != 1 || c.WakeNanos == 0 { + t.Fatalf("managed wake counted wakes=%d wake_nanos=%d", c.Wakes, c.WakeNanos) + } +} + +func TestManagedSleepHookFailureReportsTheHook(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + e.hookErr = "prepare" + err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) + ve := requireVolumeStatus(t, err, 409) + if ve.Code != "volume_hook_failed" || !strings.Contains(ve.Message, "prepare") || !strings.Contains(ve.Message, "hook failed") { + t.Fatalf("failed prepare hook = %+v, want 409 volume_hook_failed naming the action and the hook output", ve) + } +} + +type hookTransportFailEngine struct{ *sleepTestEngine } + +func (e *hookTransportFailEngine) RunVolumeHook(context.Context, string, engine.VolumeHookCall) error { + return errors.New("dial unix /var/lib/cocoon/run/cloudhypervisor/abc/vsock.sock: connect: connection refused") +} + +func TestManagedSleepHookTransportFailureStaysPrivate(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + m.eng = &hookTransportFailEngine{sleepTestEngine: e} + err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) + if err == nil { + t.Fatal("hibernate over an unreachable guest hook reported success") + } + if v, ok := errors.AsType[*VolumeError](err); ok && (v.Code == "volume_hook_failed" || strings.Contains(v.Message, "/var/lib/cocoon")) { + t.Fatalf("host transport failure reached the tenant as %d %s %q", v.Status, v.Code, v.Message) + } +} + +func TestManagedWakeOverLostBackingNeedsRecovery(t *testing.T) { + m, _, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + path, _ := m.managedPath("data") + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + ve := requireVolumeStatus(t, m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}), 503) + if ve.Code != "volume_recovery_required" { + t.Fatalf("wake over a lost backing = %+v, want volume_recovery_required, which carries no Retry-After", ve) + } + m.mu.Lock() + got := m.claimed[sb.ID] + m.mu.Unlock() + if got == nil || got.VolumeSleep == nil { + t.Fatal("a refused wake must leave the claim hibernated") + } +} diff --git a/sandboxd/pool/managed_sleep_test.go b/sandboxd/pool/managed_sleep_test.go new file mode 100644 index 00000000..26327102 --- /dev/null +++ b/sandboxd/pool/managed_sleep_test.go @@ -0,0 +1,981 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "regexp" + "slices" + "sync" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type sleepTestEngine struct { + *managedTestEngine + sleepMu sync.Mutex + generation uint64 + reason string + vmID string + sourceID string + hooks []engine.VolumeHookCall + hookErr string + hookEntered, hookContinue chan struct{} + restoreAfterErr bool + replaceSourceOnRestore bool + snapshotRemoveErr error + snapshotRemoveAfterEffect bool + snapshotRemoveMutation string + mounted bool +} + +func newSleepTest(t *testing.T) (*Manager, *sleepTestEngine, *types.Sandbox, *config.Config) { + t.Helper() + cfg := managedConfig(t) + cfg.ManagedVolumeHibernate = &config.ManagedVolumeHibernate{HookArgv: []string{"/usr/local/bin/volume-hook"}, HookTimeoutSeconds: 1} + e := &sleepTestEngine{managedTestEngine: newManagedEngine(), generation: 2, reason: "boot", vmID: "immutable-vm", mounted: true} + m, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = m.Close() }) + if err = m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + createManagedTest(t, m, "data") + opts := managedOptions("data") + opts.Volumes[0].AttachOnly = false + sb, err := m.ClaimProvision(t.Context(), testKey, opts) + if err != nil { + t.Fatal(err) + } + return m, e, sb, cfg +} + +func (e *sleepTestEngine) Inspect(ctx context.Context, name string) (types.VMRecord, bool, error) { + rec, found, err := e.managedTestEngine.Inspect(ctx, name) + e.sleepMu.Lock() + defer e.sleepMu.Unlock() + rec.ID, rec.TransitionGeneration, rec.LastTransitionReason = e.vmID, e.generation, e.reason + return rec, found, err +} + +func (e *sleepTestEngine) DiskAttach(ctx context.Context, name string, spec engine.VolumeSpec) error { + if err := e.managedTestEngine.DiskAttach(ctx, name, spec); err != nil { + return err + } + e.mu.Lock() + if e.disks == nil { + e.disks = map[string][]types.VMDisk{} + } + e.disks[name] = []types.VMDisk{{Name: spec.Name, Path: spec.Path, ReadOnly: !spec.RW}} + e.mu.Unlock() + return nil +} + +func (e *sleepTestEngine) DetachVolume(ctx context.Context, name, sock, serial, mount string) error { + if err := e.fakeEngine.DetachVolume(ctx, name, sock, serial, mount); err != nil { + return err + } + e.mu.Lock() + e.disks[name] = nil + e.mu.Unlock() + e.sleepMu.Lock() + e.mounted = false + e.sleepMu.Unlock() + return nil +} + +func (e *sleepTestEngine) MountLiveVolume(ctx context.Context, sock, name, mount string, rw bool) error { + if err := e.fakeEngine.MountLiveVolume(ctx, sock, name, mount, rw); err != nil { + return err + } + e.sleepMu.Lock() + e.mounted = true + e.sleepMu.Unlock() + return nil +} + +func (e *sleepTestEngine) VerifyMountedVolume(context.Context, string, string, string) error { + e.sleepMu.Lock() + defer e.sleepMu.Unlock() + if !e.mounted { + return errors.New("mount absent") + } + return nil +} + +func (e *sleepTestEngine) Hibernate(ctx context.Context, name, snap string) error { + err := e.fakeEngine.Hibernate(ctx, name, snap) + e.mu.Lock() + stopped := e.stopped[name] + e.mu.Unlock() + if stopped { + e.sleepMu.Lock() + e.generation++ + e.reason = "stop-user" + e.sleepMu.Unlock() + } + return err +} + +func (e *sleepTestEngine) Restore(ctx context.Context, name, snap string) (string, error) { + sock, err := e.fakeEngine.Restore(ctx, name, snap) + if err != nil { + return sock, err + } + e.mu.Lock() + e.pids[name]++ + e.mu.Unlock() + e.sleepMu.Lock() + e.generation++ + e.reason = "restore" + if e.replaceSourceOnRestore { + e.sourceID = "replacement-source" + } + postErr := e.restoreAfterErr + e.sleepMu.Unlock() + if postErr { + return "", errors.New("restore reply lost") + } + return sock, nil +} + +func (e *sleepTestEngine) InspectHibernateSource(ctx context.Context, name string) (engine.HibernateSource, bool, error) { + source, found, err := e.fakeEngine.InspectHibernateSource(ctx, name) + e.sleepMu.Lock() + defer e.sleepMu.Unlock() + if found && e.sourceID != "" { + source.ID = e.sourceID + } + return source, found, err +} + +func (e *sleepTestEngine) SnapshotRemove(ctx context.Context, name string) error { + e.sleepMu.Lock() + err := e.snapshotRemoveErr + afterEffect := e.snapshotRemoveAfterEffect + mutation := e.snapshotRemoveMutation + e.sleepMu.Unlock() + if err != nil && !afterEffect { + return err + } + if removeErr := e.fakeEngine.SnapshotRemove(ctx, name); removeErr != nil { + return removeErr + } + switch mutation { + case "vm id": + e.sleepMu.Lock() + e.vmID = "replacement-vm" + e.sleepMu.Unlock() + case "transition": + e.sleepMu.Lock() + e.generation++ + e.reason = "boot" + e.sleepMu.Unlock() + } + return err +} + +func (e *sleepTestEngine) RunVolumeHook(_ context.Context, _ string, call engine.VolumeHookCall) error { + e.sleepMu.Lock() + e.hooks = append(e.hooks, call) + fail := e.hookErr == call.Action + entered, resume := e.hookEntered, e.hookContinue + e.sleepMu.Unlock() + if entered != nil { + entered <- struct{}{} + <-resume + } + if fail { + return &engine.VolumeHookExit{Code: 1, Output: "hook failed"} + } + return nil +} + +func requireSleep(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if info, ok := m.Sandbox(sb.ID); !ok || !info.Hibernated { + t.Fatalf("not hibernated: %+v", info) + } + if _, err := m.AgentSocket(sb.ID, sb.Token); !errors.Is(err, ErrPaused) { + t.Fatalf("sleeping claim served its passive socket: %v", err) + } +} + +// requireRunning checks the passive socket, which refuses while any sleep step is journaled. +func requireRunning(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + if _, err := m.AgentSocket(sb.ID, sb.Token); err != nil { + t.Fatalf("claim is not running: %v", err) + } +} + +func managedBackingPath(t *testing.T, m *Manager, name string) string { + t.Helper() + info, err := m.GetVolume(name, "") + if err != nil { + t.Fatal(err) + } + return m.managed.path(volumeID(info.ID)) +} + +// journalSleepPhase rewrites the claim row as a crash at phase would have left it. +func journalSleepPhase(t *testing.T, m *Manager, sb *types.Sandbox, edit func(*types.VolumeSleep)) { + t.Helper() + m.mu.Lock() + next := *sb.VolumeSleep + edit(&next) + sb.VolumeSleep = &next + projectSleepState(sb, &next) + js := m.store.set(sb) + m.mu.Unlock() + if err := m.store.commit(js); err != nil { + t.Fatal(err) + } +} + +func TestManagedSleepRoundTripAndExclusiveHold(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 409) + if err = m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + requireRunning(t, m, sb) + requireExec(t, m, sb) + if len(e.restores) != 1 || len(e.hooks) != 2 || e.hooks[0].Operation != e.hooks[1].Operation { + t.Fatalf("effects restores=%v hooks=%v", e.restores, e.hooks) + } +} + +func TestManagedSleepSnapshotNameFitsEngineContract(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.hibernates) != 1 { + t.Fatalf("hibernate calls = %v, want one", e.hibernates) + } + snapshot := e.hibernates[0] + if len(snapshot) > 63 { + t.Fatalf("snapshot name %q has length %d, exceeds Cocoon's 63-character limit", snapshot, len(snapshot)) + } + if !regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$`).MatchString(snapshot) { + t.Fatalf("snapshot name %q violates Cocoon's name syntax", snapshot) + } + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } +} + +func TestManagedSleepAbortKeepsOriginalVM(t *testing.T) { + for _, failure := range []string{"prepare", "unmount", "detach", "capture"} { + t.Run(failure, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + switch failure { + case "prepare": + e.hookErr = "prepare" + case "unmount": + e.unmountVolumeErr = errors.New("busy") + case "detach": + e.detachVolumeErr = errors.New("detach failed") + case "capture": + e.hibernateErr = errors.New("capture refused") + } + err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) + if err == nil { + t.Fatal("failure hidden") + } + if v, ok := errors.AsType[*VolumeError](err); ok && v.Code != "volume_hook_failed" { + t.Fatalf("a %s failure with a clean resume reported %s, want the step's own error", failure, v.Code) + } + requireRunning(t, m, sb) + if e.stopped[sb.VMName] || len(e.restores) != 0 { + t.Fatal("failed preparation damaged VM") + } + if failure != "capture" && len(e.hibernates) != 0 { + t.Fatal("captured after failed preparation") + } + if e.hooks[len(e.hooks)-1].Action != "resume" { + t.Fatal("writers not resumed") + } + if len(e.snapRemoves) != 0 { + t.Fatalf("abort removed nonexistent source: %v", e.snapRemoves) + } + }) + } +} + +func TestManagedSleepMountConflictReportsOnlyTheConflict(t *testing.T) { + for _, resumeFails := range []bool{false, true} { + t.Run(map[bool]string{false: "resumed", true: "resume failed"}[resumeFails], func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + e.detachVolumeErr = &engine.VolumeMountConflict{Message: `cannot unmount "/volumes/data"; close guest references and retry`} + if resumeFails { + e.hookErr = "resume" + } + err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) + if _, conflict := errors.AsType[*engine.VolumeMountConflict](err); !conflict { + t.Fatalf("hibernate error %v lost the guest mount conflict", err) + } + if v, ok := errors.AsType[*VolumeError](err); ok { + t.Fatalf("abort over a verified mount reported %s, want only the guest mount conflict", v.Code) + } + requireExec(t, m, sb) + }) + } +} + +func TestManagedSleepWakeFailuresKeepRestoredProcess(t *testing.T) { + for _, failure := range []string{"restore reply", "attach", "mount"} { + t.Run(failure, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + switch failure { + case "restore reply": + e.restoreAfterErr = true + case "attach": + e.diskAttachErr = errors.New("attach unavailable") + case "mount": + e.mountVolumeErr = errors.New("mount failed") + } + err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}) + if failure != "restore reply" && err == nil { + t.Fatal("wake failure hidden") + } + if e.stopped[sb.VMName] || len(e.restores) != 1 { + t.Fatal("restored VM was discarded") + } + e.diskAttachErr, e.mountVolumeErr = nil, nil + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 1 { + t.Fatal("restore replayed over live VM") + } + }) + } +} + +func TestManagedSleepWakeVerifiesBackingFirst(t *testing.T) { + for _, replace := range []bool{false, true} { + t.Run(map[bool]string{false: "missing", true: "replaced"}[replace], func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + path := managedBackingPath(t, m, "data") + saved := filepath.Join(filepath.Dir(filepath.Dir(path)), "saved.raw") + if err := os.Rename(path, saved); err != nil { + t.Fatal(err) + } + if replace { + if err := os.WriteFile(path, nil, 0o600); err != nil { + t.Fatal(err) + } + } + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil || len(e.restores) != 0 { + t.Fatal("restore ran without exact backing") + } + info, err := m.GetVolume("data", "t1") + if err != nil || info.State != string(volumeRecovery) || info.Available { + t.Fatalf("missing backing detail: %+v %v", info, err) + } + infos := m.Volumes("t1", nil) + if len(infos) != 1 || infos[0].Available { + t.Fatalf("missing backing list: %+v", infos) + } + _, _, err = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + if v := requireVolumeStatus(t, err, 503); v.Code != "volume_storage_unavailable" { + t.Fatalf("create replay over a sleeping volume's lost backing = %s, want volume_storage_unavailable while wake owns it", v.Code) + } + _, err = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if v := requireVolumeStatus(t, err, 503); v.Code != "volume_storage_unavailable" { + t.Fatalf("claim of a sleeping volume's lost backing = %s, want volume_storage_unavailable while wake owns it", v.Code) + } + if replace { + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + } + if err := os.Rename(saved, path); err != nil { + t.Fatal(err) + } + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + }) + } +} + +func TestManagedSleepProvenanceRejectsReplacement(t *testing.T) { + for _, mismatch := range []string{"id", "transition", "disk path", "disk name"} { + t.Run(mismatch, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.mountVolumeErr = errors.New("mount failed") + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("expected suspended wake") + } + switch mismatch { + case "id": + e.vmID = "replacement" + case "transition": + e.generation++ + e.reason = "boot" + case "disk path": + e.disks[sb.VMName][0].Path = "/wrong" + case "disk name": + e.disks[sb.VMName][0].Name = "other" + } + e.mountVolumeErr = nil + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("adopted replacement") + } + if len(e.restores) != 1 { + t.Fatal("replayed restore") + } + }) + } +} + +func TestManagedSleepReleasePreservesBacking(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + path := managedBackingPath(t, m, "data") + before := slices.Clone(e.volumeOps) + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(path); err != nil { + t.Fatal(err) + } + if len(e.snapshots) != 0 { + t.Fatal("sleep snapshot leaked") + } + if !slices.Equal(before, e.volumeOps) { + t.Fatal("asleep release touched guest") + } + requireManagedClaimable(t, m, "data") +} + +func TestManagedSleepRestartEveryPhase(t *testing.T) { + for _, phase := range []types.SleepPhase{types.SleepPreparing, types.SleepAborting, types.SleepCapturing, types.SleepSleeping, types.SleepWaking, types.SleepResuming} { + t.Run(string(phase), func(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + original, err := m.GetVolume("data", "t1") + if err != nil { + t.Fatal(err) + } + requireSleep(t, m, sb) + path := managedBackingPath(t, m, "data") + if phase == types.SleepPreparing || phase == types.SleepAborting || phase == types.SleepCapturing { + e.mu.Lock() + e.stopped[sb.VMName] = false + if phase != types.SleepCapturing { + e.disks[sb.VMName] = []types.VMDisk{{Name: "data", Path: path}} + } + e.snapshots = nil + e.mu.Unlock() + e.sleepMu.Lock() + e.generation = 2 + e.reason = "boot" + e.mounted = phase != types.SleepCapturing + e.sleepMu.Unlock() + } + if phase == types.SleepResuming { + if _, err = e.Restore(t.Context(), sb.VMName, sb.VolumeSleep.Snapshot); err != nil { + t.Fatal(err) + } + } + journalSleepPhase(t, m, sb, func(s *types.VolumeSleep) { + s.Phase = phase + if phase == types.SleepPreparing || phase == types.SleepAborting || phase == types.SleepCapturing { + s.SourceID = "" + } + }) + if err = m.Close(); err != nil { + t.Fatal(err) + } + cfg.ManagedVolumeHibernate = nil + recovered := reopenSleepTest(t, cfg, e) + if _, err = recovered.AgentSocket(sb.ID, sb.Token); !errors.Is(err, ErrPaused) { + t.Fatalf("restart bypassed journal: %v", err) + } + _, err = recovered.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 409) + if err = recovered.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + requireRunning(t, recovered, sb) + if got, err := recovered.GetVolume("data", "t1"); err != nil || got.ID != original.ID { + t.Fatalf("recovery changed backing: %+v %v", got, err) + } + if !slices.Equal(e.hooks[len(e.hooks)-1].Argv, []string{"/usr/local/bin/volume-hook"}) { + t.Fatal("recovery lost pinned hook") + } + }) + } +} + +func TestManagedSleepPostCaptureErrorAdoptsAndPersists(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + e.hibernateErr = errors.New("reply lost") + e.hibernateErrCompletes = true + requireSleep(t, m, sb) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } +} + +// claimsFaultEngine makes claims.json unwritable when the named engine call starts. +type claimsFaultEngine struct { + *sleepTestEngine + on string + fault func() +} + +func (e *claimsFaultEngine) trip(call string) { + if e.on == call { + e.on = "" + e.fault() + } +} + +func (e *claimsFaultEngine) VerifyMountedVolume(ctx context.Context, sock, name, mount string) error { + e.trip("verify") + return e.sleepTestEngine.VerifyMountedVolume(ctx, sock, name, mount) +} + +func (e *claimsFaultEngine) DetachVolume(ctx context.Context, name, sock, serial, mount string) error { + e.trip("detach") + return e.sleepTestEngine.DetachVolume(ctx, name, sock, serial, mount) +} + +func (e *claimsFaultEngine) Hibernate(ctx context.Context, name, snap string) error { + e.trip("capture") + return e.sleepTestEngine.Hibernate(ctx, name, snap) +} + +func (e *claimsFaultEngine) MountLiveVolume(ctx context.Context, sock, name, mount string, rw bool) error { + e.trip("mount") + return e.sleepTestEngine.MountLiveVolume(ctx, sock, name, mount, rw) +} + +func TestManagedSleepJournalFailureStopsEffects(t *testing.T) { + for _, tc := range []struct { + phase types.SleepPhase + on string + }{{types.SleepPreparing, "verify"}, {types.SleepCapturing, "detach"}, {types.SleepSleeping, "capture"}, {types.SleepWaking, "now"}, {types.SleepResuming, "mount"}} { + t.Run(string(tc.phase), func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + wake := tc.phase == types.SleepWaking || tc.phase == types.SleepResuming + if wake { + requireSleep(t, m, sb) + } + tmp := m.store.path + ".tmp" + fault := func() { _ = os.Mkdir(tmp, 0o700) } + t.Cleanup(func() { + _ = os.Remove(tmp) + waitFor(t, func() bool { return !m.store.recommitting.Load() }) + }) + if tc.on == "now" { + fault() + } + m.eng = &claimsFaultEngine{sleepTestEngine: e, on: tc.on, fault: fault} + var err error + if wake { + err = m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}) + } else { + err = m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) + } + if err == nil { + t.Fatal("persistence failure hidden") + } + beforeHooks, beforeRestores, beforeCaptures := len(e.hooks), len(e.restores), len(e.hibernates) + if tc.phase != types.SleepPreparing { + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("unpersisted intent served guest") + } + } + if len(e.hooks) != beforeHooks || len(e.restores) != beforeRestores || len(e.hibernates) != beforeCaptures { + t.Fatal("effect after an unpersisted intent") + } + switch tc.phase { + case types.SleepPreparing: + if len(e.hooks) != 0 { + t.Fatal("hook preceded durable intent") + } + requireRunning(t, m, sb) + case types.SleepCapturing: + if len(e.hibernates) != 0 { + t.Fatal("capture preceded durable intent") + } + case types.SleepWaking: + if len(e.restores) != 0 { + t.Fatal("restore preceded durable intent") + } + case types.SleepResuming: + if e.hooks[len(e.hooks)-1].Action == "resume" { + t.Fatal("resume preceded durable intent") + } + } + }) + } +} + +func TestManagedSleepUnknownSourceAndDeadRestoredRefuse(t *testing.T) { + for _, failure := range []string{"missing source", "source identity", "dead restored", "unknown phase"} { + t.Run(failure, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + switch failure { + case "missing source": + e.snapshots = nil + case "source identity": + journalSleepPhase(t, m, sb, func(s *types.VolumeSleep) { s.SourceID = "other" }) + case "dead restored": + e.mountVolumeErr = errors.New("mount failed") + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("expected failed wake") + } + e.mountVolumeErr = nil + e.stopped[sb.VMName] = true + case "unknown phase": + journalSleepPhase(t, m, sb, func(s *types.VolumeSleep) { s.Phase = "unknown" }) + } + restores := len(e.restores) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("uncertain source served guest") + } + if len(e.restores) != restores { + t.Fatal("unsafe memory replay") + } + }) + } +} + +func TestManagedSleepDeadBoundVMIsReleaseOnly(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + path := managedBackingPath(t, m, "data") + e.mountVolumeErr = errors.New("mount failed") + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("mount failure hidden") + } + e.mountVolumeErr = nil + if len(e.snapshots) != 0 { + t.Fatalf("wake did not consume its source before attaching: snapshots=%v", e.snapshots) + } + e.mu.Lock() + e.stopped[sb.VMName] = true + e.mu.Unlock() + restores := len(e.restores) + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("dead bound VM resumed") + } + if len(e.restores) != restores { + t.Fatal("dead bound VM replayed restore") + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(path); err != nil { + t.Fatal(err) + } + requireManagedClaimable(t, m, "data") +} + +func TestManagedSleepExpiryIgnoresArchivePool(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + m.mu.Lock() + sb.Deadline = time.Now().Add(-time.Minute) + m.pools[sb.Key].archiveAfter = time.Second + m.mu.Unlock() + m.reapOnce(t.Context()) + waitFor(t, func() bool { _, live := m.Sandbox(sb.ID); return !live && !e.vmAlive(sb.VMName) }) + waitFor(t, func() bool { + e.mu.Lock() + defer e.mu.Unlock() + return len(e.snapshots) == 0 + }) + requireManagedClaimable(t, m, "data") +} + +func TestManagedSleepUnsupportedPreflightHasNoEffects(t *testing.T) { + for _, bad := range []string{"disabled", "desired mismatch", "extra disk", "missing mount", "egress"} { + t.Run(bad, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + switch bad { + case "disabled": + m.managedSleepHook = nil + case "desired mismatch": + sb.VolumesWant = nil + case "extra disk": + e.disks[sb.VMName] = append(e.disks[sb.VMName], types.VMDisk{Name: "extra", Path: "/extra"}) + case "missing mount": + e.mounted = false + case "egress": + sb.Key.Net = types.NetEgress + } + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("unsupported sleep accepted") + } + if len(e.hooks) != 0 || len(e.hibernates) != 0 || slices.Contains(e.volumeOps, "detach:data") { + t.Fatal("unsupported preflight changed guest") + } + }) + } +} + +func TestManagedSleepRestoreBeforeEffectRetriesSameSource(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.restoreErr = errors.New("restore preflight refused") + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("restore failure hidden") + } + if !e.stopped[sb.VMName] { + t.Fatal("failed restore changed VM") + } + e.restoreErr = nil + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 2 { + t.Fatal("safe stopped restore was not retried") + } +} + +func TestManagedSleepWakeWaitsForSourceConsumption(t *testing.T) { + for _, tc := range []struct { + name string + err error + }{ + {"active restore lease", errors.New("snapshot is in use")}, + {"remove failure", errors.New("snapshot delete failed")}, + } { + t.Run(tc.name, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + snapshot := e.hibernates[0] + e.mu.Lock() + beforeOps := slices.Clone(e.volumeOps) + e.mu.Unlock() + e.sleepMu.Lock() + beforeHooks := len(e.hooks) + e.snapshotRemoveErr = tc.err + e.sleepMu.Unlock() + + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("wake passed unavailable source-consumption barrier") + } + if _, err := m.AgentSocket(sb.ID, sb.Token); !errors.Is(err, ErrPaused) { + t.Fatalf("failed barrier published guest: %v", err) + } + e.mu.Lock() + afterOps := slices.Clone(e.volumeOps) + sourcePresent := slices.Contains(e.snapshots, snapshot) + e.mu.Unlock() + e.sleepMu.Lock() + afterHooks := len(e.hooks) + e.snapshotRemoveErr = nil + e.sleepMu.Unlock() + if !slices.Equal(afterOps, beforeOps) || afterHooks != beforeHooks || !sourcePresent { + t.Fatalf("barrier failure reached guest: ops=%v hooks=%d source=%v", afterOps, afterHooks, sourcePresent) + } + generation := e.generation + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 1 || e.generation != generation { + t.Fatalf("barrier retry replayed restore: restores=%d generation=%d want=%d", len(e.restores), e.generation, generation) + } + }) + } +} + +func TestManagedSleepWakeAcceptsRemoveErrorAfterSourceGone(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.snapshotRemoveErr = errors.New("snapshot remove reply lost") + e.snapshotRemoveAfterEffect = true + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.snapRemoves) != 1 || len(e.snapshots) != 0 { + t.Fatalf("remove after-effect was not accepted: removes=%v snapshots=%v", e.snapRemoves, e.snapshots) + } +} + +func TestManagedSleepWakeRetriesAfterConsumedSource(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + requireSleep(t, m, sb) + e.diskAttachErr = errors.New("attach unavailable") + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("attach failure hidden") + } + if len(e.snapshots) != 0 || len(e.snapRemoves) != 1 { + t.Fatalf("source was not consumed before attach: removes=%v snapshots=%v", e.snapRemoves, e.snapshots) + } + generation := e.generation + if err := m.Close(); err != nil { + t.Fatal(err) + } + e.diskAttachErr = nil + recovered := reopenSleepTest(t, cfg, e) + if err := recovered.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 1 || len(e.snapRemoves) != 1 || e.generation != generation { + t.Fatalf("consumed-source retry repeated effect: restores=%d removes=%d generation=%d want=%d", len(e.restores), len(e.snapRemoves), e.generation, generation) + } +} + +func TestManagedSleepWakeRefusesReplacementSourceAfterRestore(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.replaceSourceOnRestore = true + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("replacement source crossed restore barrier") + } + if len(e.snapRemoves) != 0 { + t.Fatalf("replacement source was consumed: %v", e.snapRemoves) + } + e.mu.Lock() + defer e.mu.Unlock() + if len(e.disks[sb.VMName]) != 0 { + t.Fatalf("replacement source reached attach: %+v", e.disks[sb.VMName]) + } +} + +func TestManagedSleepWakeRevalidatesAfterSourceConsumption(t *testing.T) { + for _, mutation := range []string{"vm id", "transition"} { + t.Run(mutation, func(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + e.mu.Lock() + beforeOps := slices.Clone(e.volumeOps) + e.mu.Unlock() + e.snapshotRemoveMutation = mutation + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("changed restored VM crossed source barrier") + } + e.mu.Lock() + defer e.mu.Unlock() + if len(e.snapshots) != 0 || !slices.Equal(e.volumeOps, beforeOps) { + t.Fatalf("revalidation failure reached guest: before=%v after=%v snapshots=%v", beforeOps, e.volumeOps, e.snapshots) + } + }) + } +} + +func TestManagedSleepResumingWaitsForSourceConsumption(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + requireSleep(t, m, sb) + if _, err := e.Restore(t.Context(), sb.VMName, e.hibernates[0]); err != nil { + t.Fatal(err) + } + journalSleepPhase(t, m, sb, func(s *types.VolumeSleep) { s.Phase = types.SleepResuming }) + e.mu.Lock() + beforeOps := slices.Clone(e.volumeOps) + e.mu.Unlock() + e.sleepMu.Lock() + beforeHooks := len(e.hooks) + e.snapshotRemoveErr = errors.New("snapshot is in use") + e.sleepMu.Unlock() + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("resuming guest passed unavailable source-consumption barrier") + } + e.mu.Lock() + afterOps := slices.Clone(e.volumeOps) + e.mu.Unlock() + e.sleepMu.Lock() + afterHooks := len(e.hooks) + e.snapshotRemoveErr = nil + e.sleepMu.Unlock() + if !slices.Equal(afterOps, beforeOps) || afterHooks != beforeHooks { + t.Fatalf("resuming barrier failure reached guest: ops=%v hooks=%d", afterOps, afterHooks) + } + if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 1 { + t.Fatalf("resuming barrier retry replayed restore: %v", e.restores) + } +} + +func TestManagedSleepAdoptsRestoredProcessAcrossCrash(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + requireSleep(t, m, sb) + journalSleepPhase(t, m, sb, func(s *types.VolumeSleep) { s.Phase = types.SleepWaking }) + if _, err := e.Restore(t.Context(), sb.VMName, e.hibernates[0]); err != nil { + t.Fatal(err) + } + e.snapshots = nil + if err := m.Close(); err != nil { + t.Fatal(err) + } + recovered := reopenSleepTest(t, cfg, e) + if err := recovered.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.restores) != 1 { + t.Fatal("crashed wake replayed restore over live process") + } +} + +func TestManagedSleepCaptureCrashAdoptsStoppedSnapshot(t *testing.T) { + m, e, sb, cfg := newSleepTest(t) + tmp := m.store.path + ".tmp" + m.eng = &claimsFaultEngine{sleepTestEngine: e, on: "capture", fault: func() { _ = os.Mkdir(tmp, 0o700) }} + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("lost commit hidden") + } + if !e.stopped[sb.VMName] { + t.Fatal("capture did not take effect") + } + saved, err := os.ReadFile(m.store.path) + if err != nil { + t.Fatal(err) + } + if err = os.Remove(tmp); err != nil { + t.Fatal(err) + } + waitFor(t, func() bool { return !m.store.recommitting.Load() && m.store.synced() }) + if err = m.Close(); err != nil { + t.Fatal(err) + } + if err = os.WriteFile(m.store.path, saved, 0o600); err != nil { + t.Fatal(err) + } + recovered := reopenSleepTest(t, cfg, e) + if err = recovered.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatal(err) + } + if len(e.hibernates) != 1 || len(e.restores) != 1 { + t.Fatal("recovery repeated capture") + } +} + +func TestManagedSleepInitialClaimBarrierPrecedesIntent(t *testing.T) { + m, e, sb, _ := newSleepTest(t) + m.store.path = filepath.Join(t.TempDir(), "missing", "claims.json") + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("claim barrier failed silently") + } + requireRunning(t, m, sb) + if len(e.hooks) != 0 || len(e.hibernates) != 0 { + t.Fatal("effect preceded durable credentials") + } +} diff --git a/sandboxd/pool/managed_sleep_vm.go b/sandboxd/pool/managed_sleep_vm.go new file mode 100644 index 00000000..7bd9c9b9 --- /dev/null +++ b/sandboxd/pool/managed_sleep_vm.go @@ -0,0 +1,149 @@ +package pool + +import ( + "context" + "errors" + "fmt" + "net/http" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +const ( + vmReasonStopUser = "stop-user" + vmReasonRestore = "restore" +) + +func exactSleepDisk(rec types.VMRecord, vol resolvedVolume) bool { + return len(rec.AttachedDevices.Disks) == 1 && rec.AttachedDevices.Disks[0].Name == vol.applied.Name && rec.AttachedDevices.Disks[0].Path == vol.disk.Path && !rec.AttachedDevices.Disks[0].ReadOnly +} + +// sleepVM returns the running VM the sleep record names: the original before capture, the restored one after. +func (m *Manager) sleepVM(ctx context.Context, sb *types.Sandbox, restored bool) (types.VMRecord, error) { + rec, found, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return rec, err + } + s := sb.VolumeSleep + current := rec.TransitionGeneration == s.Generation + if restored { + current = rec.TransitionGeneration > s.Generation && rec.LastTransitionReason == vmReasonRestore + } + if !found || rec.Config.Name != sb.VMName || rec.ID != s.VMID || rec.State != vmStateRunning || rec.LastTransitionReason == "" || !current { + return rec, sleepRecoveryError() + } + return rec, nil +} + +func (m *Manager) sleepHook(ctx context.Context, sb *types.Sandbox, vol resolvedVolume, sock, action string) error { + s := sb.VolumeSleep + err := m.eng.RunVolumeHook(ctx, sock, engine.VolumeHookCall{Argv: s.Hook.HookArgv, Action: action, Operation: s.Operation, VolumeID: string(vol.managedID), VolumeName: vol.applied.Name, Mount: vol.applied.Mount, Timeout: types.Seconds(s.Hook.HookTimeoutSeconds)}) + // only the guest's own exit is the tenant's to see; a transport error names host paths + if _, exited := errors.AsType[*engine.VolumeHookExit](err); !exited { + return err + } + return volumeError(http.StatusConflict, "volume_hook_failed", fmt.Sprintf("guest %s hook failed: %.512s", action, err.Error())) +} + +func (m *Manager) restoreManagedSleep(ctx context.Context, sb *types.Sandbox) (types.VMRecord, error) { + s := sb.VolumeSleep + rec, found, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return rec, err + } + if !found || rec.ID != s.VMID || rec.Config.Name != sb.VMName { + return rec, sleepRecoveryError() + } + if rec.State == vmStateRunning { + return m.sleepVM(ctx, sb, true) + } + // a stopped origin or a restore that failed before any guest ran; the intact source guards the retry + if rec.TransitionGeneration <= s.Generation { + return rec, sleepRecoveryError() + } + source, exists, err := m.eng.InspectHibernateSource(ctx, s.Snapshot) + if err != nil { + return rec, err + } + if !exists || source.ID != s.SourceID { + return rec, sleepRecoveryError() + } + if err = m.diskErr(); err != nil { + return rec, err + } + if err = m.eng.CheckLiveVolumeSupport(ctx); err != nil { + return rec, err + } + _, effectErr := m.eng.Restore(ctx, sb.VMName, s.Snapshot) + rec, err = m.sleepVM(ctx, sb, true) + if err != nil { + return rec, errors.Join(effectErr, err) + } + return rec, nil +} + +// consumeManagedSleepSource removes the memory image only under the restored VM it produced. +func (m *Manager) consumeManagedSleepSource(ctx context.Context, sb *types.Sandbox) (types.VMRecord, error) { + s := sb.VolumeSleep + if _, err := m.sleepVM(ctx, sb, true); err != nil { + return types.VMRecord{}, err + } + source, exists, err := m.eng.InspectHibernateSource(ctx, s.Snapshot) + if err != nil { + return types.VMRecord{}, err + } + if exists && source.ID != s.SourceID { + return types.VMRecord{}, sleepRecoveryError() + } + var effectErr error + if exists { + effectErr = m.eng.SnapshotRemove(ctx, s.Snapshot) + } + _, exists, err = m.eng.InspectHibernateSource(ctx, s.Snapshot) + if err != nil { + return types.VMRecord{}, errors.Join(effectErr, err) + } + if exists { + return types.VMRecord{}, errors.Join(effectErr, sleepRecoveryError()) + } + return m.sleepVM(ctx, sb, true) +} + +func (m *Manager) attachManagedSleep(ctx context.Context, sb *types.Sandbox, vol resolvedVolume, sock string, restored bool) error { + if err := m.verifySleepBacking(sb, vol); err != nil { + return err + } + rec, err := m.sleepVM(ctx, sb, restored) + if err != nil { + return err + } + if len(rec.AttachedDevices.Disks) == 0 { + effectErr := m.eng.DiskAttach(ctx, sb.VMName, vol.disk) + rec, err = m.sleepVM(ctx, sb, restored) + if err != nil { + return errors.Join(effectErr, err) + } + if !exactSleepDisk(rec, vol) { + return errors.Join(effectErr, sleepRecoveryError()) + } + } else if !exactSleepDisk(rec, vol) { + return sleepRecoveryError() + } + if err := m.eng.MountLiveVolume(ctx, sock, vol.applied.Name, vol.applied.Mount, true); err != nil { + return err + } + return m.eng.VerifyMountedVolume(ctx, sock, vol.applied.Name, vol.applied.Mount) +} + +func (m *Manager) verifySleepingVM(ctx context.Context, sb *types.Sandbox) error { + s := sb.VolumeSleep + rec, found, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil { + return err + } + if !found || rec.Config.Name != sb.VMName || rec.ID != s.VMID || rec.State != vmStateStopped || rec.TransitionGeneration <= s.Generation || rec.LastTransitionReason != vmReasonStopUser { + return sleepRecoveryError() + } + return nil +} diff --git a/sandboxd/pool/managed_startup_boundary_test.go b/sandboxd/pool/managed_startup_boundary_test.go new file mode 100644 index 00000000..2a225d83 --- /dev/null +++ b/sandboxd/pool/managed_startup_boundary_test.go @@ -0,0 +1,50 @@ +package pool + +import ( + "os" + "path/filepath" + "testing" +) + +func TestManagedDirectoryRequiresTrustedAncestorOwner(t *testing.T) { + if os.Geteuid() != 0 { + t.Skip("fixture needs root to create a root-owned child below a different owner") + } + for _, tc := range []struct { + name string + owner int + sticky bool + allow bool + }{ + {name: "untrusted-0755", owner: 65534}, + {name: "untrusted-sticky", owner: 65534, sticky: true}, + {name: "root-0755", owner: 0, allow: true}, + {name: "root-sticky", owner: 0, sticky: true, allow: true}, + } { + t.Run(tc.name, func(t *testing.T) { + parent := filepath.Join(t.TempDir(), "parent") + if err := os.Mkdir(parent, 0o700); err != nil { + t.Fatal(err) + } + mode := os.FileMode(0o755) + if tc.sticky { + mode = 0o777 | os.ModeSticky + } + if err := os.Chmod(parent, mode); err != nil { + t.Fatal(err) + } + managed := filepath.Join(parent, "managed") + if err := os.Mkdir(managed, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Chown(parent, tc.owner, tc.owner); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chown(parent, 0, 0) }) + err := privateVolumeDir(managed) + if (err == nil) != tc.allow { + t.Fatalf("trusted=%v, validation error=%v", tc.allow, err) + } + }) + } +} diff --git a/sandboxd/pool/managed_volume.go b/sandboxd/pool/managed_volume.go new file mode 100644 index 00000000..eddfbe7b --- /dev/null +++ b/sandboxd/pool/managed_volume.go @@ -0,0 +1,498 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "slices" + "strings" + "syscall" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +const ( + volumeBusyCode = "volume_busy" + volumeBusyMessage = "volume is held by a sandbox" + volumeKindManaged = "managed" +) + +// VolumeError carries caller-safe details without storage paths or process IDs. +type VolumeError struct { + Status int `json:"-"` + Message string `json:"error"` + Code string `json:"code"` + Volume string `json:"volume,omitempty"` + Holders []types.VolumeHolder `json:"holders,omitempty"` +} + +func (e *VolumeError) Error() string { return e.Message } +func (e *VolumeError) Is(target error) bool { + return target == ErrVolumeBusy && e.Code == volumeBusyCode +} + +func volumeError(status int, code, message string) *VolumeError { + return &VolumeError{Status: status, Code: code, Message: message} +} + +func managedStorageError() *VolumeError { + return volumeError(503, "volume_storage_unavailable", "volume storage requires recovery; retry after recovery") +} + +func managedRecoveryError() *VolumeError { + return volumeError(503, "volume_recovery_required", "volume requires operator recovery; inspect the volume") +} + +func managedNotFound() *VolumeError { return volumeError(404, "volume_not_found", "unknown volume") } + +func (m *Manager) CreateVolume(ctx context.Context, name string, req types.VolumeCreate, tenant string) (types.ManagedVolumeInfo, bool, error) { + owner, err := m.volumeCreateOwner(ctx, name, req, tenant) + if err != nil { + return types.ManagedVolumeInfo{}, false, err + } + r := m.managed + if r == nil { + return types.ManagedVolumeInfo{}, false, managedNotFound() + } + m.mu.Lock() + admission := m.volumeAdmission[name] + m.mu.Unlock() + sleeping := m.sleepHolds(name) + r.mu.Lock() + if err := r.claimName(name); err != nil { + r.mu.Unlock() + return types.ManagedVolumeInfo{}, false, err + } + defer r.releaseName(name) + defer r.mu.Unlock() + if r.closed || r.writeFault != nil { + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + _, managedName := r.data.Volumes[name] + if _, ok := m.volumes[name]; ok || !managedName && (admission.readers > 0 || admission.writers > 0) { + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_name_conflict", "volume name is unavailable") + } + v, exists := r.data.Volumes[name] + if exists && tenant != "" && v.Tenant != tenant { + return types.ManagedVolumeInfo{}, false, managedNotFound() + } + size := req.SizeGiB << 30 + if exists && (v.Size != size || v.Tenant != owner) { + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_create_conflict", "volume already exists with different size or owner") + } + if !exists { + v = managedVolume{ID: volumeID("vol_" + randHex(16)), Name: name, Tenant: owner, Node: r.node, Size: size, State: volumeCreating, CreatedAt: time.Now().UTC()} + r.data.Volumes[name] = v + if err := r.commitAndRelock(); err != nil { + return types.ManagedVolumeInfo{}, false, err + } + } + switch v.State { + case volumeCreating: + r.mu.Unlock() + err := r.completeCreate(v) + r.mu.Lock() + if err != nil { + return types.ManagedVolumeInfo{}, false, err + } + v = r.data.Volumes[name] + case volumeReady: + r.mu.Unlock() + err := r.verify(v) + r.mu.Lock() + if err != nil { + if errors.Is(err, errBackingMismatch) { + return types.ManagedVolumeInfo{}, false, r.markRecovery(v, sleeping) + } + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + case volumeDeleting: + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_deleting", "volume deletion is pending") + case volumeRecovery: + return types.ManagedVolumeInfo{}, false, managedRecoveryError() + default: + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + return r.info(v), !exists, nil +} + +func (r *volumeRegistry) completeCreate(v managedVolume) error { + if err := r.verifyDirectories(); err != nil { + return managedStorageError() + } + path := r.path(v.ID) + f, err := openPrivateFile(path, syscall.O_CREAT|syscall.O_EXCL|syscall.O_RDWR) + if errors.Is(err, os.ErrExist) { + // no recorded inode means no VM ever saw this file, so it is safe to start over + if err = os.Remove(path); err == nil { + f, err = openPrivateFile(path, syscall.O_CREAT|syscall.O_EXCL|syscall.O_RDWR) + } + } + if err != nil { + return managedStorageError() + } + if err = f.Truncate(v.Size); err == nil { + err = f.Sync() + } + st, statErr := f.Stat() + closeErr := f.Close() + if err == nil { + err = errors.Join(statErr, closeErr, syncDir(r.dir+"/disks")) + } + if err != nil { + return managedStorageError() + } + raw := st.Sys().(*syscall.Stat_t) + v.Backing = backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()} + v.State = volumeReady + r.mu.Lock() + defer r.mu.Unlock() + r.data.Volumes[v.Name] = v + return r.commitAndRelock() +} + +// A sleep or a growth reconciles its own backing, so a loss under either stays transient. +func (r *volumeRegistry) recoverable(v managedVolume, sleeping bool) bool { + current, ok := r.data.Volumes[v.Name] + return ok && current == v && v.GrowTo == 0 && !sleeping +} + +func (r *volumeRegistry) markRecovery(v managedVolume, sleeping bool) error { + if !r.recoverable(v, sleeping) { + return managedStorageError() + } + v.State = volumeRecovery + r.data.Volumes[v.Name] = v + if err := r.commitAndRelock(); err != nil { + return err + } + return managedRecoveryError() +} + +func (m *Manager) GetVolume(name, tenant string) (types.ManagedVolumeInfo, error) { + r := m.managed + if r == nil { + return types.ManagedVolumeInfo{}, managedNotFound() + } + m.mu.Lock() + holders := m.volumeHoldersLocked(name) + m.mu.Unlock() + r.mu.Lock() + defer r.mu.Unlock() + if r.closed { + return types.ManagedVolumeInfo{}, managedStorageError() + } + v, ok := r.data.Volumes[name] + if !ok || (tenant != "" && v.Tenant != tenant) { + return types.ManagedVolumeInfo{}, managedNotFound() + } + info := r.info(v) + info.Holders = holders + r.mu.Unlock() + defer r.mu.Lock() + if v.State == volumeReady { + if err := r.verify(v); err != nil { + if !errors.Is(err, errBackingMismatch) { + return types.ManagedVolumeInfo{}, managedStorageError() + } + info.State = string(volumeRecovery) + info.Available = false + } + } + return info, nil +} + +func (m *Manager) DeleteVolume(ctx context.Context, name, expected, tenant string) error { + if err := ctx.Err(); err != nil { + return err + } + r := m.managed + if r == nil { + return managedNotFound() + } + r.mu.Lock() + if err := r.claimName(name); err != nil { + r.mu.Unlock() + return err + } + defer r.releaseName(name) + v, err := r.admitDelete(name, expected, tenant) + r.mu.Unlock() + if err != nil { + return err + } + path := r.path(v.ID) + actual, inspectErr := r.inspectManaged(path) + if errors.Is(inspectErr, errManagedDirectory) { + return managedStorageError() + } + missing := errors.Is(inspectErr, os.ErrNotExist) + m.mu.Lock() + r.mu.Lock() + defer r.mu.Unlock() + v = r.data.Volumes[name] + refusal := m.volumeRefusalLocked(v) + mismatch := refusal == nil && !missing && v.State != volumeCreating && (inspectErr != nil || v.Backing.Inode == 0 || actual != v.Backing) + marked := refusal == nil && !mismatch && v.State != volumeDeleting + if marked { + // marked under m.mu, so a claim that reserves the name afterwards refuses on the state + v.State = volumeDeleting + r.data.Volumes[name] = v + } + m.mu.Unlock() + switch { + case refusal != nil: + return refusal + case mismatch: + return volumeError(503, "volume_backing_mismatch", "volume backing requires operator recovery") + } + if marked { + if err := r.commitAndRelock(); err != nil { + return err + } + } + r.mu.Unlock() + removeErr := removeManagedBacking(path, r.dir+"/disks", missing) + r.mu.Lock() + if removeErr != nil { + return managedStorageError() + } + delete(r.data.Volumes, name) + return r.commitAndRelock() +} + +func (r *volumeRegistry) admitDelete(name, expected, tenant string) (managedVolume, error) { + if r.closed || r.writeFault != nil { + return managedVolume{}, managedStorageError() + } + if expected == "" { + return managedVolume{}, volumeError(428, "volume_precondition_required", "If-Match must identify the volume") + } + v, ok := r.data.Volumes[name] + if !ok || (tenant != "" && v.Tenant != tenant) { + return managedVolume{}, managedNotFound() + } + if v.ID != volumeID(expected) { + return managedVolume{}, volumeError(412, "volume_precondition_failed", "volume identity does not match") + } + return v, nil +} + +// volumeRefusalLocked refuses a delete or grow while growth is pending or any claim or queued removal holds the name; callers hold m.mu and r.mu. +func (m *Manager) volumeRefusalLocked(v managedVolume) error { + if v.GrowTo != 0 { + return growInProgress(v.Name) + } + if held := m.volumeAdmission[v.Name]; held.readers > 0 || held.writers > 0 { + return &VolumeError{Status: 409, Code: volumeBusyCode, Message: volumeBusyMessage, Volume: v.Name, Holders: m.volumeHoldersLocked(v.Name)} + } + return nil +} + +const ( + holderClaimed = "claimed" + holderReleasing = "releasing" +) + +// volumeHoldersLocked lists the claims and queued VM removals holding name, for refusal bodies; callers hold m.mu. +func (m *Manager) volumeHoldersLocked(name string) []types.VolumeHolder { + holders := []types.VolumeHolder{} + add := func(id string, volumes []types.Volume, state string) { + if i := volumeIndex(volumes, name); i >= 0 { + mode := types.VolumeModeRO + if volumes[i].RW() { + mode = types.VolumeModeRW + } + holders = append(holders, types.VolumeHolder{SandboxID: id, Mode: mode, State: state}) + } + } + for _, sb := range m.claimed { + add(sb.ID, heldVolumes(sb), holderClaimed) + } + for _, pending := range m.pendingRemovals { + add(pending.sandboxID, pending.volumes.holds, holderReleasing) + } + slices.SortFunc(holders, func(a, b types.VolumeHolder) int { return strings.Compare(a.SandboxID, b.SandboxID) }) + return holders +} + +func (r *volumeRegistry) info(v managedVolume) types.ManagedVolumeInfo { + return types.ManagedVolumeInfo{Name: v.Name, DefaultMount: types.DefaultVolumeMount(v.Name), SizeBytes: v.Size, Available: v.State == volumeReady, Nodes: 1, Writable: true, Kind: volumeKindManaged, ID: string(v.ID), Tenant: v.Tenant, Node: v.Node, State: string(v.State), CreatedAt: v.CreatedAt} +} + +func (m *Manager) managedInfos(tenant string) []types.VolumeInfo { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + if r.closed { + r.mu.Unlock() + return nil + } + var volumes []managedVolume + for _, v := range r.data.Volumes { + if tenant == "" || tenant == v.Tenant { + volumes = append(volumes, v) + } + } + infos := make([]types.VolumeInfo, len(volumes)) + for i, v := range volumes { + infos[i] = r.info(v).VolumeInfo + } + r.mu.Unlock() + for i, v := range volumes { + if v.State != volumeReady { + continue + } + if err := r.verify(v); err != nil { + if errors.Is(err, errBackingMismatch) { + infos[i].State = string(volumeRecovery) + } + infos[i].Available = false + } + } + return infos +} + +func (m *Manager) lookupManaged(name string) (catalogVolume, bool, error) { + r := m.managed + if r == nil { + return catalogVolume{}, false, nil + } + r.mu.Lock() + defer r.mu.Unlock() + // a write fault leaves memory ahead of disk, so no claim may build on an unrecorded volume + if r.closed || r.writeFault != nil { + return catalogVolume{}, false, managedStorageError() + } + v, ok := r.data.Volumes[name] + if !ok { + return catalogVolume{}, false, nil + } + entry := catalogVolume{disk: engine.VolumeSpec{Name: name, Path: r.path(v.ID), DirectIO: types.DirectIOOff}, tenants: []string{v.Tenant}, writable: true, managedID: v.ID} + switch v.State { + case volumeReady: + case volumeRecovery: + return entry, true, managedRecoveryError() + default: + return entry, true, managedStorageError() + } + if v.GrowTo != 0 { + return entry, true, growInProgress(name) + } + r.mu.Unlock() + err := r.verify(v) + sleeping := err != nil && m.sleepHolds(name) + r.mu.Lock() + switch { + case err == nil: + return entry, true, nil + case errors.Is(err, errBackingMismatch) && r.recoverable(v, sleeping): + return entry, true, managedRecoveryError() + default: + return entry, true, managedStorageError() + } +} + +// managedVolumeNames holds every managed name rw, for a VM whose attached disks are unknown. +func (m *Manager) managedVolumeNames() []types.Volume { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + names := make([]types.Volume, 0, len(r.data.Volumes)) + for name := range r.data.Volumes { + names = append(names, types.Volume{Name: name, Mode: types.VolumeModeRW}) + } + return names +} + +func (m *Manager) managedGossipNames() []string { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + names := make([]string, 0, len(r.data.Volumes)) + for name, v := range r.data.Volumes { + if v.State != volumeDeleting { + names = append(names, types.ManagedVolumeGossipName(v.Tenant, name)) + } + } + return names +} + +// managedPath returns the backing a managed name resolves to. +func (m *Manager) managedPath(name string) (string, bool) { + r := m.managed + if r == nil { + return "", false + } + r.mu.Lock() + defer r.mu.Unlock() + v, ok := r.data.Volumes[name] + return r.path(v.ID), ok +} + +func (m *Manager) managedCatalogAlias(path string) bool { + r := m.managed + if r == nil { + return false + } + clean := filepath.Clean(path) + if canonical, err := filepath.EvalSymlinks(path); err == nil { + clean = canonical + } + if clean == r.dir || strings.HasPrefix(clean, r.dir+string(filepath.Separator)) { + return true + } + info, err := os.Stat(path) + if err != nil { + return false + } + identity := info.Sys().(*syscall.Stat_t) + r.mu.Lock() + defer r.mu.Unlock() + return r.aliases[backingIdentity{Device: backingDeviceID(identity.Dev), Inode: identity.Ino}] +} + +func removeManagedBacking(path, dir string, missing bool) error { + if !missing { + if err := os.Remove(path); err != nil { + return err + } + } + if err := syncDir(dir); err != nil { + return err + } + return clearVolumeDirty(path) +} + +func (m *Manager) volumeCreateOwner(ctx context.Context, name string, req types.VolumeCreate, tenant string) (string, error) { + if !types.ValidVolumeName(name) || req.SizeGiB < 1 || req.SizeGiB > 1024 { + return "", volumeError(400, "invalid_volume", "invalid volume name or size_gib (1 through 1024)") + } + if tenant != "" && req.Tenant != "" && tenant != req.Tenant { + return "", volumeError(403, "volume_owner_forbidden", "tenant cannot select another volume owner") + } + owner := req.Tenant + if tenant != "" { + owner = tenant + } + if !types.NameRe.MatchString(owner) { + return "", volumeError(400, "invalid_volume_owner", "an existing tenant owner is required") + } + if _, ok, err := m.tenants.Lookup(ctx, owner); err != nil { + return "", ErrTenantStoreDown + } else if !ok { + return "", volumeError(400, "invalid_volume_owner", "an existing tenant owner is required") + } + return owner, nil +} diff --git a/sandboxd/pool/managed_volume_test.go b/sandboxd/pool/managed_volume_test.go new file mode 100644 index 00000000..c27c02db --- /dev/null +++ b/sandboxd/pool/managed_volume_test.go @@ -0,0 +1,632 @@ +package pool + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json/v2" + "errors" + "fmt" + "os" + "path/filepath" + "slices" + "sync" + "sync/atomic" + "syscall" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type managedTestEngine struct { + *fakeEngine + next atomic.Int64 + stuckRemove atomic.Bool + missingInventory atomic.Bool + attachEntered chan struct{} + attachResume chan struct{} +} + +func newManagedEngine() *managedTestEngine { return &managedTestEngine{fakeEngine: newFakeEngine()} } + +func (e *managedTestEngine) RunCold(ctx context.Context, name string, key types.PoolKey, storage string) (types.VMRecord, error) { + e.mu.Lock() + e.pids[name] = 1000 + int(e.next.Add(1)) + e.mu.Unlock() + return e.fakeEngine.RunCold(ctx, name, key, storage) +} + +func (e *managedTestEngine) Remove(ctx context.Context, name string) error { + if e.stuckRemove.Load() { + return errors.New("vm rm: device busy") + } + return e.fakeEngine.Remove(ctx, name) +} + +func (e *managedTestEngine) Inspect(ctx context.Context, name string) (types.VMRecord, bool, error) { + if e.missingInventory.Load() { + return types.VMRecord{}, false, nil + } + return e.fakeEngine.Inspect(ctx, name) +} + +func (e *managedTestEngine) DiskAttach(ctx context.Context, name string, spec engine.VolumeSpec) error { + if e.attachEntered != nil { + e.attachEntered <- struct{}{} + <-e.attachResume + } + if err := e.fakeEngine.DiskAttach(ctx, name, spec); err != nil { + return err + } + e.mu.Lock() + defer e.mu.Unlock() + if e.disks == nil { + e.disks = map[string][]types.VMDisk{} + } + e.disks[name] = append(e.disks[name], types.VMDisk{Name: spec.Name, Path: spec.Path, ReadOnly: !spec.RW}) + return nil +} + +func managedConfig(t *testing.T) *config.Config { + t.Helper() + dir := t.TempDir() + return &config.Config{DataDir: filepath.Join(dir, "data"), ManagedVolumeDir: filepath.Join(dir, "managed"), APIToken: "root", Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 1}}} +} + +func openManagedTest(t *testing.T, cfg *config.Config, e *managedTestEngine) *Manager { + t.Helper() + m, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { m.Close() }) + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t2", Token: "t2-token"}); err != nil { + t.Fatal(err) + } + return m +} + +func createManagedTest(t *testing.T, m *Manager, name string) types.ManagedVolumeInfo { + t.Helper() + v, created, err := m.CreateVolume(t.Context(), name, types.VolumeCreate{SizeGiB: 1}, "t1") + if err != nil || !created { + t.Fatalf("create: created=%v err=%v", created, err) + } + return v +} + +func managedOptions(names ...string) ClaimOptions { + o := ClaimOptions{Tenant: "t1"} + for _, name := range names { + o.Volumes = append(o.Volumes, types.Volume{Name: name, Mode: types.VolumeModeRW, AttachOnly: true}) + } + return o +} + +func requireVolumeStatus(t *testing.T, err error, status int) *VolumeError { + t.Helper() + var v *VolumeError + if !errors.As(err, &v) || v.Status != status { + t.Fatalf("error=%v, want volume status %d", err, status) + } + return v +} + +func TestManagedCreateReplaySparseOwnershipAndDeleteIdentity(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + st, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if st.Size() != 1<<30 || st.Sys().(*syscall.Stat_t).Blocks*512 >= st.Size() { + t.Fatal("backing is not a sparse GiB raw image") + } + f, err := os.OpenFile(path, os.O_WRONLY, 0o600) + if err != nil { + t.Fatal(err) + } + _, err = f.WriteAt([]byte("preserved"), 4096) + if err != nil { + t.Fatal(err) + } + f.Close() + if v.Kind != "managed" || v.State != "ready" || v.Tenant != "t1" || v.Node == "" || v.CreatedAt.IsZero() { + t.Fatalf("projection=%+v", v) + } + _, _, err = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 2}, "t1") + requireVolumeStatus(t, err, 409) + _, _, err = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t2") + requireVolumeStatus(t, err, 404) + _, err = m.GetVolume("data", "t2") + requireVolumeStatus(t, err, 404) + m.Close() + m = openManagedTest(t, cfg, e) + repeat, created, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1, Tenant: "t1"}, "t1") + if err != nil || created || repeat.ID != v.ID { + t.Fatalf("replay=%+v created=%v err=%v", repeat, created, err) + } + f, err = os.Open(path) + if err != nil { + t.Fatal(err) + } + buf := make([]byte, 9) + _, err = f.ReadAt(buf, 4096) + f.Close() + if err != nil || string(buf) != "preserved" { + t.Fatalf("replay changed bytes: %q %v", buf, err) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", "", "t1"), 428) + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", "wrong", "t1"), 412) + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + newer := createManagedTest(t, m, "data") + if newer.ID == v.ID { + t.Fatal("identity reused") + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 412) + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t2"), 404) + if _, err := os.Stat(m.managed.path(volumeID(newer.ID))); err != nil { + t.Fatal("stale delete removed replacement", err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if got, err := m.GetVolume("data", "t1"); err != nil || got.ID != newer.ID { + t.Fatalf("restart=%+v %v", got, err) + } +} + +func TestManagedConcurrentCreateAndWriterAdmission(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + var created atomic.Int32 + var wg sync.WaitGroup + ids := make(chan string, 12) + for range 12 { + wg.Go(func() { + v, wasCreated, err := m.CreateVolume(t.Context(), "race", types.VolumeCreate{SizeGiB: 1}, "t1") + if err != nil { + t.Error(err) + return + } + if wasCreated { + created.Add(1) + } + ids <- v.ID + }) + } + wg.Wait() + close(ids) + first := "" + for id := range ids { + if first == "" { + first = id + } + if id != first { + t.Error("concurrent identity changed") + } + } + if created.Load() != 1 { + t.Fatalf("created=%d", created.Load()) + } + successes := make(chan *types.Sandbox, 8) + for range 8 { + wg.Go(func() { + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("race")) + if err == nil { + successes <- sb + return + } + var ve *VolumeError + if !errors.As(err, &ve) || ve.Status != 409 { + t.Errorf("conflict=%+v", err) + } + }) + } + wg.Wait() + close(successes) + var winner *types.Sandbox + count := 0 + for sb := range successes { + winner = sb + count++ + } + if count != 1 { + t.Fatalf("writers=%d", count) + } + if err := m.Release(t.Context(), winner.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + if err := m.DeleteVolume(t.Context(), "race", first, "t1"); err != nil { + t.Fatal(err) + } +} + +func TestManagedActiveClaimAdoption(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + t.Run(fmt.Sprint(syncClaims), func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + adopted, ok := m.Sandbox(sb.ID) + if !ok || adopted.Tenant != "t1" { + t.Fatal("writer not adopted") + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + e.stuckRemove.Store(true) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err == nil { + t.Fatal("a surviving VM reported removed") + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + e.stuckRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + }) + } +} + +func TestManagedCreateInterruptedRetriesOrDeletes(t *testing.T) { + for _, finish := range []string{"put", "delete"} { + t.Run(finish, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + writes := 0 + m.managed.commitFile = func(path string, raw []byte) error { + writes++ + if writes == 2 { + return errors.New("ready metadata failed") + } + return writeVolumeSnapshot(path, raw) + } + _, _, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + requireVolumeStatus(t, err, 503) + m.Close() + m = openManagedTest(t, cfg, e) + v, err := m.GetVolume("data", "t1") + if err != nil || v.State != "creating" { + t.Fatalf("interrupted create = %+v %v, want creating", v, err) + } + path := m.managed.path(volumeID(v.ID)) + before, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if finish == "put" { + if err = os.WriteFile(path, []byte("stale"), before.Mode().Perm()); err != nil { + t.Fatal(err) + } + got, _, putErr := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + head := make([]byte, 5) + f, openErr := os.Open(path) + if openErr != nil { + t.Fatal(openErr) + } + _, readErr := f.ReadAt(head, 0) + st, statErr := f.Stat() + _ = f.Close() + if putErr != nil || got.State != "ready" || got.ID != v.ID || readErr != nil || statErr != nil || st.Size() != 1<<30 || !bytes.Equal(head, make([]byte, 5)) { + t.Fatalf("identical PUT after an interrupted create = %+v %v, want the same volume ready on a fresh 1 GiB file without the leftover bytes", got, putErr) + } + return + } + if delErr := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); delErr != nil { + t.Fatalf("DELETE of an interrupted create = %v, want it removed", delErr) + } + if _, statErr := os.Stat(path); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("interrupted allocation file still present after DELETE: %v", statErr) + } + }) + } +} + +func TestManagedDeleteOfLostBackingDropsRecord(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + lost := createManagedTest(t, m, "lost") + replaced := createManagedTest(t, m, "replaced") + if err := os.Remove(m.managed.path(volumeID(lost.ID))); err != nil { + t.Fatal(err) + } + replacedPath := m.managed.path(volumeID(replaced.ID)) + if err := os.Remove(replacedPath); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(replacedPath, nil, 0o600); err != nil { + t.Fatal(err) + } + if got, err := m.GetVolume("lost", "t1"); err != nil || got.State != string(volumeRecovery) { + t.Fatalf("lost backing = %+v %v, want recovery_required", got, err) + } + if err := m.DeleteVolume(t.Context(), "lost", lost.ID, "t1"); err != nil { + t.Fatalf("DELETE of an unheld volume whose backing is gone = %v, want the record dropped", err) + } + _, err := m.GetVolume("lost", "t1") + requireVolumeStatus(t, err, 404) + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "lost", lost.ID, "t1"), 404) + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "replaced", replaced.ID, "t1"), 503) + if _, err := os.Stat(replacedPath); err != nil { + t.Fatalf("refused DELETE removed a replaced backing: %v", err) + } + if got, err := m.GetVolume("replaced", "t1"); err != nil || got.ID != replaced.ID { + t.Fatalf("refused DELETE dropped the replaced volume's record: %+v %v", got, err) + } +} + +func TestManagedSlowDeleteDoesNotStallOtherCreates(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + slow := createManagedTest(t, m, "slow") + createManagedTest(t, m, "other") + entered, release := make(chan struct{}), make(chan struct{}) + var once sync.Once + m.managed.commitFile = func(path string, raw []byte) error { + once.Do(func() { close(entered); <-release }) + return writeVolumeSnapshot(path, raw) + } + deleted := make(chan error, 1) + go func() { deleted <- m.DeleteVolume(t.Context(), "slow", slow.ID, "t1") }() + <-entered + replayed := make(chan error, 1) + go func() { + _, _, err := m.CreateVolume(t.Context(), "other", types.VolumeCreate{SizeGiB: 1}, "t1") + replayed <- err + }() + select { + case err := <-replayed: + close(release) + if err != nil { + t.Fatal(err) + } + case <-time.After(time.Second): + close(release) + <-replayed + t.Fatal("a PUT replay of one volume waited behind another volume's delete") + } + if err := <-deleted; err != nil { + t.Fatal(err) + } +} + +func TestManagedRegistryFormatHasNoVersion(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "kept") + m.Close() + path := filepath.Join(cfg.ManagedVolumeDir, "registry.json") + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + var doc map[string]any + if err = json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + delete(doc, "version") + doc["future"] = map[string]any{"field": true} + doc["volumes"].(map[string]any)["kept"].(map[string]any)["future"] = "ignored" + if raw, err = json.Marshal(doc); err != nil { + t.Fatal(err) + } + if err = os.WriteFile(path, raw, 0o600); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if got, getErr := m.GetVolume("kept", "t1"); getErr != nil || got.State != "ready" { + t.Fatalf("registry without version and with unknown keys: %+v %v, want kept ready", got, getErr) + } + gone := createManagedTest(t, m, "gone") + if err = m.DeleteVolume(t.Context(), "gone", gone.ID, "t1"); err != nil { + t.Fatal(err) + } + if raw, err = os.ReadFile(path); err != nil { + t.Fatal(err) + } + var keys map[string]any + if err := json.Unmarshal(raw, &keys); err != nil { + t.Fatal(err) + } + for _, key := range []string{"version", "deleted"} { + if _, ok := keys[key]; ok { + t.Fatalf("registry.json after create and delete has %q: %s", key, raw) + } + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "gone", gone.ID, "t1"), 404) +} + +func TestManagedCorruptAndUnsafeStorage(t *testing.T) { + for _, fault := range []string{"truncated", "state", "missing-registry", "replaced", "symlink", "missing", "permissions"} { + t.Run(fault, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + m.Close() + switch fault { + case "truncated": + if err := os.WriteFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json"), []byte("{"), 0o600); err != nil { + t.Fatal(err) + } + case "state": + raw, err := os.ReadFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json")) + if err != nil { + t.Fatal(err) + } + var data volumeRegistryData + if err := json.Unmarshal(raw, &data); err != nil { + t.Fatal(err) + } + corrupt := data.Volumes["data"] + corrupt.State = "future_state" + data.Volumes["data"] = corrupt + raw, _ = json.Marshal(data) + if err := os.WriteFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json"), raw, 0o600); err != nil { + t.Fatal(err) + } + case "missing-registry": + if err := os.Remove(filepath.Join(cfg.ManagedVolumeDir, "registry.json")); err != nil { + t.Fatal(err) + } + case "replaced": + if err := os.Rename(path, path+".original"); err != nil { + t.Fatal(err) + } + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0o600) + if err != nil { + t.Fatal(err) + } + if err := f.Truncate(1 << 30); err != nil { + t.Fatal(err) + } + f.Close() + case "symlink": + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + if err := os.Symlink("/etc/passwd", path); err != nil { + t.Fatal(err) + } + case "missing": + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + case "permissions": + if err := os.Chmod(path, 0o644); err != nil { + t.Fatal(err) + } + } + if fault == "truncated" || fault == "state" || fault == "missing-registry" || fault == "replaced" { + if _, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("an unsafe registry opened") + } + return + } + m = openManagedTest(t, cfg, e) + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 503) + if len(e.volumeSpecs) != 0 { + t.Fatal("unsafe backing attached") + } + if fault == "symlink" || fault == "permissions" { + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 503) + } + }) + } +} + +func TestManagedStoreExclusiveAndDirectoryPermissions(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + if _, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("second daemon accepted locked store") + } + m.Close() + if err := os.Chmod(cfg.ManagedVolumeDir, 0o755); err != nil { + t.Fatal(err) + } + if _, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("public managed directory accepted") + } +} + +func TestManagedStalledAttachDoesNotBlockOtherClaims(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + unrelated, claimErr := m.ClaimProvision(t.Context(), testKey, ClaimOptions{}) + if claimErr != nil { + t.Fatal(claimErr) + } + e.attachEntered = make(chan struct{}, 1) + e.attachResume = make(chan struct{}) + done := make(chan error, 1) + go func() { _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")); done <- err }() + <-e.attachEntered + if _, err := m.Renew(t.Context(), unrelated.ID, Cred{Operator: true}, time.Minute, ""); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), unrelated.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + close(e.attachResume) + if err := <-done; err != nil { + t.Fatal(err) + } +} + +func TestManagedListReportsLostDirectoryAsUnavailableNotRecovery(t *testing.T) { + cfg := managedConfig(t) + m := openManagedTest(t, cfg, newManagedEngine()) + createManagedTest(t, m, "data") + disks := filepath.Join(cfg.ManagedVolumeDir, "disks") + if err := os.Rename(disks, disks+".aside"); err != nil { + t.Fatal(err) + } + defer func() { _ = os.Rename(disks+".aside", disks) }() + for _, info := range m.Volumes("t1", nil) { + if info.Name == "data" && (info.Available || info.State != "ready") { + t.Fatalf("lost directory listed as state=%q available=%v, want ready and unavailable like GET's storage error", info.State, info.Available) + } + } +} + +func TestVolumeRefusalsIndependentOfManagedDir(t *testing.T) { + cfg := managedConfig(t) + m := openManagedTest(t, cfg, newManagedEngine()) + createManagedTest(t, m, "data") + for name, opts := range map[string]ClaimOptions{ + "unknown name": {Tenant: "t1", Volumes: []types.Volume{{Name: "nope", AttachOnly: true}}}, + "foreign volume": {Tenant: "t2", Volumes: []types.Volume{{Name: "data", Mode: types.VolumeModeRW, AttachOnly: true}}}, + } { + if _, err := m.ClaimProvision(t.Context(), testKey, opts); !errors.Is(err, ErrVolumeUnavailable) { + t.Errorf("%s: claim = %v, want ErrVolumeUnavailable like a node without managed storage", name, err) + } + local, advertised, err := m.VolumePlacement(testKey, opts.Tenant, []string{opts.Volumes[0].Name}) + if err != nil || local || !slices.Equal(advertised, []string{types.ManagedVolumeGossipName(opts.Tenant, opts.Volumes[0].Name)}) { + t.Errorf("%s: placement = local %v keys %v err %v, want a mesh lookup by the caller's own tenant key", name, local, advertised, err) + } + } +} + +func TestVolumeNamesAdvertiseManagedKeys(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + createManagedTest(t, m, "data") + sum := sha256.Sum256([]byte("volume|t1|data")) + key := "managed/" + hex.EncodeToString(sum[:16]) + names := m.VolumeNames() + if !slices.Contains(names, key) || slices.Contains(names, "data") { + t.Fatalf("gossiped volume names = %v, want the tenant-scoped key %s and never the bare name", names, key) + } + for _, info := range m.Volumes("", map[string]int{key: 1}) { + if info.Name == key { + t.Fatalf("root listing shows the gossip key %s as a volume", key) + } + } +} diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 9e0124cb..a8a499b0 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -72,6 +72,7 @@ const ( hibernatePrefix = "sbx-hib-" forkPrefix = "sbx-fork-" vmStateRunning = "running" + vmStateStopped = "stopped" vmStateCreating = "creating" goldenStampSuffix = ".stamp" @@ -91,6 +92,7 @@ var ( ErrArchiveStoreDown = archives.ErrUnavailable ErrTenantRemoved = errors.New("the claim's tenant was removed: it runs to its deadline but cannot renew, fork, or wake from the archive") ErrVolumeUnavailable = fmt.Errorf("%w: unknown or unavailable volume", ErrBadVolume) + ErrVolumePlacement = errors.New("no node holds both the template and the volumes") ErrNoWarm = errors.New("no warm sandbox for key") ErrUnknownSandbox = errors.New("unknown sandbox or bad token") ErrUnknownTemplate = errors.New("unknown promoted template") @@ -144,6 +146,9 @@ type Engine interface { Warmup(ctx context.Context, vsockSocket string, argv []string) error TrimCow(ctx context.Context, vsockSocket string) error CheckLiveVolumeSupport(ctx context.Context) error + RunVolumeHook(ctx context.Context, socket string, call engine.VolumeHookCall) error + VerifyMountedVolume(ctx context.Context, socket, name, mount string) error + InspectHibernateSource(ctx context.Context, name string) (engine.HibernateSource, bool, error) DiskAttach(ctx context.Context, vmName string, spec engine.VolumeSpec) error DetachVolume(ctx context.Context, vmName, sock, name, mount string) error MountVolume(ctx context.Context, vsockSocket, name, mount string, rw bool) error @@ -214,6 +219,8 @@ type pendingRemoval struct { staleCreate bool volumes volumeTeardown notBefore time.Time + // inventoryUnknown marks a stale VM whose disks could not be inspected; it holds every managed name. + inventoryUnknown bool } type pool struct { @@ -328,9 +335,11 @@ type Manager struct { store *claimStore volumes map[string]catalogVolume // volumeAdmission counts the live holders of each volume name; guarded by m.mu. - volumeAdmission map[string]volumeHolders - volumeRuns map[*types.Sandbox]*volumeRun - volumeErrors map[*types.Sandbox]map[string]string + volumeAdmission map[string]volumeHolders + managed *volumeRegistry + managedSleepHook *config.ManagedVolumeHibernate + volumeRuns map[*types.Sandbox]*volumeRun + volumeErrors map[*types.Sandbox]map[string]string poolStore *poolStore // poolShared is the cell's pool set with a meta store; poolVersion, under m.mu, is its newest version applied. @@ -438,44 +447,48 @@ type Manager struct { // NewManager builds a manager from the node config over backends the caller selected from it. func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *egress.SecretStore, b Backends) (*Manager, error) { + if err := cfg.ValidateManagedVolumeStore(); err != nil { + return nil, err + } refill := cmp.Or(cfg.RefillConcurrency, defaultRefill) m := &Manager{ - eng: eng, - dataDir: cfg.DataDir, - egress: cfg.HasEgress(), - releaseDelay: time.Duration(cfg.ReleaseDelaySeconds) * time.Second, - vmmRestart: cfg.VMMRestart, - cocoondSocket: cfg.CocoondSocket, - maxFork: cmp.Or(cfg.MaxForkCount, defaultMaxFork), - store: newClaimStore(cfg.DataDir, cfg.SyncClaims), - volumes: make(map[string]catalogVolume, len(cfg.Volumes)), - volumeAdmission: map[string]volumeHolders{}, - volumeRuns: map[*types.Sandbox]*volumeRun{}, - volumeErrors: map[*types.Sandbox]map[string]string{}, - poolStore: newPoolStore(cfg.DataDir), - poolShared: b.CellPools, - archives: b.Archives, - lease: leaseTiming{ttl: leaseTTL, every: leaseRenew, margin: leaseMargin}, - tenants: b.Tenants, - ckpts: b.Checkpoints, - ckptsShared: b.SharedCheckpoints, - tpls: b.Templates, - pools: make(map[types.PoolKey]*pool, len(cfg.Pools)), - claimed: map[string]*types.Sandbox{}, - pendingRemovals: map[string]pendingRemoval{}, - tenantLive: map[string]int{}, - readers: &readerSet{live: map[string]*reader{}}, - archiving: map[string]struct{}{}, - pendingCks: map[string]struct{}{}, - recLocks: map[string]*sync.RWMutex{}, - recRefs: map[string]int{}, - healPending: map[string]struct{}{}, - healAbort: map[string]struct{}{}, - refillSem: make(chan struct{}, refill), - maintSem: make(chan struct{}, max(1, refill-max(1, refill/4))), - probeSem: make(chan struct{}, refill), - refillKick: make(chan struct{}, 1), - healSem: make(chan struct{}, maxConcurrentHeals), + eng: eng, + dataDir: cfg.DataDir, + egress: cfg.HasEgress(), + releaseDelay: time.Duration(cfg.ReleaseDelaySeconds) * time.Second, + vmmRestart: cfg.VMMRestart, + cocoondSocket: cfg.CocoondSocket, + maxFork: cmp.Or(cfg.MaxForkCount, defaultMaxFork), + store: newClaimStore(cfg.DataDir, cfg.SyncClaims), + volumes: make(map[string]catalogVolume, len(cfg.Volumes)), + volumeAdmission: map[string]volumeHolders{}, + volumeRuns: map[*types.Sandbox]*volumeRun{}, + managedSleepHook: cfg.ManagedVolumeHibernate, + volumeErrors: map[*types.Sandbox]map[string]string{}, + poolStore: newPoolStore(cfg.DataDir), + poolShared: b.CellPools, + archives: b.Archives, + lease: leaseTiming{ttl: leaseTTL, every: leaseRenew, margin: leaseMargin}, + tenants: b.Tenants, + ckpts: b.Checkpoints, + ckptsShared: b.SharedCheckpoints, + tpls: b.Templates, + pools: make(map[types.PoolKey]*pool, len(cfg.Pools)), + claimed: map[string]*types.Sandbox{}, + pendingRemovals: map[string]pendingRemoval{}, + tenantLive: map[string]int{}, + readers: &readerSet{live: map[string]*reader{}}, + archiving: map[string]struct{}{}, + pendingCks: map[string]struct{}{}, + recLocks: map[string]*sync.RWMutex{}, + recRefs: map[string]int{}, + healPending: map[string]struct{}{}, + healAbort: map[string]struct{}{}, + refillSem: make(chan struct{}, refill), + maintSem: make(chan struct{}, max(1, refill-max(1, refill/4))), + probeSem: make(chan struct{}, refill), + refillKick: make(chan struct{}, 1), + healSem: make(chan struct{}, maxConcurrentHeals), } for _, volume := range cfg.Volumes { m.volumes[volume.Name] = catalogVolume{ @@ -550,6 +563,26 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg } m.warnVolumeTenants(ctx, cfg.Volumes) m.warnMissingClasses(ctx) + if cfg.ManagedVolumeDir != "" { + node, hostErr := os.Hostname() + if cfg.Mesh != nil { + node = cfg.Mesh.ID() + } + if node == "" { + return nil, fmt.Errorf("managed volume node identity unavailable: %v", hostErr) + } + var err error + m.managed, err = openVolumeRegistry(cfg.ManagedVolumeDir, node) + if err != nil { + return nil, err + } + for name, entry := range m.volumes { + _, collision := m.managed.data.Volumes[name] + if collision || m.managedCatalogAlias(entry.disk.Path) { + return nil, errors.Join(errors.New("catalog conflicts with existing managed resource"), m.managed.close()) + } + } + } return m, nil } diff --git a/sandboxd/pool/pool_test.go b/sandboxd/pool/pool_test.go index 136cd118..179dc956 100644 --- a/sandboxd/pool/pool_test.go +++ b/sandboxd/pool/pool_test.go @@ -2009,3 +2009,14 @@ func seedGolden(t *testing.T, m *Manager, imageID string) string { } return g } + +func (f *fakeEngine) RunVolumeHook(_ context.Context, _ string, _ engine.VolumeHookCall) error { + return nil +} + +func (f *fakeEngine) InspectHibernateSource(ctx context.Context, name string) (engine.HibernateSource, bool, error) { + names, err := f.SnapshotList(ctx) + return engine.HibernateSource{ID: name, Name: name}, slices.Contains(names, name), err +} + +func (f *fakeEngine) VerifyMountedVolume(_ context.Context, _, _, _ string) error { return nil } diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index 3bfa3cf1..c8fff28a 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -44,6 +44,9 @@ func (m *Manager) Reconcile(ctx context.Context) error { sb.SetEnv(sb.Env) rec, ok := live[sb.VMName] switch { + case ok && sb.VolumeSleep != nil: + // the sleep record, not the VM state, says where the next dial resumes + projectSleepState(sb, sb.VolumeSleep) case sb.ArchiveCk != "": // archived: no local VM by design, so adopt the stub and let the first exec wake it m.addClaimLocked(sb) @@ -69,6 +72,9 @@ func (m *Manager) Reconcile(ctx context.Context) error { m.adoptVolumes(heldVolumes(sb)) owned[sb.VMName] = true referenced[sb.HibernateSnap] = true + if sb.VolumeSleep != nil { + referenced[sb.VolumeSleep.Snapshot] = true + } } saveErr := m.store.save(m.claimed) pooled := make(map[string]bool, len(m.pools)) @@ -174,18 +180,31 @@ func (m *Manager) removeStaleVM(ctx context.Context, name string, rec types.VMRe if m.removeVM(ctx, name) { return true } - m.queueRemoval(name, "", rec.TapDevice(), volumeTeardown{holds: m.holdStaleVolumes(ctx, name, rec)}, time.Time{}) + pending := pendingRemoval{tap: rec.TapDevice()} + if rec.State == vmStateRunning { + pending.volumes.holds, pending.inventoryUnknown = m.holdStaleVolumes(ctx, name) + } + m.requeueRemoval(name, pending) return false } -func (m *Manager) holdStaleVolumes(ctx context.Context, name string, rec types.VMRecord) []types.Volume { - if rec.State != vmStateRunning { - return nil +// holdStaleVolumes adopts the disks a stale VM has attached; with its inventory unknown it holds every managed name. +func (m *Manager) holdStaleVolumes(ctx context.Context, name string) ([]types.Volume, bool) { + holds, err := m.staleVolumes(ctx, name) + if err != nil { + log.WithFunc("pool.holdStaleVolumes").Warnf(ctx, "inspect stale vm %s, holding every managed volume: %v", name, err) + holds = m.managedVolumeNames() } + m.mu.Lock() + m.adoptVolumes(holds) + m.mu.Unlock() + return holds, err != nil +} + +func (m *Manager) staleVolumes(ctx context.Context, name string) ([]types.Volume, error) { inspected, _, err := m.eng.Inspect(ctx, name) if err != nil { - log.WithFunc("pool.holdStaleVolumes").Warnf(ctx, "inspect stale vm %s, its volumes stay unheld: %v", name, err) - return nil + return nil, err } holds := make([]types.Volume, len(inspected.AttachedDevices.Disks)) for i, disk := range inspected.AttachedDevices.Disks { @@ -194,10 +213,7 @@ func (m *Manager) holdStaleVolumes(ctx context.Context, name string, rec types.V holds[i].Mode = types.VolumeModeRW } } - m.mu.Lock() - m.adoptVolumes(holds) - m.mu.Unlock() - return holds + return holds, nil } // resyncEgress re-locks adopted egress claims, quarantining any it cannot lock. diff --git a/sandboxd/pool/remove.go b/sandboxd/pool/remove.go index 75efec26..5ee385fe 100644 --- a/sandboxd/pool/remove.go +++ b/sandboxd/pool/remove.go @@ -53,8 +53,12 @@ func (m *Manager) removeOrRetry(ctx context.Context, name, sandboxID string, td } func (m *Manager) queueRemoval(name, sandboxID, tap string, td volumeTeardown, notBefore time.Time) { + m.requeueRemoval(name, pendingRemoval{sandboxID: sandboxID, tap: tap, volumes: td, notBefore: notBefore}) +} + +func (m *Manager) requeueRemoval(name string, pending pendingRemoval) { m.mu.Lock() - m.pendingRemovals[name] = pendingRemoval{sandboxID: sandboxID, tap: tap, volumes: td, notBefore: notBefore} + m.pendingRemovals[name] = pending m.mu.Unlock() } @@ -97,13 +101,30 @@ func (m *Manager) retryRemoval(ctx context.Context, name string, pending pending return } } + if pending.inventoryUnknown { + pending = m.reinspectStale(ctx, name, pending) + } if !m.removeVM(ctx, name) { - m.queueRemoval(name, pending.sandboxID, pending.tap, pending.volumes, time.Time{}) + m.requeueRemoval(name, pendingRemoval{sandboxID: pending.sandboxID, tap: pending.tap, volumes: pending.volumes, inventoryUnknown: pending.inventoryUnknown}) return } m.finishRemoval(ctx, pending) } +// reinspectStale narrows a stale VM's blanket hold to the disks it has attached once inspect answers. +func (m *Manager) reinspectStale(ctx context.Context, name string, pending pendingRemoval) pendingRemoval { + holds, err := m.staleVolumes(ctx, name) + if err != nil { + return pending + } + m.mu.Lock() + m.adoptVolumes(holds) + m.releaseVolumes(pending.volumes.holds) + m.mu.Unlock() + pending.volumes.holds, pending.inventoryUnknown = holds, false + return pending +} + func (m *Manager) finishRemoval(ctx context.Context, pending pendingRemoval) { m.finishVolumeTeardown(ctx, pending.volumes) if pending.sandboxID != "" { diff --git a/sandboxd/pool/volume.go b/sandboxd/pool/volume.go index 5a1e8eed..bf7ef0f2 100644 --- a/sandboxd/pool/volume.go +++ b/sandboxd/pool/volume.go @@ -31,8 +31,9 @@ type volumeHolders struct { } type resolvedVolume struct { - disk engine.VolumeSpec - applied types.Volume + disk engine.VolumeSpec + applied types.Volume + managedID volumeID } // volumeTeardown captures marker paths at quiesce time; a later config can re-point the entry. @@ -42,9 +43,10 @@ type volumeTeardown struct { } type catalogVolume struct { - disk engine.VolumeSpec - tenants []string - writable bool + managedID volumeID + disk engine.VolumeSpec + tenants []string + writable bool } func (v catalogVolume) allowed(tenant string) bool { @@ -71,10 +73,15 @@ func (m *Manager) Volumes(tenant string, holders map[string]int) []types.VolumeI } infos = append(infos, info) } + infos = append(infos, m.managedInfos(tenant)...) // root can see peer-only names; a tenant needs local catalog metadata to apply the access list if tenant == "" { for name, nodes := range holders { - if _, ok := m.volumes[name]; !ok { + if !types.ValidVolumeName(name) { + continue + } + _, managed, _ := m.lookupManaged(name) + if _, ok := m.volumes[name]; !ok && !managed { infos = append(infos, types.VolumeInfo{ Name: name, DefaultMount: types.DefaultVolumeMount(name), @@ -95,36 +102,46 @@ func (m *Manager) VolumeNames() []string { names = append(names, name) } } + names = append(names, m.managedGossipNames()...) slices.Sort(names) return names } -// VolumePlacement checks catalog access and reports whether every named image is local. -func (m *Manager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) { +// VolumePlacement checks catalog access, reports whether every named image is local, and returns the mesh keys to place them by. +func (m *Manager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, []string, error) { if err := m.validate(key); err != nil { - return false, err - } - if len(names) == 0 { - return true, nil + return false, nil, err } local := true + advertised := make([]string, 0, len(names)) for _, name := range names { - volume, ok := m.volumes[name] - if !ok { + volume, ok, err := m.lookupVolume(name) + if !ok && err != nil { + return false, nil, err + } + if !ok || !volume.allowed(tenant) { if tenant == "" { - local = false - continue + advertised = append(advertised, name) + } else { + // the tenant may own this name on a peer; a miss there answers like an unknown name + advertised = append(advertised, types.ManagedVolumeGossipName(tenant, name)) } - return false, ErrVolumeUnavailable + local = false + continue } - if !volume.allowed(tenant) { - return false, ErrVolumeUnavailable + if err != nil { + return false, nil, err + } + if volume.managedID != "" { + advertised = append(advertised, types.ManagedVolumeGossipName(volume.tenants[0], name)) + } else { + advertised = append(advertised, name) } if _, err := os.Stat(volume.disk.Path); err != nil { local = false } } - return local, nil + return local, advertised, nil } func (m *Manager) resolveVolumes(ctx context.Context, tenant string, requested []types.Volume) ([]resolvedVolume, error) { @@ -138,10 +155,16 @@ func (m *Manager) resolveVolumes(ctx context.Context, tenant string, requested [ } resolved := make([]resolvedVolume, 0, len(applied)) for _, volume := range applied { - entry, ok := m.volumes[volume.Name] + entry, ok, lookupErr := m.lookupVolume(volume.Name) + if !ok && lookupErr != nil { + return nil, lookupErr + } if !ok || !entry.allowed(tenant) { return nil, ErrVolumeUnavailable } + if lookupErr != nil { + return nil, lookupErr + } resolvedEntry, resolveErr := m.resolveCatalogVolume(ctx, volume) if resolveErr != nil { return nil, resolveErr @@ -152,7 +175,10 @@ func (m *Manager) resolveVolumes(ctx context.Context, tenant string, requested [ } func (m *Manager) resolveCatalogVolume(ctx context.Context, volume types.Volume) (resolvedVolume, error) { - entry, ok := m.volumes[volume.Name] + entry, ok, lookupErr := m.lookupVolume(volume.Name) + if lookupErr != nil { + return resolvedVolume{}, lookupErr + } if !ok { return resolvedVolume{}, ErrVolumeUnavailable } @@ -167,7 +193,17 @@ func (m *Manager) resolveCatalogVolume(ctx context.Context, volume types.Volume) disk := entry.disk disk.RW = volume.RW() volume.AttachOnly = false // past validation the empty mount carries it - return resolvedVolume{disk: disk, applied: volume}, nil + return resolvedVolume{disk: disk, applied: volume, managedID: entry.managedID}, nil +} + +func (m *Manager) lookupVolume(name string) (catalogVolume, bool, error) { + if entry, ok := m.volumes[name]; ok { + if m.managedCatalogAlias(entry.disk.Path) { + return catalogVolume{}, false, managedStorageError() + } + return entry, true, nil + } + return m.lookupManaged(name) } // confirmVolumesClean refuses a read-only claim of a dirty image; run after holds are taken. @@ -232,6 +268,18 @@ func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeT if mounts == 0 { return td } + // a sleep step owns the guest mount; removing the VM ends it + if sb.VolumeSleep != nil { + // capture is journaled only once prepare unmounted and detached the disk + if phase := sb.VolumeSleep.Phase; phase == types.SleepCapturing || phase == types.SleepSleeping { + for _, volume := range sb.Volumes { + if path, ok := m.managedPath(volume.Name); ok && needsUnmount(volume) { + td.clears = append(td.clears, path) + } + } + } + return td + } logger := log.WithFunc("pool.quiesceVolumes") // cancellation-immune like removal: a caller hanging up must not skip the flush. ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), quiesceBudget(mounts)) @@ -248,6 +296,8 @@ func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeT } if entry, ok := m.volumes[volume.Name]; ok { td.clears = append(td.clears, entry.disk.Path) + } else if path, ok := m.managedPath(volume.Name); ok { + td.clears = append(td.clears, path) } } if stuck { diff --git a/sandboxd/pool/volume_device_test.go b/sandboxd/pool/volume_device_test.go new file mode 100644 index 00000000..998f68a8 --- /dev/null +++ b/sandboxd/pool/volume_device_test.go @@ -0,0 +1,31 @@ +package pool + +import ( + "encoding/json/v2" + "testing" +) + +func TestBackingDeviceIDPreservesPersistedNumbers(t *testing.T) { + for _, tc := range []struct { + name string + device uint64 + want string + }{ + {"linux-zero", backingDeviceID(uint64(0)), "0"}, + {"linux-high-bit", backingDeviceID(uint64(1) << 63), "9223372036854775808"}, + {"linux-max", backingDeviceID(^uint64(0)), "18446744073709551615"}, + {"darwin-positive", backingDeviceID(int32(16777234)), "16777234"}, + {"darwin-negative", backingDeviceID(int32(-1)), "18446744073709551615"}, + {"darwin-min", backingDeviceID(int32(-2147483648)), "18446744071562067968"}, + } { + t.Run(tc.name, func(t *testing.T) { + raw, err := json.Marshal(backingIdentity{Device: tc.device}) + if err != nil { + t.Fatal(err) + } + if want := `{"device":` + tc.want + `,"inode":0,"size":0}`; string(raw) != want { + t.Fatalf("persisted backing identity = %s, want %s", raw, want) + } + }) + } +} diff --git a/sandboxd/pool/volume_mutation.go b/sandboxd/pool/volume_mutation.go index 33dc50e2..cfdaca0f 100644 --- a/sandboxd/pool/volume_mutation.go +++ b/sandboxd/pool/volume_mutation.go @@ -98,6 +98,9 @@ func (m *Manager) changeVolumes(ctx context.Context, id, tenant string, change v if err = validateLiveTarget(held, target); err != nil { return view, err } + if err = m.validateManagedTarget(held, target); err != nil { + return view, err + } additions := volumeDifference(target, held) if len(additions) != 0 { if err = m.tenantRemoved(ctx, sb.Tenant); err != nil { @@ -108,6 +111,9 @@ func (m *Manager) changeVolumes(ctx context.Context, id, tenant string, change v if err != nil { return view, err } + if slices.ContainsFunc(resolved, func(v resolvedVolume) bool { return v.managedID != "" }) { + return view, fmt.Errorf("%w: managed volumes support claim-time attachment only", ErrBadVolume) + } if err = m.checkVolumeRun(ctx, sb, run); err != nil { return view, err } diff --git a/sandboxd/pool/volume_registry.go b/sandboxd/pool/volume_registry.go new file mode 100644 index 00000000..84c6f9d0 --- /dev/null +++ b/sandboxd/pool/volume_registry.go @@ -0,0 +1,439 @@ +package pool + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "strings" + "sync" + "sync/atomic" + "syscall" + "time" + + "github.com/projecteru2/core/log" + + "github.com/cocoonstack/sandbox/sandboxd/types" + "github.com/cocoonstack/sandbox/sandboxd/utils" +) + +type ( + volumeID string + volumePhase string +) + +const ( + volumeCreating volumePhase = "creating" + volumeReady volumePhase = "ready" + volumeDeleting volumePhase = "deleting" + volumeRecovery volumePhase = "recovery_required" +) + +type backingIdentity struct { + Device uint64 `json:"device"` + Inode uint64 `json:"inode"` + Size int64 `json:"size"` +} + +type managedVolume struct { + ID volumeID `json:"id"` + Name string `json:"name"` + Tenant string `json:"tenant"` + Node string `json:"node"` + Size int64 `json:"size"` + GrowTo int64 `json:"grow_to,omitempty"` + Backing backingIdentity `json:"backing"` + State volumePhase `json:"state"` + CreatedAt time.Time `json:"created_at"` +} + +// volumeRegistryData is registry.json: resources only, since claims.json rows are the holds. +type volumeRegistryData struct { + Volumes map[string]managedVolume `json:"volumes"` +} + +type volumeRegistry struct { + rootDir, disksDir os.FileInfo + mu sync.Mutex + writeMu sync.Mutex + seq, written uint64 + writeFault error + retrying atomic.Bool + closed bool + aliases map[backingIdentity]bool + dir, node string + lock *os.File + data volumeRegistryData + commitFile func(string, []byte) error + // busy names a create or delete in flight, so its file I/O excludes only that name + busy map[string]bool + idle *sync.Cond +} + +var ( + errBackingMismatch = errors.New("managed backing is missing or changed") + errManagedDirectory = errors.New("managed directory changed") +) + +var managedIDPattern = regexp.MustCompile(`^vol_[0-9a-f]{32}$`) + +func openVolumeRegistry(dir, node string) (*volumeRegistry, error) { + if !filepath.IsAbs(dir) || filepath.Clean(dir) != dir || dir == "/" { + return nil, errors.New("invalid managed volume directory") + } + if err := privateVolumeDir(dir); err != nil { + return nil, err + } + if err := privateVolumeDir(filepath.Join(dir, "disks")); err != nil { + return nil, err + } + lock, err := openPrivateFile(filepath.Join(dir, "lock"), syscall.O_CREAT|syscall.O_RDWR) + if err != nil { + return nil, fmt.Errorf("open managed volume lock: %w", err) + } + if err = syscall.Flock(int(lock.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil { + return nil, errors.Join(fmt.Errorf("managed volume directory is already in use: %w", err), lock.Close()) + } + r := &volumeRegistry{dir: dir, node: node, lock: lock, commitFile: writeVolumeSnapshot, busy: map[string]bool{}} + r.idle = sync.NewCond(&r.mu) + if err = r.load(); err != nil { + return nil, errors.Join(fmt.Errorf("open managed volume registry: %w", err), lock.Close()) + } + return r, nil +} + +func (r *volumeRegistry) load() error { + var err error + if r.rootDir, err = os.Lstat(r.dir); err != nil { + return err + } + if r.disksDir, err = os.Lstat(filepath.Join(r.dir, "disks")); err != nil { + return err + } + r.data = volumeRegistryData{Volumes: map[string]managedVolume{}} + f, err := openPrivateFile(filepath.Join(r.dir, "registry.json"), syscall.O_RDONLY) + if errors.Is(err, os.ErrNotExist) { + entries, readErr := os.ReadDir(filepath.Join(r.dir, "disks")) + if readErr != nil || len(entries) != 0 { + return errors.Join(errors.New("missing registry with existing backing files"), readErr) + } + r.mu.Lock() + defer r.mu.Unlock() + // a failed first write leaves writeFault for the retrier, like any later write + _ = r.commitAndRelock() + return nil + } + if err != nil { + return err + } + defer func() { _ = f.Close() }() + raw, err := io.ReadAll(f) + if err != nil { + return err + } + if err = json.Unmarshal(raw, &r.data); err != nil { + return err + } + if err = r.validate(); err != nil { + return err + } + for name, v := range r.data.Volumes { + if v.GrowTo != 0 { + actual, _ := inspectBacking(r.path(v.ID)) + r.data.Volumes[name] = settleGrowth(v, actual) + } + } + r.indexBackings() + return nil +} + +func privateVolumeDir(path string) error { + for p := path; p != "/"; p = filepath.Dir(p) { + st, err := os.Lstat(p) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return err + } + owner := st.Sys().(*syscall.Stat_t).Uid + if owner != 0 && int64(owner) != int64(os.Geteuid()) { + return errors.New("managed directory ancestors must belong to root or the daemon user") + } + if st.Mode().Perm()&0o022 != 0 && st.Mode()&os.ModeSticky == 0 { + return errors.New("managed directory ancestors must not be writable by others") + } + if !st.IsDir() || st.Mode()&os.ModeSymlink != 0 { + return errors.New("managed directory component is not a directory") + } + } + if err := os.MkdirAll(path, 0o700); err != nil { + return err + } + st, err := os.Lstat(path) + if err != nil { + return err + } + if !st.IsDir() || st.Mode().Perm()&0o077 != 0 { + return errors.New("managed directory must be private (0700)") + } + if int64(st.Sys().(*syscall.Stat_t).Uid) != int64(os.Geteuid()) { + return errors.New("managed directory must belong to the daemon user") + } + return nil +} + +func openPrivateFile(path string, flags int) (*os.File, error) { + fd, err := syscall.Open(path, flags|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0o600) + if err != nil { + return nil, err + } + f := os.NewFile(uintptr(fd), path) + st, err := f.Stat() + if err != nil || !st.Mode().IsRegular() || st.Mode().Perm()&0o077 != 0 || st.Sys().(*syscall.Stat_t).Nlink != 1 || int64(st.Sys().(*syscall.Stat_t).Uid) != int64(os.Geteuid()) { + _ = f.Close() + return nil, errors.New("managed file must be private, owned, regular, and unlinked elsewhere") + } + return f, nil +} + +func (r *volumeRegistry) path(id volumeID) string { + return filepath.Join(r.dir, "disks", string(id)+".raw") +} + +func inspectBacking(path string) (backingIdentity, error) { + f, err := openPrivateFile(path, syscall.O_RDONLY) + if err != nil { + return backingIdentity{}, err + } + defer func() { _ = f.Close() }() + st, err := f.Stat() + if err != nil { + return backingIdentity{}, err + } + raw := st.Sys().(*syscall.Stat_t) + return backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()}, nil +} + +// Darwin uses signed device numbers; retain their existing uint64 representation. +func backingDeviceID[T ~uint64 | ~int32](device T) uint64 { + if device < 0 { + return ^uint64(^device) + } + return uint64(device) +} + +func (r *volumeRegistry) verify(v managedVolume) error { + actual, err := r.inspectManaged(r.path(v.ID)) + if errors.Is(err, errManagedDirectory) { + return err + } + if errors.Is(err, os.ErrNotExist) || err == nil && (v.Backing.Inode == 0 || actual.Device != v.Backing.Device || actual.Inode != v.Backing.Inode || (actual.Size != v.Size && (v.GrowTo == 0 || actual.Size != v.GrowTo))) { + return errBackingMismatch + } + return err +} + +func (r *volumeRegistry) inspectManaged(path string) (backingIdentity, error) { + if err := r.verifyDirectories(); err != nil { + return backingIdentity{}, errors.Join(errManagedDirectory, err) + } + actual, err := inspectBacking(path) + if errors.Is(err, os.ErrNotExist) { + // a directory renamed between the check and the open must stay a transient error + if dirErr := r.verifyDirectories(); dirErr != nil { + return actual, errors.Join(errManagedDirectory, dirErr) + } + } + return actual, err +} + +func (r *volumeRegistry) validate() error { + if r.data.Volumes == nil { + return errors.New("invalid volume registry format") + } + ids := map[volumeID]bool{} + for name, v := range r.data.Volumes { + if err := validateVolumeRecord(name, v, ids); err != nil { + return err + } + } + entries, err := os.ReadDir(filepath.Join(r.dir, "disks")) + if err != nil { + return err + } + for _, entry := range entries { + name := strings.TrimSuffix(entry.Name(), volumeDirtySuffix) + id := volumeID(strings.TrimSuffix(name, ".raw")) + if name != string(id)+".raw" || !ids[id] { + return errors.New("unregistered managed backing requires quarantine") + } + } + return nil +} + +func validateVolumeRecord(name string, v managedVolume, ids map[volumeID]bool) error { + if name != v.Name || !types.ValidVolumeName(name) || !managedIDPattern.MatchString(string(v.ID)) || !types.NameRe.MatchString(v.Tenant) || v.Size < 1<<30 || v.Size > 1024<<30 || v.Size%(1<<30) != 0 || v.CreatedAt.IsZero() || v.Node == "" || ids[v.ID] { + return errors.New("invalid managed volume record") + } + switch v.State { + case volumeCreating, volumeReady, volumeDeleting, volumeRecovery: + default: + return errors.New("invalid managed volume state") + } + if (v.State == volumeReady || v.Backing.Inode != 0) && (v.Backing.Inode == 0 || v.Backing.Size != v.Size) { + return errors.New("invalid managed backing record") + } + if v.GrowTo != 0 && (v.State != volumeReady || v.GrowTo <= v.Size || v.GrowTo > 1024<<30 || v.GrowTo%(1<<30) != 0) { + return errors.New("invalid managed growth intent") + } + ids[v.ID] = true + return nil +} + +func (r *volumeRegistry) commitAndRelock() error { + if r.closed { + return managedStorageError() + } + r.seq++ + seq := r.seq + r.indexBackings() + r.mu.Unlock() + r.writeMu.Lock() + r.mu.Lock() + if r.closed { + r.writeMu.Unlock() + return managedStorageError() + } + if r.written >= seq { + r.writeMu.Unlock() + if r.writeFault != nil { + return managedStorageError() + } + return nil + } + seq = r.seq + raw, err := json.Marshal(r.data) + write := r.commitFile + r.mu.Unlock() + if err == nil { + err = r.verifyDirectories() + } + if err == nil { + err = write(filepath.Join(r.dir, "registry.json"), raw) + } + r.mu.Lock() + previousFault := r.writeFault + r.writeFault = err + if err == nil { + r.written = seq + } + r.writeMu.Unlock() + r.mu.Unlock() + if previousFault == nil && err != nil { + log.WithFunc("pool.volumeRegistry").Error(context.Background(), err, "managed registry write failed; retaining ownership for retry") + } else if previousFault != nil && err == nil { + log.WithFunc("pool.volumeRegistry").Info(context.Background(), "managed registry persistence recovered") + } + r.mu.Lock() + if err != nil || r.writeFault != nil { + return managedStorageError() + } + return nil +} + +func (r *volumeRegistry) indexBackings() { + r.aliases = make(map[backingIdentity]bool, len(r.data.Volumes)) + for _, v := range r.data.Volumes { + if v.Backing.Inode != 0 { + r.aliases[backingIdentity{Device: v.Backing.Device, Inode: v.Backing.Inode}] = true + } + } +} + +func (m *Manager) retryManagedSnapshot() { + if m.managed == nil { + return + } + r := m.managed + r.mu.Lock() + retry := r.writeFault != nil && !r.closed && r.retrying.CompareAndSwap(false, true) + r.mu.Unlock() + if !retry { + return + } + go func() { + defer r.retrying.Store(false) + r.mu.Lock() + defer r.mu.Unlock() + _ = r.commitAndRelock() + }() +} + +func writeVolumeSnapshot(path string, raw []byte) error { + return utils.WriteFileSync(path, raw, 0o600) +} + +// claimName waits out another create or delete of name, then marks it busy; callers hold r.mu. +func (r *volumeRegistry) claimName(name string) error { + for r.busy[name] && !r.closed { + r.idle.Wait() + } + if r.closed { + return managedStorageError() + } + r.busy[name] = true + return nil +} + +func (r *volumeRegistry) releaseName(name string) { + r.mu.Lock() + defer r.mu.Unlock() + delete(r.busy, name) + r.idle.Broadcast() +} + +func (r *volumeRegistry) close() error { + r.mu.Lock() + r.closed = true + for len(r.busy) > 0 { + r.idle.Wait() + } + r.mu.Unlock() + r.writeMu.Lock() + defer r.writeMu.Unlock() + if r.lock == nil { + return nil + } + err := r.lock.Close() + r.lock = nil + return err +} + +// Close releases the managed directory lock after requests and workers have stopped. +func (m *Manager) Close() error { + if m.managed == nil { + return nil + } + return m.managed.close() +} + +func (r *volumeRegistry) verifyDirectories() error { + for _, pair := range []struct { + path string + original os.FileInfo + }{{r.dir, r.rootDir}, {filepath.Join(r.dir, "disks"), r.disksDir}} { + st, err := os.Lstat(pair.path) + if err != nil { + return err + } + if !st.IsDir() || st.Mode().Perm()&0o077 != 0 || !os.SameFile(st, pair.original) { + return errors.New("managed directory identity or permissions changed") + } + } + return nil +} diff --git a/sandboxd/pool/volume_test.go b/sandboxd/pool/volume_test.go index 9367ae04..e12c2e68 100644 --- a/sandboxd/pool/volume_test.go +++ b/sandboxd/pool/volume_test.go @@ -454,22 +454,23 @@ func TestVolumePlacementChecksAccessAndLocalAvailability(t *testing.T) { {Name: "remote", Path: filepath.Join(t.TempDir(), "missing.img"), Tenants: []string{"acme"}}, }) - if local, err := m.VolumePlacement(testKey, "acme", []string{"local"}); err != nil || !local { + if local, _, err := m.VolumePlacement(testKey, "acme", []string{"local"}); err != nil || !local { t.Errorf("local placement=(%v, %v), want true, nil", local, err) } - if local, err := m.VolumePlacement(testKey, "acme", []string{"local", "remote"}); err != nil || local { + if local, _, err := m.VolumePlacement(testKey, "acme", []string{"local", "remote"}); err != nil || local { t.Errorf("partial local placement=(%v, %v), want false, nil", local, err) } - _, forbiddenErr := m.VolumePlacement(testKey, "beta", []string{"local"}) - _, unknownErr := m.VolumePlacement(testKey, "beta", []string{"unknown"}) - if forbiddenErr == nil || unknownErr == nil || forbiddenErr.Error() != unknownErr.Error() { - t.Errorf("forbidden=%q unknown=%q, want byte-identical errors", forbiddenErr, unknownErr) + for _, name := range []string{"local", "unknown"} { + local, advertised, err := m.VolumePlacement(testKey, "beta", []string{name}) + if err != nil || local || !slices.Equal(advertised, []string{types.ManagedVolumeGossipName("beta", name)}) { + t.Errorf("beta placement of %s = (%v, %v, %v), want a mesh lookup by beta's own key, alike for forbidden and unknown", name, local, advertised, err) + } } - if local, err := m.VolumePlacement(testKey, "", []string{"peer-only"}); err != nil || local { + if local, _, err := m.VolumePlacement(testKey, "", []string{"peer-only"}); err != nil || local { t.Errorf("root peer-only placement=(%v, %v), want false, nil", local, err) } badKey := types.PoolKey{Template: "rt:24.04", Net: "lan", Size: types.SizeSmall} - if _, err := m.VolumePlacement(badKey, "", []string{"local"}); !errors.Is(err, ErrBadKey) { + if _, _, err := m.VolumePlacement(badKey, "", []string{"local"}); !errors.Is(err, ErrBadKey) { t.Errorf("invalid key error=%v, want ErrBadKey", err) } } diff --git a/sandboxd/server/api.go b/sandboxd/server/api.go index 6e26e0c9..f612d71b 100644 --- a/sandboxd/server/api.go +++ b/sandboxd/server/api.go @@ -6,6 +6,10 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) +type VolumeResponse struct { + Volume types.ManagedVolumeInfo `json:"volume"` +} + // TTLField is the shared requested-lease field; zero means the server default. type TTLField struct { TTLSeconds int `json:"ttl_seconds,omitzero"` diff --git a/sandboxd/server/managed_volumes.go b/sandboxd/server/managed_volumes.go new file mode 100644 index 00000000..15591450 --- /dev/null +++ b/sandboxd/server/managed_volumes.go @@ -0,0 +1,111 @@ +package server + +import ( + "cmp" + "io" + "net/http" + "strings" + + "github.com/cocoonstack/sandbox/sandboxd/pool" + "github.com/cocoonstack/sandbox/sandboxd/types" + "github.com/cocoonstack/sandbox/sandboxd/utils" +) + +func volumeHTTPError(w http.ResponseWriter, status int, code, message string) { + writeJSON(w, status, &pool.VolumeError{Status: status, Code: code, Message: message}) +} + +func volumeStorageUnavailable(w http.ResponseWriter) { + w.Header().Set("Retry-After", retryAfterSeconds) + volumeHTTPError(w, 503, "volume_storage_unavailable", "volume storage unavailable; retry") +} + +func (s *Server) handleCreateVolume(w http.ResponseWriter, r *http.Request) { + raw, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxBodyBytes)) + var req types.VolumeCreate + if err != nil || utils.DecodeStrictJSON(raw, &req) != nil { + volumeHTTPError(w, 400, "invalid_volume", "invalid request body") + return + } + name, tenant := r.PathValue("name"), tenantFrom(r.Context()) + if owner := cmp.Or(tenant, req.Tenant); s.placer != nil && owner != "" { + // gossip lag leaves a short window in which two nodes can both accept the name + if peers := s.placer.VolumeOwners([]string{types.ManagedVolumeGossipName(owner, name)}); len(peers) > 0 { + volumeHTTPError(w, 409, "volume_name_conflict", "volume exists on another node: "+strings.Join(s.clientAddrs(peers), ", ")) + return + } + } + v, created, err := s.mgr.CreateVolume(r.Context(), name, req, tenant) + if err != nil { + if !writePoolErr(w, err) { + volumeStorageUnavailable(w) + } + return + } + status := http.StatusOK + if created { + status = http.StatusCreated + } + w.Header().Set("ETag", `"`+v.ID+`"`) + writeJSON(w, status, VolumeResponse{Volume: v}) +} + +func (s *Server) handleGetVolume(w http.ResponseWriter, r *http.Request) { + v, err := s.mgr.GetVolume(r.PathValue("name"), tenantFrom(r.Context())) + if err != nil { + writePoolErr(w, err) + return + } + w.Header().Set("ETag", `"`+v.ID+`"`) + writeJSON(w, http.StatusOK, VolumeResponse{Volume: v}) +} + +func (s *Server) handleDeleteVolume(w http.ResponseWriter, r *http.Request) { + id, ok := volumeIfMatch(w, r) + if !ok { + return + } + if err := s.mgr.DeleteVolume(r.Context(), r.PathValue("name"), id, tenantFrom(r.Context())); err != nil { + if !writePoolErr(w, err) { + volumeStorageUnavailable(w) + } + return + } + w.WriteHeader(http.StatusNoContent) +} + +func (s *Server) handleGrowVolume(w http.ResponseWriter, r *http.Request) { + id, ok := volumeIfMatch(w, r) + if !ok { + return + } + raw, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxBodyBytes)) + var req types.VolumeGrow + if err != nil || utils.DecodeStrictJSON(raw, &req) != nil { + volumeHTTPError(w, 400, "invalid_volume_grow", "invalid request body") + return + } + v, err := s.mgr.GrowVolume(r.Context(), r.PathValue("name"), id, req.SizeGiB, tenantFrom(r.Context())) + if err != nil { + if !writePoolErr(w, err) { + volumeStorageUnavailable(w) + } + return + } + w.Header().Set("ETag", `"`+v.ID+`"`) + writeJSON(w, http.StatusOK, VolumeResponse{Volume: v}) +} + +// volumeIfMatch returns the one quoted volume ID in If-Match, answering 428 or 412 itself. +func volumeIfMatch(w http.ResponseWriter, r *http.Request) (string, bool) { + expected := r.Header.Get("If-Match") + if expected == "" { + volumeHTTPError(w, 428, "volume_precondition_required", "If-Match must identify the volume") + return "", false + } + if len(expected) < 3 || expected[0] != '"' || expected[len(expected)-1] != '"' || strings.ContainsAny(expected[1:len(expected)-1], "\",\\") { + volumeHTTPError(w, 412, "volume_precondition_failed", "If-Match must contain one quoted volume identity") + return "", false + } + return expected[1 : len(expected)-1], true +} diff --git a/sandboxd/server/managed_volumes_test.go b/sandboxd/server/managed_volumes_test.go new file mode 100644 index 00000000..e1fb5b5d --- /dev/null +++ b/sandboxd/server/managed_volumes_test.go @@ -0,0 +1,85 @@ +package server + +import ( + "context" + "io" + "net/http" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/pool" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type httpReply struct { + status int + retryAfter string + body string +} + +func sendVolumeRequest(t *testing.T, ts, method, path, token, match, body string) httpReply { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), method, ts+path, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Authorization", "Bearer "+token) + if match != "" { + req.Header.Set("If-Match", match) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + raw, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + return httpReply{status: resp.StatusCode, retryAfter: resp.Header.Get("Retry-After"), body: string(raw)} +} + +func TestManagedVolumeRoutesUseRequireToken(t *testing.T) { + routes := []struct{ method, path, match, body string }{ + {http.MethodPut, "/v1/volumes/x", "", `{"size_gib":1}`}, + {http.MethodGet, "/v1/volumes/x", "", ""}, + {http.MethodDelete, "/v1/volumes/x", `"vol_123"`, ""}, + {http.MethodPost, "/v1/volumes/x/grow", `"vol_123"`, growBody}, + } + fm := &fakeManager{tenants: []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}}} + mgr := &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: fm, err: pool.ErrTenantStoreDown, get: func(string, string) (types.ManagedVolumeInfo, error) { + t.Error("an unauthenticated request reached the volume manager") + return types.ManagedVolumeInfo{}, nil + }}, grow: func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + t.Error("an unauthenticated request reached growth") + return types.ManagedVolumeInfo{}, nil + }} + ts := newTestServer(t, "root-token", mgr, nil) + + for _, scope := range []struct { + name, token string + tenantErr error + status int + }{ + {"sandbox token", "sandbox-token", nil, http.StatusUnauthorized}, + {"tenant store outage", "t1-token", pool.ErrTenantStoreDown, http.StatusServiceUnavailable}, + } { + fm.tenantErr = scope.tenantErr + want := sendVolumeRequest(t, ts.URL, http.MethodGet, "/v1/sandboxes", scope.token, "", "") + if want.status != scope.status || scope.status == http.StatusServiceUnavailable && want.retryAfter == "" { + t.Fatalf("%s: GET /v1/sandboxes = %+v, want %d", scope.name, want, scope.status) + } + for _, r := range routes { + if got := sendVolumeRequest(t, ts.URL, r.method, r.path, scope.token, r.match, r.body); got != want { + t.Errorf("%s: %s %s = %+v, want the GET /v1/sandboxes reply %+v", scope.name, r.method, r.path, got, want) + } + } + } + + fm.tenantErr = nil + created := sendVolumeRequest(t, ts.URL, http.MethodPut, "/v1/volumes/x", "root-token", "", `{"size_gib":1,"tenant":"t1"}`) + if created.status != http.StatusServiceUnavailable || created.retryAfter == "" { + t.Fatalf("create during a tenant store outage = %+v, want 503 with Retry-After", created) + } +} diff --git a/sandboxd/server/server.go b/sandboxd/server/server.go index 78b44fd6..bb7ca2f6 100644 --- a/sandboxd/server/server.go +++ b/sandboxd/server/server.go @@ -53,6 +53,7 @@ var poolErrHTTP = []struct { {pool.ErrBadTenant, http.StatusBadRequest, ""}, {pool.ErrBadConfig, http.StatusBadRequest, ""}, {pool.ErrReloadRefused, http.StatusConflict, ""}, + {pool.ErrVolumePlacement, http.StatusConflict, ""}, {pool.ErrTenantRemoved, http.StatusForbidden, ""}, {pool.ErrTenantStoreDown, http.StatusServiceUnavailable, "tenant store down; retry"}, {pool.ErrPoolStoreDown, http.StatusServiceUnavailable, "pool set store down; retry"}, @@ -95,8 +96,12 @@ type Manager interface { Checkpoint(ctx context.Context, id string, cred pool.Cred, name, tenant string) (types.Checkpoint, error) Counters() pool.Counters TenantClaims() map[string]int - VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) + VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, []string, error) Volumes(tenant string, holders map[string]int) []types.VolumeInfo + CreateVolume(ctx context.Context, name string, req types.VolumeCreate, tenant string) (types.ManagedVolumeInfo, bool, error) + GetVolume(name, tenant string) (types.ManagedVolumeInfo, error) + GrowVolume(ctx context.Context, name, expectedID string, sizeGiB int64, tenant string) (types.ManagedVolumeInfo, error) + DeleteVolume(ctx context.Context, name, expectedID, tenant string) error GetVolumes(id, tenant string) (types.SandboxVolumesResponse, error) SetVolumes(ctx context.Context, id, tenant string, volumes []types.Volume) (types.SandboxVolumesResponse, error) PatchVolumes(ctx context.Context, id, tenant string, patch types.VolumePatch) (types.SandboxVolumesResponse, error) @@ -269,6 +274,10 @@ func (s *Server) Handler() http.Handler { mux := http.NewServeMux() mux.HandleFunc("POST /v1/claim", s.requireToken(s.handleClaim)) mux.HandleFunc("GET /v1/volumes", s.requireToken(s.handleVolumes)) + mux.HandleFunc("PUT /v1/volumes/{name}", s.requireToken(s.handleCreateVolume)) + mux.HandleFunc("GET /v1/volumes/{name}", s.requireToken(s.handleGetVolume)) + mux.HandleFunc("POST /v1/volumes/{name}/grow", s.requireToken(s.handleGrowVolume)) + mux.HandleFunc("DELETE /v1/volumes/{name}", s.requireToken(s.handleDeleteVolume)) mux.HandleFunc("POST /v1/sandboxes/{id}/release", s.handleSandboxVerb("release", s.mgr.Release)) mux.HandleFunc("POST /v1/sandboxes/{id}/hibernate", s.handleSandboxVerb("hibernate", s.mgr.Hibernate)) mux.HandleFunc("POST /v1/sandboxes/{id}/wake", s.handleSandboxVerb("wake", s.mgr.Wake)) @@ -356,7 +365,7 @@ func (s *Server) handleVolumeClaim(w http.ResponseWriter, r *http.Request, req C return } req.Volumes = volumes - redirected, err := s.redirectVolumeClaim(r.Context(), w, &req, key, hash, tenant) + redirected, advertised, err := s.redirectVolumeClaim(r.Context(), w, &req, key, hash, tenant) if err != nil { writeResult(w, r, "claim", key.Template, "provisioning failed", err, func() {}) return @@ -371,7 +380,7 @@ func (s *Server) handleVolumeClaim(w http.ResponseWriter, r *http.Request, req C } else { sb, err = s.mgr.ClaimWarm(r.Context(), key, o) if errors.Is(err, pool.ErrNoWarm) { - if s.placer != nil && !req.NoRedirect && s.writeRedirect(w, s.placer.VolumeCandidates(hash, types.VolumeNames(req.Volumes))) { + if s.placer != nil && !req.NoRedirect && s.writeRedirect(w, s.placer.VolumeCandidates(hash, advertised)) { return } sb, err = s.mgr.ClaimProvision(r.Context(), key, o) @@ -392,11 +401,10 @@ func (s *Server) handleVolumes(w http.ResponseWriter, r *http.Request) { }) } -func (s *Server) redirectVolumeClaim(ctx context.Context, w http.ResponseWriter, req *ClaimRequest, key types.PoolKey, hash, tenant string) (bool, error) { - names := types.VolumeNames(req.Volumes) - localVolumes, err := s.mgr.VolumePlacement(key, tenant, names) +func (s *Server) redirectVolumeClaim(ctx context.Context, w http.ResponseWriter, req *ClaimRequest, key types.PoolKey, hash, tenant string) (bool, []string, error) { + localVolumes, names, err := s.mgr.VolumePlacement(key, tenant, types.VolumeNames(req.Volumes)) if err != nil { - return false, err + return false, nil, err } localTemplate := s.mgr.HasPromotedTemplate(ctx, key, tenant) @@ -408,10 +416,10 @@ func (s *Server) redirectVolumeClaim(ctx context.Context, w http.ResponseWriter, promoted := req.RequirePromoted || localTemplate || len(templateOwners) > 0 req.RequirePromoted = promoted if localVolumes && (!promoted || localTemplate || pooled) { - return false, nil + return false, names, nil } if s.placer == nil || req.NoRedirect { - return false, pool.ErrVolumeUnavailable + return false, nil, pool.ErrVolumeUnavailable } var owners []string @@ -425,11 +433,14 @@ func (s *Server) redirectVolumeClaim(ctx context.Context, w http.ResponseWriter, if len(owners) == 0 { owners = s.placer.VolumeOwners(names) } + if len(owners) == 0 && localVolumes { + return false, nil, pool.ErrVolumePlacement + } if len(owners) == 0 { - return false, pool.ErrVolumeUnavailable + return false, nil, pool.ErrVolumeUnavailable } writeJSON(w, http.StatusOK, ClaimResponse{Redirect: s.clientAddrs(owners), RequirePromoted: promoted}) - return true, nil + return true, nil, nil } func (s *Server) templateOwners(query func(string) []string, hash, tenant string) []string { diff --git a/sandboxd/server/server_http.go b/sandboxd/server/server_http.go index 48bf2a94..acac07cd 100644 --- a/sandboxd/server/server_http.go +++ b/sandboxd/server/server_http.go @@ -15,6 +15,7 @@ import ( "github.com/projecteru2/core/log" + "github.com/cocoonstack/sandbox/sandboxd/pool" "github.com/cocoonstack/sandbox/sandboxd/tenants" "github.com/cocoonstack/sandbox/sandboxd/types" "github.com/cocoonstack/sandbox/sandboxd/utils" @@ -170,6 +171,11 @@ func metadataFilter(pairs []string) (types.Metadata, error) { } func writePoolErr(w http.ResponseWriter, err error) bool { + if volumeErr, ok := errors.AsType[*pool.VolumeError](err); ok { + mapPoolErr(w, err) + writeJSON(w, volumeErr.Status, volumeErr) + return true + } code, msg, ok := mapPoolErr(w, err) if ok { writeErr(w, code, msg) @@ -180,12 +186,28 @@ func writePoolErr(w http.ResponseWriter, err error) bool { func mapPoolErr(w http.ResponseWriter, err error) (code int, msg string, ok bool) { code, msg, ok = lookupPoolErr(err) if ok && code == http.StatusServiceUnavailable { + if _, unsupported := errors.AsType[*pool.LiveVolumeUnsupported](err); unsupported { + return code, msg, ok + } + if volumeErr, managed := errors.AsType[*pool.VolumeError](err); managed && + (volumeErr.Code == "volume_backing_mismatch" || volumeErr.Code == "volume_recovery_required") { + return code, msg, ok + } w.Header().Set("Retry-After", retryAfterSeconds) } return code, msg, ok } func lookupPoolErr(err error) (code int, msg string, ok bool) { + if volumeErr, ok := errors.AsType[*pool.VolumeError](err); ok { + return volumeErr.Status, volumeErr.Message, true + } + if conflict, ok := errors.AsType[*pool.VolumeMountConflict](err); ok { + return http.StatusConflict, conflict.Error(), true + } + if unsupported, ok := errors.AsType[*pool.LiveVolumeUnsupported](err); ok { + return http.StatusServiceUnavailable, unsupported.Error(), true + } for _, m := range poolErrHTTP { if errors.Is(err, m.err) { return m.code, cmp.Or(m.msg, err.Error()), true diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 6debba66..0d64c9fe 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -143,6 +143,38 @@ func TestHibernateVolumeCaptureMapsConflict(t *testing.T) { } } +func TestManagedSleepReportsGuestMountAndRuntimeRefusals(t *testing.T) { + const busy = `cannot unmount "/volumes/data"; close guest references and retry` + for _, tt := range []struct { + name string + err error + code int + cause string + }{ + {"busy unmount", errors.Join(&pool.VolumeMountConflict{Message: busy}, nil), http.StatusConflict, busy}, + {"below cocoon floor", &pool.LiveVolumeUnsupported{Version: "v0.7.1"}, http.StatusServiceUnavailable, "Cocoon >= v0.7.2"}, + } { + for _, verb := range []string{"hibernate", "wake"} { + t.Run(tt.name+"/"+verb, func(t *testing.T) { + refuse := func(string, string) error { return tt.err } + ts := newTestServer(t, "", &fakeManager{hibernate: refuse, wake: refuse}, nil) + resp := doReq(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/"+verb, "tok", "") + defer resp.Body.Close() + var got map[string]string + if err := json.UnmarshalRead(resp.Body, &got); err != nil { + t.Fatal(err) + } + if resp.StatusCode != tt.code || !strings.Contains(got["error"], tt.cause) { + t.Fatalf("status=%d body=%v, want %d naming %q", resp.StatusCode, got, tt.code, tt.cause) + } + if resp.Header.Get("Retry-After") != "" { + t.Fatalf("Retry-After=%q on a refusal that retrying alone cannot clear", resp.Header.Get("Retry-After")) + } + }) + } + } +} + func TestRenewGrantsAndReportsTheDeadline(t *testing.T) { want := time.Now().Add(time.Hour).UTC().Truncate(time.Second) var gotTTL time.Duration @@ -2762,12 +2794,13 @@ func (f *fakeManager) DeleteTenant(_ context.Context, name string) error { return nil } -func (f *fakeManager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) { +func (f *fakeManager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, []string, error) { f.gotTenant = tenant if f.volumePlacement == nil { - return true, nil + return true, names, nil } - return f.volumePlacement(key, tenant, names) + local, err := f.volumePlacement(key, tenant, names) + return local, names, err } func (f *fakeManager) Volumes(tenant string, holders map[string]int) []types.VolumeInfo { @@ -2778,6 +2811,20 @@ func (f *fakeManager) Volumes(tenant string, holders map[string]int) []types.Vol return f.volumeCatalog(tenant, holders) } +func (f *fakeManager) CreateVolume(context.Context, string, types.VolumeCreate, string) (types.ManagedVolumeInfo, bool, error) { + return types.ManagedVolumeInfo{}, false, nil +} + +func (f *fakeManager) GetVolume(string, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, nil +} + +func (f *fakeManager) GrowVolume(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, nil +} + +func (f *fakeManager) DeleteVolume(context.Context, string, string, string) error { return nil } + func (f *fakeManager) Sandboxes(tenant, claimRef string, metadata types.Metadata) []pool.SandboxSummary { f.gotTenant = tenant f.gotMetadata = metadata diff --git a/sandboxd/server/volume_grow_test.go b/sandboxd/server/volume_grow_test.go new file mode 100644 index 00000000..e7af9ad8 --- /dev/null +++ b/sandboxd/server/volume_grow_test.go @@ -0,0 +1,225 @@ +package server + +import ( + "context" + "encoding/json/v2" + "errors" + "io" + "net/http" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/pool" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +const growBody = `{"size_gib":40}` + +func TestManagedVolumeRecoveryRetryHints(t *testing.T) { + for _, tc := range []struct { + code string + retry bool + }{ + {"volume_backing_mismatch", false}, + {"volume_storage_unavailable", true}, + } { + t.Run(tc.code, func(t *testing.T) { + mgr := &volumeManagerFake{fakeManager: &fakeManager{}, get: func(string, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, &pool.VolumeError{Status: http.StatusServiceUnavailable, Code: tc.code, Message: "volume unavailable"} + }} + ts := newTestServer(t, "root", mgr, nil) + resp := doReq(t, http.MethodGet, ts.URL+"/v1/volumes/data", "root", "") + defer resp.Body.Close() + var body pool.VolumeError + if err := json.UnmarshalRead(resp.Body, &body); err != nil { + t.Fatal(err) + } + if resp.StatusCode != http.StatusServiceUnavailable || body.Code != tc.code || body.Message != "volume unavailable" { + t.Fatalf("status=%d error=%+v", resp.StatusCode, body) + } + if retry := resp.Header.Get("Retry-After") != ""; retry != tc.retry { + t.Fatalf("Retry-After=%q, want retry hint=%v", resp.Header.Get("Retry-After"), tc.retry) + } + }) + } +} + +func TestManagedVolumeStorageFallbackKeepsRetryAfter(t *testing.T) { + outage := errors.New("write /private/claims.json: no space left on device") + for _, tc := range []struct { + name, method, path, match, body string + mgr Manager + }{ + {"create", http.MethodPut, "/v1/volumes/data", "", `{"size_gib":1}`, &volumeManagerFake{fakeManager: &fakeManager{}, err: outage}}, + {"delete", http.MethodDelete, "/v1/volumes/data", `"vol_123"`, "", &volumeManagerFake{fakeManager: &fakeManager{}, err: outage}}, + {"grow", http.MethodPost, "/v1/volumes/data/grow", `"vol_123"`, growBody, &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: &fakeManager{}}, grow: func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, outage + }}}, + } { + t.Run(tc.name, func(t *testing.T) { + ts := newTestServer(t, "root-token", tc.mgr, nil) + req, _ := http.NewRequest(tc.method, ts.URL+tc.path, strings.NewReader(tc.body)) + req.Header.Set("Authorization", "Bearer root-token") + if tc.match != "" { + req.Header.Set("If-Match", tc.match) + } + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + var body pool.VolumeError + if err := json.UnmarshalRead(resp.Body, &body); err != nil { + t.Fatal(err) + } + if resp.StatusCode != http.StatusServiceUnavailable || body.Code != "volume_storage_unavailable" || strings.Contains(body.Message, "/private") { + t.Fatalf("status=%d body=%+v", resp.StatusCode, body) + } + if resp.Header.Get("Retry-After") == "" { + t.Fatalf("storage outage %q without Retry-After", body.Message) + } + }) + } +} + +type volumeManagerFake struct { + *fakeManager + get func(string, string) (types.ManagedVolumeInfo, error) + err error +} + +func (f *volumeManagerFake) CreateVolume(context.Context, string, types.VolumeCreate, string) (types.ManagedVolumeInfo, bool, error) { + return types.ManagedVolumeInfo{}, false, f.err +} + +func (f *volumeManagerFake) GetVolume(name, tenant string) (types.ManagedVolumeInfo, error) { + return f.get(name, tenant) +} + +func (f *volumeManagerFake) DeleteVolume(context.Context, string, string, string) error { return f.err } + +type volumeGrowFake struct { + *volumeManagerFake + grow func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) +} + +func (f *volumeGrowFake) GrowVolume(ctx context.Context, name, id string, sizeGiB int64, tenant string) (types.ManagedVolumeInfo, error) { + return f.grow(ctx, name, id, sizeGiB, tenant) +} + +func TestGrowVolumeHTTP(t *testing.T) { + calls := 0 + mgr := &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: &fakeManager{ + tenants: []tenants.TenantSpec{{Name: "tenant-1", Token: "tenant-token"}}, + }}, grow: func(ctx context.Context, name, id string, sizeGiB int64, tenant string) (types.ManagedVolumeInfo, error) { + calls++ + if ctx.Err() != nil || name != "data" || id != "vol_123" || tenant != "tenant-1" || sizeGiB != 40 { + t.Errorf("grow args = %q %q %d %q, context %v", name, id, sizeGiB, tenant, ctx.Err()) + } + return types.ManagedVolumeInfo{ID: "vol_123", SizeBytes: 40 << 30}, nil + }} + ts := newTestServer(t, "root-token", mgr, nil) + req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/volumes/data/grow", strings.NewReader(growBody)) + req.Header.Set("Authorization", "Bearer tenant-token") + req.Header.Set("If-Match", `"vol_123"`) + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + var body VolumeResponse + if err := json.UnmarshalRead(resp.Body, &body); err != nil || resp.StatusCode != http.StatusOK || calls != 1 || body.Volume.SizeBytes != 40<<30 { + t.Fatalf("status %d calls %d body %+v err %v", resp.StatusCode, calls, body, err) + } + if resp.Header.Get("ETag") != `"vol_123"` { + t.Fatalf("ETag %q", resp.Header.Get("ETag")) + } +} + +func TestGrowVolumeHTTPRefusesBeforeAdmission(t *testing.T) { + for _, tc := range []struct { + name, token, match, body string + status int + }{ + {"no token", "", `"vol_123"`, growBody, 401}, + {"sandbox token", "sandbox-token", `"vol_123"`, growBody, 401}, + {"missing identity", "root-token", "", growBody, 428}, + {"wildcard", "root-token", "*", growBody, 412}, + {"weak identity", "root-token", `W/"vol_123"`, growBody, 412}, + {"identity list", "root-token", `"vol_123", "vol_456"`, growBody, 412}, + {"unknown member", "root-token", `"vol_123"`, `{"extra":true,"size_gib":40}`, 400}, + {"duplicate member", "root-token", `"vol_123"`, `{"size_gib":20,"size_gib":40}`, 400}, + {"trailing body", "root-token", `"vol_123"`, growBody + `{}`, 400}, + {"fractional target", "root-token", `"vol_123"`, `{"size_gib":40.5}`, 400}, + } { + t.Run(tc.name, func(t *testing.T) { + mgr := &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: &fakeManager{}}, grow: func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + t.Error("invalid request reached growth admission") + return types.ManagedVolumeInfo{}, nil + }} + ts := newTestServer(t, "root-token", mgr, nil) + req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/volumes/data/grow", strings.NewReader(tc.body)) + if tc.token != "" { + req.Header.Set("Authorization", "Bearer "+tc.token) + } + req.Header.Set("If-Match", tc.match) + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != tc.status { + raw, _ := io.ReadAll(resp.Body) + t.Fatalf("status %d want %d, %s", resp.StatusCode, tc.status, raw) + } + }) + } +} + +func TestGrowVolumeHTTPErrorPrivacyAndPreconditions(t *testing.T) { + for _, tc := range []struct { + status int + code string + }{ + {404, "volume_not_found"}, {412, "volume_precondition_failed"}, {409, "volume_grow_conflict"}, {409, "volume_busy"}, + } { + t.Run(tc.code, func(t *testing.T) { + mgr := &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: &fakeManager{}}, grow: func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, &pool.VolumeError{Status: tc.status, Code: tc.code, Message: "volume unavailable"} + }} + ts := newTestServer(t, "root-token", mgr, nil) + req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/volumes/data/grow", strings.NewReader(growBody)) + req.Header.Set("Authorization", "Bearer root-token") + req.Header.Set("If-Match", `"vol_123"`) + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + var failure pool.VolumeError + if err := json.UnmarshalRead(resp.Body, &failure); err != nil || resp.StatusCode != tc.status || failure.Code != tc.code { + t.Fatalf("status %d failure %+v err %v", resp.StatusCode, failure, err) + } + }) + } +} + +func TestGrowVolumeHTTPDoesNotExposeInternalErrors(t *testing.T) { + mgr := &volumeGrowFake{volumeManagerFake: &volumeManagerFake{fakeManager: &fakeManager{}}, grow: func(context.Context, string, string, int64, string) (types.ManagedVolumeInfo, error) { + return types.ManagedVolumeInfo{}, errors.New("runtime /secret/backing pid 123 failed") + }} + ts := newTestServer(t, "root-token", mgr, nil) + req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/volumes/data/grow", strings.NewReader(growBody)) + req.Header.Set("Authorization", "Bearer root-token") + req.Header.Set("If-Match", `"vol_123"`) + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + raw, _ := io.ReadAll(resp.Body) + if resp.StatusCode != 503 || strings.Contains(string(raw), "/secret") || strings.Contains(string(raw), "pid 123") { + t.Fatalf("unsafe response %d %s", resp.StatusCode, raw) + } +} diff --git a/sandboxd/server/volume_mutation.go b/sandboxd/server/volume_mutation.go index 1e4f86f9..d3c52618 100644 --- a/sandboxd/server/volume_mutation.go +++ b/sandboxd/server/volume_mutation.go @@ -44,13 +44,7 @@ func (s *Server) handleSandboxVolumes(w http.ResponseWriter, r *http.Request) { code := http.StatusOK if err != nil { code, view.Error = http.StatusInternalServerError, "volume convergence failed; inspect the volume states before retrying" - conflict, isConflict := errors.AsType[*pool.VolumeMountConflict](err) - unsupported, isUnsupported := errors.AsType[*pool.LiveVolumeUnsupported](err) switch { - case isConflict: - code, view.Error = http.StatusConflict, conflict.Error() - case isUnsupported: - code, view.Error = http.StatusServiceUnavailable, unsupported.Error() case errors.Is(err, context.DeadlineExceeded): code, view.Error = http.StatusGatewayTimeout, "volume change timed out; inspect the volume states before retrying" case errors.Is(err, context.Canceled): diff --git a/sandboxd/types/api.go b/sandboxd/types/api.go index ac0cd6b4..e294add8 100644 --- a/sandboxd/types/api.go +++ b/sandboxd/types/api.go @@ -13,15 +13,37 @@ const ( // VolumeInfo is the caller-visible, host-path-free catalog projection. type VolumeInfo struct { - Name string `json:"name"` - DefaultMount string `json:"default_mount"` - SizeBytes int64 `json:"size_bytes"` - Available bool `json:"available"` - Nodes int `json:"nodes"` + Kind string `json:"kind,omitempty"` + ID string `json:"id,omitempty"` + Tenant string `json:"tenant,omitempty"` + Node string `json:"node,omitempty"` + State string `json:"state,omitempty"` + CreatedAt time.Time `json:"created_at,omitzero"` + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` // Writable reports whether the operator allows rw claims of this name. Writable bool `json:"writable,omitzero"` } +type ManagedVolumeInfo struct { + VolumeInfo + Holders []VolumeHolder `json:"holders,omitempty"` +} + +type VolumeHolder struct { + SandboxID string `json:"sandbox_id"` + Mode string `json:"mode"` + State string `json:"state"` +} + +type VolumeCreate struct { + SizeGiB int64 `json:"size_gib"` + Tenant string `json:"tenant,omitempty"` +} + // VolumePatch is the wire body of PATCH /v1/sandboxes/{id}/volumes. type VolumePatch struct { Add []Volume `json:"add,omitzero"` @@ -51,3 +73,8 @@ type SandboxVolumesResponse struct { func Seconds(n int) time.Duration { return time.Duration(min(max(int64(n), 0), math.MaxInt64/int64(time.Second))) * time.Second } + +// VolumeGrow is the wire body of POST /v1/volumes/{name}/grow: an absolute size. +type VolumeGrow struct { + SizeGiB int64 `json:"size_gib"` +} diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index 4cc8d307..a79b57c8 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -582,6 +582,9 @@ type Sandbox struct { // ReplicaPending is the checkpoint a replication round is about to register; the row may already name it. ReplicaPending string `json:"replica_pending,omitempty"` + // VolumeSleep journals a managed volume sleep in flight; written under Transition and the manager mutex. + VolumeSleep *VolumeSleep `json:"volume_sleep,omitempty"` + // FromCheckpoint names the checkpoint this sandbox branched from; empty otherwise. FromCheckpoint string `json:"from_checkpoint,omitempty"` // TemplateDigest identifies the promoted-template export this sandbox was cloned from. @@ -613,6 +616,36 @@ type Sandbox struct { Transition sync.Mutex `json:"-"` } +// VolumeSleep is the claim-row record of a managed volume hibernate or wake; a value is never edited in place. +type VolumeSleep struct { + Operation string `json:"operation"` + Phase SleepPhase `json:"phase"` + Snapshot string `json:"snapshot"` + SourceID string `json:"source_id,omitempty"` + VMID string `json:"vm_id"` + Generation uint64 `json:"generation"` + Hook VolumeHook `json:"hook"` +} + +// VolumeHook is the guest hook a managed sleep pins when it starts. +type VolumeHook struct { + HookArgv []string `json:"hook_argv"` + HookTimeoutSeconds int `json:"hook_timeout_seconds"` +} + +// SleepPhase is the managed sleep step whose intent the claim row last journaled. +type SleepPhase string + +// Managed sleep phases in journal order; aborting returns a failed preparation to running. +const ( + SleepPreparing SleepPhase = "preparing" + SleepAborting SleepPhase = "aborting" + SleepCapturing SleepPhase = "capturing" + SleepSleeping SleepPhase = "sleeping" + SleepWaking SleepPhase = "waking" + SleepResuming SleepPhase = "resuming" +) + // Touch stamps last data-plane activity; lock-free, called on the relay hot path. func (s *Sandbox) Touch() { s.lastActivity.Store(time.Now().UnixNano()) } @@ -721,6 +754,7 @@ type VMConfig struct { // VMDisk is one hot-attached disk that `cocoon vm inspect` reports. type VMDisk struct { Name string `json:"name"` + Path string `json:"path"` ReadOnly bool `json:"readonly,omitzero"` } @@ -731,12 +765,15 @@ type VMDevices struct { // VMRecord is the subset of cocoon's VM records the control plane reads. type VMRecord struct { - State string `json:"state"` - PID int `json:"pid"` - VsockSocket string `json:"vsock_socket"` - NetworkConfigs []VMNetConfig `json:"network_configs,omitempty"` - Config VMConfig `json:"config"` - AttachedDevices VMDevices `json:"attached_devices,omitzero"` + ID string `json:"id"` + TransitionGeneration uint64 `json:"transition_generation"` + LastTransitionReason string `json:"last_transition_reason"` + State string `json:"state"` + PID int `json:"pid"` + VsockSocket string `json:"vsock_socket"` + NetworkConfigs []VMNetConfig `json:"network_configs,omitempty"` + Config VMConfig `json:"config"` + AttachedDevices VMDevices `json:"attached_devices,omitzero"` } // TapDevice returns the first NIC's host tap; empty when the record carries none. @@ -765,6 +802,12 @@ func TemplateGossipHash(keyHash, tenant string) string { return hex.EncodeToString(sum[:16]) } +// ManagedVolumeGossipName is the mesh key a node advertises for a tenant's managed volume; it never parses as a volume name. +func ManagedVolumeGossipName(tenant, name string) string { + sum := sha256.Sum256([]byte("volume|" + tenant + "|" + name)) + return "managed/" + hex.EncodeToString(sum[:16]) +} + // ValidVolumeName reports whether name is a legal cocoon data-disk serial. func ValidVolumeName(name string) bool { return VolumeNameRe.MatchString(name) && !strings.HasPrefix(name, "cocoon-") diff --git a/sandboxd/types/types_test.go b/sandboxd/types/types_test.go index 982763b2..34dd6e8c 100644 --- a/sandboxd/types/types_test.go +++ b/sandboxd/types/types_test.go @@ -168,3 +168,13 @@ func pairsOf(n int, value string) Metadata { } return md } + +func TestVolumeCatalogWireShape(t *testing.T) { + catalog := VolumeInfo{Name: "data", DefaultMount: "/volumes/data", SizeBytes: 42, Available: true, Nodes: 1} + const baseline = `{"name":"data","default_mount":"/volumes/data","size_bytes":42,"available":true,"nodes":1}` + for _, v := range []any{catalog, ManagedVolumeInfo{VolumeInfo: catalog}} { + if raw, err := json.Marshal(v); err != nil || string(raw) != baseline { + t.Fatalf("catalog changed from baseline: %s %v", raw, err) + } + } +} diff --git a/sdk/go/client.go b/sdk/go/client.go index ccb932fd..80f9b5f4 100644 --- a/sdk/go/client.go +++ b/sdk/go/client.go @@ -19,7 +19,8 @@ import ( ) const ( - idleConnsPerHost = 100 + idleConnsPerHost = 100 + maxErrorBodyBytes = 1 << 20 templateQueryParam = "template" netQueryParam = "net" @@ -71,7 +72,10 @@ type renewResponse struct { } type errorResponse struct { - Error string `json:"error"` + Error string `json:"error"` + Code string `json:"code"` + Volume string `json:"volume"` + Holders []VolumeHolder `json:"holders"` } // APIError is a non-2xx control-plane reply. @@ -442,8 +446,16 @@ func encodeBody(verb string, v any) ([]byte, error) { } func apiError(verb string, resp *http.Response) error { + body, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBodyBytes+1)) var er errorResponse - _ = json.UnmarshalRead(io.LimitReader(resp.Body, 4096), &er) - _, _ = io.Copy(io.Discard, resp.Body) - return &APIError{Verb: verb, Status: resp.StatusCode, Message: er.Error} + if len(body) > maxErrorBodyBytes { + er.Error = fmt.Sprintf("error response body exceeds %d bytes", maxErrorBodyBytes) + } else { + _ = json.Unmarshal(body, &er) + } + base := &APIError{Verb: verb, Status: resp.StatusCode, Message: er.Error} + if er.Code != "" || len(er.Holders) != 0 { + return &ManagedVolumeError{APIError: base, Code: er.Code, Volume: er.Volume, Holders: er.Holders} + } + return base } diff --git a/sdk/go/client_error_test.go b/sdk/go/client_error_test.go new file mode 100644 index 00000000..d523a5c4 --- /dev/null +++ b/sdk/go/client_error_test.go @@ -0,0 +1,110 @@ +package sandbox + +import ( + "encoding/json/v2" + "errors" + "fmt" + "io" + "net/http" + "reflect" + "strings" + "testing" +) + +func TestAPIErrorBoundsResponseBody(t *testing.T) { + const limit = 1 << 20 + const prefix = `{"error":"` + const suffix = `"}` + oversizeMessage := fmt.Sprintf("error response body exceeds %d bytes", limit) + for _, tt := range []struct { + name string + body string + terminalErr error + wantMessage string + wantBytes int + }{ + {name: "normal JSON", body: `{"error":"slow down"}`, wantMessage: "slow down", wantBytes: 21}, + {name: "invalid JSON", body: "not JSON", wantBytes: 8}, + { + name: "exact limit", body: prefix + strings.Repeat("x", limit-len(prefix)-len(suffix)) + suffix, + wantMessage: strings.Repeat("x", limit-len(prefix)-len(suffix)), wantBytes: limit, + }, + { + name: "oversized valid JSON", body: prefix + strings.Repeat("x", 2*limit) + suffix, + wantMessage: oversizeMessage, wantBytes: limit + 1, + }, + { + name: "oversized invalid JSON", body: strings.Repeat("x", 2*limit), + wantMessage: oversizeMessage, wantBytes: limit + 1, + }, + { + name: "oversized trailing bytes", body: `{"error":"held","code":"volume_busy","holders":[{"sandbox_id":"sb_1"}]}` + strings.Repeat(" ", 2*limit), + wantMessage: oversizeMessage, wantBytes: limit + 1, + }, + { + name: "error after cap", body: strings.Repeat("x", limit+1), terminalErr: errors.New("reader failed after cap"), + wantMessage: oversizeMessage, wantBytes: limit + 1, + }, + } { + t.Run(tt.name, func(t *testing.T) { + body := &countedErrorBody{reader: strings.NewReader(tt.body), terminalErr: tt.terminalErr} + err := apiError("claim", &http.Response{StatusCode: http.StatusTooManyRequests, Body: io.NopCloser(body)}) + got, ok := errors.AsType[*APIError](err) + if !ok || got.Verb != "claim" || got.Status != http.StatusTooManyRequests { + t.Fatalf("apiError = %v, want claim 429 APIError", err) + } + if body.bytes != tt.wantBytes { + t.Errorf("read %d response bytes, want %d", body.bytes, tt.wantBytes) + } + if body.terminalErrors != 0 { + t.Errorf("read beyond body cap and observed %d terminal errors", body.terminalErrors) + } + if got.Message != tt.wantMessage { + t.Errorf("message length %d, want %d; matched=%v", len(got.Message), len(tt.wantMessage), got.Message == tt.wantMessage) + } + if _, managed := errors.AsType[*ManagedVolumeError](err); managed { + t.Errorf("returned partially decoded managed error %v", err) + } + }) + } +} + +func TestAPIErrorReadsCompleteManagedHolders(t *testing.T) { + holders := make([]VolumeHolder, 100) + for i := range holders { + holders[i] = VolumeHolder{SandboxID: fmt.Sprintf("sb_%032d", i), Mode: "ro", State: "claimed"} + } + raw, err := json.Marshal(errorResponse{Error: "volume held", Code: "volume_busy", Volume: "data", Holders: holders}) + if err != nil { + t.Fatal(err) + } + if len(raw) <= 4096 { + t.Fatalf("holder response has %d bytes, want more than former 4096-byte limit", len(raw)) + } + body := &countedErrorBody{reader: strings.NewReader(string(raw))} + err = apiError("claim", &http.Response{StatusCode: http.StatusConflict, Body: io.NopCloser(body)}) + got, ok := errors.AsType[*ManagedVolumeError](err) + if !ok || got.Status != http.StatusConflict || got.Message != "volume held" || got.Code != "volume_busy" || got.Volume != "data" { + t.Fatalf("apiError = %v, want intact managed volume conflict", err) + } + if body.bytes != len(raw) || !reflect.DeepEqual(got.Holders, holders) { + t.Fatalf("read %d/%d bytes and received %d/%d intact holders", body.bytes, len(raw), len(got.Holders), len(holders)) + } +} + +type countedErrorBody struct { + reader *strings.Reader + terminalErr error + bytes int + terminalErrors int +} + +func (r *countedErrorBody) Read(p []byte) (int, error) { + n, err := r.reader.Read(p) + r.bytes += n + if err == io.EOF && r.terminalErr != nil { + r.terminalErrors++ + return n, r.terminalErr + } + return n, err +} diff --git a/sdk/go/managed_volume.go b/sdk/go/managed_volume.go new file mode 100644 index 00000000..3d219532 --- /dev/null +++ b/sdk/go/managed_volume.go @@ -0,0 +1,129 @@ +package sandbox + +import ( + "bytes" + "context" + "encoding/json/v2" + "fmt" + "io" + "net/http" + "net/url" +) + +type ManagedVolumeInfo struct { + VolumeInfo + Holders []VolumeHolder `json:"holders,omitempty"` +} + +type ManagedVolumeError struct { + *APIError + Code string + Volume string + Holders []VolumeHolder +} + +func (e *ManagedVolumeError) Unwrap() error { return e.APIError } + +// VolumeCreate requests a raw managed disk owned by Tenant. A tenant token may leave Tenant empty. +type VolumeCreate struct { + SizeGiB int64 `json:"size_gib"` + Tenant string `json:"tenant,omitempty"` +} + +type volumeResponse struct { + Volume *ManagedVolumeInfo `json:"volume"` +} + +// CreateVolume creates a managed disk or returns the same disk for an identical retry. +func (c *Client) CreateVolume(ctx context.Context, name string, create VolumeCreate) (*ManagedVolumeInfo, error) { + body, err := encodeBody("create volume", create) + if err != nil { + return nil, err + } + resp, err := c.roundTrip(ctx, http.MethodPut, c.addr, volumePath(name), bytes.NewReader(body), c.apiToken) + if err != nil { + return nil, err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK { + return nil, apiError("create volume", resp) + } + return decodeVolumeResponse("create volume", resp.Body) +} + +// GetVolume inspects one managed disk visible to this token. +func (c *Client) GetVolume(ctx context.Context, name string) (*ManagedVolumeInfo, error) { + resp, err := c.roundTrip(ctx, http.MethodGet, c.addr, volumePath(name), nil, c.apiToken) + if err != nil { + return nil, err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return nil, apiError("get volume", resp) + } + return decodeVolumeResponse("get volume", resp.Body) +} + +// GrowVolume grows an unheld managed disk to sizeGiB; repeating the current size is a no-op. +func (c *Client) GrowVolume(ctx context.Context, name, id string, sizeGiB int64) (*ManagedVolumeInfo, error) { + body, err := encodeBody("grow volume", struct { + SizeGiB int64 `json:"size_gib"` + }{sizeGiB}) + if err != nil { + return nil, err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.requestURL(c.addr, volumePath(name)+"/grow"), bytes.NewReader(body)) + if err != nil { + return nil, err + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("If-Match", `"`+id+`"`) + if c.apiToken != "" { + req.Header.Set("Authorization", "Bearer "+c.apiToken) + } + resp, err := c.hc.Do(req) + if err != nil { + return nil, err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return nil, apiError("grow volume", resp) + } + return decodeVolumeResponse("grow volume", resp.Body) +} + +// DeleteVolume deletes exactly id, including when an earlier response was lost. +func (c *Client) DeleteVolume(ctx context.Context, name, id string) error { + req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.requestURL(c.addr, volumePath(name)), nil) + if err != nil { + return err + } + req.Header.Set("If-Match", `"`+id+`"`) + if c.apiToken != "" { + req.Header.Set("Authorization", "Bearer "+c.apiToken) + } + resp, err := c.hc.Do(req) + if err != nil { + return err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusNoContent { + return apiError("delete volume", resp) + } + return nil +} + +func volumePath(name string) string { + return "/v1/volumes/" + url.PathEscape(name) +} + +func decodeVolumeResponse(verb string, body io.Reader) (*ManagedVolumeInfo, error) { + var reply volumeResponse + if err := json.UnmarshalRead(body, &reply); err != nil { + return nil, fmt.Errorf("decode %s response: %w", verb, err) + } + if reply.Volume == nil { + return nil, fmt.Errorf("decode %s response: missing volume", verb) + } + return reply.Volume, nil +} diff --git a/sdk/go/managed_volume_test.go b/sdk/go/managed_volume_test.go new file mode 100644 index 00000000..bb9801bf --- /dev/null +++ b/sdk/go/managed_volume_test.go @@ -0,0 +1,186 @@ +package sandbox + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" + "time" +) + +func TestManagedVolumeRequests(t *testing.T) { + want := ManagedVolumeInfo{Name: "data", Kind: "managed", ID: "vol_123", Tenant: "t1", Node: "node-1", State: "ready", CreatedAt: time.Date(2026, 10, 19, 0, 0, 0, 0, time.UTC), SizeBytes: 20 << 30, Available: true, Writable: true, Holders: []VolumeHolder{{SandboxID: "sb_1", Mode: "rw", State: "attached"}}} + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.URL.Path != "/v1/volumes/data" || r.Header.Get("Authorization") != "Bearer token" { + t.Errorf("request = %s %s, authorization %q", r.Method, r.URL.Path, r.Header.Get("Authorization")) + } + switch r.Method { + case http.MethodPut: + var body map[string]any + if err := json.UnmarshalRead(r.Body, &body); err != nil { + t.Errorf("decode create: %v", err) + } + if body["size_gib"] != float64(20) || body["tenant"] != "t1" { + t.Errorf("create body = %v", body) + } + if requests == 1 { + w.WriteHeader(http.StatusCreated) + } + _ = json.MarshalWrite(w, volumeResponse{Volume: &want}) + case http.MethodGet: + w.Header().Set("ETag", `"vol_123"`) + _ = json.MarshalWrite(w, volumeResponse{Volume: &want}) + case http.MethodDelete: + if got := r.Header.Get("If-Match"); got != `"vol_123"` { + t.Errorf("If-Match = %q", got) + } + w.WriteHeader(http.StatusNoContent) + default: + t.Errorf("unexpected method %s", r.Method) + } + })) + t.Cleanup(ts.Close) + c := testClient(t, ts, WithAPIToken("token")) + for range 2 { + got, err := c.CreateVolume(t.Context(), "data", VolumeCreate{SizeGiB: 20, Tenant: "t1"}) + if err != nil || !reflect.DeepEqual(got, &want) { + t.Fatalf("create = %+v, %v", got, err) + } + } + got, err := c.GetVolume(t.Context(), "data") + if err != nil || !reflect.DeepEqual(got, &want) { + t.Fatalf("get = %+v, %v", got, err) + } + if err := c.DeleteVolume(t.Context(), "data", want.ID); err != nil { + t.Fatal(err) + } + if err := c.DeleteVolume(t.Context(), "data", want.ID); err != nil { + t.Fatal(err) + } + if requests != 5 { + t.Fatalf("requests = %d, want 5", requests) + } +} + +func TestManagedVolumeFailures(t *testing.T) { + statuses := []int{400, 401, 403, 404, 409, 412, 428, 503} + for _, status := range statuses { + t.Run(http.StatusText(status), func(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(status) + _ = json.MarshalWrite(w, map[string]any{"error": "volume unavailable", "code": "volume_busy", "volume": "data", "holders": []map[string]string{{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring", "host_path": "/secret"}}}) + })) + t.Cleanup(ts.Close) + _, err := testClient(t, ts).GetVolume(t.Context(), "data") + var apiErr *ManagedVolumeError + if !errors.As(err, &apiErr) || apiErr.Status != status || apiErr.Code != "volume_busy" || apiErr.Volume != "data" || !reflect.DeepEqual(apiErr.Holders, []VolumeHolder{{SandboxID: "sb_1", Mode: "rw", State: "retiring"}}) { + t.Fatalf("error = %#v", err) + } + if strings.Contains(apiErr.Error(), "/secret") { + t.Fatal("error exposed a host path") + } + }) + } +} + +func TestManagedVolumeRejectsReplyWithoutVolume(t *testing.T) { + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + requests++ + _, _ = w.Write([]byte(`{}`)) + })) + t.Cleanup(ts.Close) + c := testClient(t, ts) + if _, err := c.GetVolume(t.Context(), "data"); err == nil { + t.Fatal("accepted missing response volume") + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + if _, err := c.GetVolume(ctx, "data"); !errors.Is(err, context.Canceled) { + t.Fatalf("cancellation = %v", err) + } + if requests != 1 { + t.Fatalf("canceled request reached server, count %d", requests) + } +} + +func TestCreateVolumeForwardsInvalidName(t *testing.T) { + const refusal = "invalid volume name or size_gib (1 through 1024)" + type request struct { + path string + size float64 + } + var got []request + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var body map[string]any + if err := json.UnmarshalRead(r.Body, &body); err != nil { + t.Errorf("decode create: %v", err) + } + size, _ := body["size_gib"].(float64) + got = append(got, request{r.URL.EscapedPath(), size}) + w.WriteHeader(http.StatusBadRequest) + _ = json.MarshalWrite(w, map[string]string{"error": refusal}) + })) + t.Cleanup(ts.Close) + c := testClient(t, ts) + cases := []struct { + name string + size int64 + want request + }{ + {"bad/name", 20, request{"/v1/volumes/bad%2Fname", 20}}, + {"cocoon-data", 20, request{"/v1/volumes/cocoon-data", 20}}, + {"data", 0, request{"/v1/volumes/data", 0}}, + } + for _, tc := range cases { + _, err := c.CreateVolume(t.Context(), tc.name, VolumeCreate{SizeGiB: tc.size}) + var apiErr *APIError + if !errors.As(err, &apiErr) || apiErr.Status != http.StatusBadRequest || apiErr.Message != refusal { + t.Errorf("create %q size %d = %v, want the server's 400", tc.name, tc.size, err) + } + } + want := make([]request, len(cases)) + for i, tc := range cases { + want[i] = tc.want + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("server saw %+v, want %+v", got, want) + } +} + +func TestManagedVolumeConflictPreservesAllHolders(t *testing.T) { + holders := make([]VolumeHolder, 100) + for i := range holders { + holders[i] = VolumeHolder{SandboxID: fmt.Sprintf("sb_%032d", i), Mode: "ro", State: "claimed"} + } + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusConflict) + _ = json.MarshalWrite(w, map[string]any{"error": "volume held", "code": "volume_busy", "volume": "data", "holders": holders}) + })) + t.Cleanup(ts.Close) + _, err := testClient(t, ts).GetVolume(t.Context(), "data") + var conflict *ManagedVolumeError + if !errors.As(err, &conflict) || conflict.Status != http.StatusConflict || conflict.Code != "volume_busy" { + t.Fatalf("expected typed volume conflict, got %#v", err) + } + if !reflect.DeepEqual(conflict.Holders, holders) { + t.Fatalf("conflict returned %d holders, want all %d with intact fields", len(conflict.Holders), len(holders)) + } +} + +func TestCatalogVolumeInfoWireShape(t *testing.T) { + catalog := VolumeInfo{Name: "data", DefaultMount: "/volumes/data", SizeBytes: 42, Available: true, Nodes: 1} + const baseline = `{"name":"data","default_mount":"/volumes/data","size_bytes":42,"available":true,"nodes":1}` + for _, v := range []any{catalog, ManagedVolumeInfo{VolumeInfo: catalog}} { + if raw, err := json.Marshal(v); err != nil || string(raw) != baseline { + t.Fatalf("catalog changed from baseline: %s %v", raw, err) + } + } +} diff --git a/sdk/go/options.go b/sdk/go/options.go index 20b75401..0d04873a 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -39,12 +39,24 @@ type Volume struct { // VolumeInfo describes one visible fleet catalog entry. type VolumeInfo struct { - Name string `json:"name"` - DefaultMount string `json:"default_mount"` - SizeBytes int64 `json:"size_bytes"` - Available bool `json:"available"` - Nodes int `json:"nodes"` - Writable bool `json:"writable,omitzero"` + Kind string `json:"kind,omitempty"` + ID string `json:"id,omitempty"` + Tenant string `json:"tenant,omitempty"` + Node string `json:"node,omitempty"` + State string `json:"state,omitempty"` + CreatedAt time.Time `json:"created_at,omitzero"` + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` + Writable bool `json:"writable,omitzero"` +} + +type VolumeHolder struct { + SandboxID string `json:"sandbox_id"` + Mode string `json:"mode"` + State string `json:"state"` } // EnvInject makes the node's egress proxy send a host-only entry's value to Hosts, exact or "*." suffix, as Header or as the Query parameter, where the guest sent a set Placeholder. diff --git a/sdk/go/volume_grow_test.go b/sdk/go/volume_grow_test.go new file mode 100644 index 00000000..2510288a --- /dev/null +++ b/sdk/go/volume_grow_test.go @@ -0,0 +1,66 @@ +package sandbox + +import ( + "context" + "encoding/json/v2" + "errors" + "net/http" + "net/http/httptest" + "reflect" + "testing" + "time" +) + +func TestGrowVolumeRequest(t *testing.T) { + want := ManagedVolumeInfo{Name: "data", Kind: "managed", ID: "vol_123", Tenant: "t1", Node: "node-1", State: "ready", CreatedAt: time.Date(2026, 10, 19, 0, 0, 0, 0, time.UTC), SizeBytes: 40 << 30} + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.URL.Path != "/v1/volumes/data/grow" || r.Method != http.MethodPost || r.Header.Get("If-Match") != `"vol_123"` || r.Header.Get("Authorization") != "Bearer token" || r.Header.Get("Content-Type") != "application/json" { + t.Errorf("request = %s %s %+v", r.Method, r.URL.Path, r.Header) + } + var body map[string]any + if err := json.UnmarshalRead(r.Body, &body); err != nil || !reflect.DeepEqual(body, map[string]any{"size_gib": float64(40)}) { + t.Errorf("body = %v, %v", body, err) + } + _ = json.MarshalWrite(w, volumeResponse{Volume: &want}) + })) + t.Cleanup(ts.Close) + got, err := testClient(t, ts, WithAPIToken("token")).GrowVolume(t.Context(), "data", "vol_123", 40) + if err != nil || !reflect.DeepEqual(got, &want) || requests != 1 { + t.Fatalf("grow = %+v, %v, requests %d", got, err, requests) + } +} + +func TestGrowVolumeCanceledBeforeSending(t *testing.T) { + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { requests++; w.WriteHeader(503) })) + t.Cleanup(ts.Close) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + if _, err := testClient(t, ts).GrowVolume(ctx, "data", "vol_123", 40); !errors.Is(err, context.Canceled) { + t.Fatalf("cancellation = %v", err) + } + if requests != 0 { + t.Fatalf("canceled requests sent = %d", requests) + } +} + +func TestGrowVolumeErrors(t *testing.T) { + for _, status := range []int{http.StatusAccepted, 400, 401, 404, 409, 412, 428, 503} { + t.Run(http.StatusText(status), func(t *testing.T) { + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + requests++ + w.WriteHeader(status) + _ = json.MarshalWrite(w, map[string]string{"error": "growth unavailable"}) + })) + t.Cleanup(ts.Close) + _, err := testClient(t, ts).GrowVolume(t.Context(), "data", "vol_123", 40) + var failure *APIError + if !errors.As(err, &failure) || failure.Status != status || failure.Message != "growth unavailable" || requests != 1 { + t.Fatalf("error = %#v, requests %d", err, requests) + } + }) + } +} diff --git a/sdk/python/cocoonsandbox/client.py b/sdk/python/cocoonsandbox/client.py index aa3c8fd1..ef24fabb 100644 --- a/sdk/python/cocoonsandbox/client.py +++ b/sdk/python/cocoonsandbox/client.py @@ -133,6 +133,44 @@ def volumes(self) -> list[dict[str, Any]]: reply = self._request(self.addr, "GET", "/v1/volumes", None, "list volumes") return [dict(volume) for volume in reply.get("volumes") or []] + def create_volume(self, name: str, size_gib: int, tenant: str = "") -> dict[str, Any]: + """Creates a raw managed disk or returns it for an identical retry.""" + body: dict[str, Any] = {"size_gib": size_gib} + if tenant: + body["tenant"] = tenant + reply = self._request(self.addr, "PUT", _volume_path(name), body, "create volume", expected_status=(200, 201)) + return dict(require_field(reply, "volume", "create volume")) + + def get_volume(self, name: str) -> dict[str, Any]: + """Inspects one managed disk visible to this token.""" + reply = self._request(self.addr, "GET", _volume_path(name), None, "get volume", expected_status=(200,)) + return dict(require_field(reply, "volume", "get volume")) + + def delete_volume(self, name: str, resource_id: str) -> None: + """Deletes exactly resource_id, including when an earlier response was lost.""" + self._request( + self.addr, + "DELETE", + _volume_path(name), + None, + "delete volume", + headers={"If-Match": f'"{resource_id}"'}, + expected_status=(204,), + ) + + def grow_volume(self, name: str, resource_id: str, size_gib: int) -> dict[str, Any]: + """Grows an unheld managed disk to size_gib; repeating the current size is a no-op.""" + reply = self._request( + self.addr, + "POST", + _volume_path(name) + "/grow", + {"size_gib": size_gib}, + "grow volume", + headers={"If-Match": f'"{resource_id}"'}, + expected_status=(200,), + ) + return dict(require_field(reply, "volume", "grow volume")) + def info(self) -> dict[str, Any]: """The node's pool/claim counters, as served by GET /v1/info.""" return self._request(self.addr, "GET", "/v1/info", None, "info") @@ -198,11 +236,13 @@ def _request( bearer: str = "", timeout: float = 0.0, deadline: float | None = None, + headers: dict[str, str] | None = None, + expected_status: tuple[int, ...] | None = None, ) -> dict[str, Any]: timeout = remaining_timeout(timeout or self.timeout, deadline, verb) data = json.dumps(body).encode() if body is not None else None url = urllib.parse.urlsplit(addr + path if "://" in addr else f"{self._scheme}://{addr}{path}") - headers = {"Content-Type": "application/json"} if data is not None else {} + headers = {**(headers or {}), **({"Content-Type": "application/json"} if data is not None else {})} token = bearer or self.api_token if token: headers["Authorization"] = f"Bearer {token}" @@ -218,6 +258,8 @@ def _request( except (ValueError, TypeError): reply = None raise APIError(verb, status, _error_message(raw), reply=reply if isinstance(reply, dict) else None) + if expected_status is not None and status not in expected_status: + raise APIError(verb, status, "unexpected response status") if not raw: return {} try: @@ -347,3 +389,7 @@ def run(addr: str) -> None: return cast(T, value) last_error = error raise cast(Exception, last_error) + + +def _volume_path(name: str) -> str: + return "/v1/volumes/" + urllib.parse.quote(name, safe="") diff --git a/sdk/python/cocoonsandbox/errors.py b/sdk/python/cocoonsandbox/errors.py index 4e1ed4dc..314b2fe5 100644 --- a/sdk/python/cocoonsandbox/errors.py +++ b/sdk/python/cocoonsandbox/errors.py @@ -25,6 +25,14 @@ def __init__(self, verb: str, status: int, message: str, *, reply: dict[str, Any self.status = status self.message = message self.reply = reply + self.code = (reply or {}).get("code", "") + self.volume = (reply or {}).get("volume", "") + holders = (reply or {}).get("holders") + self.holders = [ + {key: entry[key] for key in ("sandbox_id", "mode", "state") if isinstance(entry.get(key), str)} + for entry in (holders if isinstance(holders, list) else []) + if isinstance(entry, dict) + ] class VolumeError(APIError): diff --git a/sdk/python/cocoonsandbox/sandbox.py b/sdk/python/cocoonsandbox/sandbox.py index 1a83fa86..0b21e56e 100644 --- a/sdk/python/cocoonsandbox/sandbox.py +++ b/sdk/python/cocoonsandbox/sandbox.py @@ -77,7 +77,7 @@ def set_volumes( self, volumes: list[str | Mapping[str, str] | Volume], *, timeout: float | None = None ) -> SandboxVolumesResponse: """Replaces the desired set and converges; an explicit empty list clears mutable mounts.""" - if volumes is None or isinstance(volumes, str): + if volumes is None or isinstance(volumes, (str, Mapping)): raise TypeError("set volumes requires an explicit list; use [] to clear volumes") return self._volumes_request( "PUT", {"volumes": [_volume_body(volume, True) for volume in volumes]}, "set volumes", timeout @@ -91,7 +91,7 @@ def patch_volumes( timeout: float | None = None, ) -> SandboxVolumesResponse: """Adds or removes mounted read-only entries by name; omitted arguments leave the set unchanged.""" - if isinstance(add, str) or isinstance(remove, str): + if isinstance(add, (str, Mapping)) or isinstance(remove, str): raise TypeError("patch volumes takes lists of volumes and names") body: dict[str, Any] = {} if add is not None: diff --git a/sdk/python/cocoonsandbox/template.py b/sdk/python/cocoonsandbox/template.py index 5e468aa0..7d123b3f 100644 --- a/sdk/python/cocoonsandbox/template.py +++ b/sdk/python/cocoonsandbox/template.py @@ -59,6 +59,8 @@ def _claim_body( mount: bool = True, claim_ref: str = "", ) -> dict[str, Any]: + if isinstance(volumes, (str, Mapping)): + raise TypeError("volumes takes a list of names or mappings") claim: dict[str, Any] = {"template": template} if net: claim["net"] = net diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 90b799dc..1e3c8f7f 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -84,6 +84,16 @@ def test_claim_rejects_invalid_volumes(node, volumes, match): Client(node).new("rt:24.04", volumes=volumes) +@pytest.mark.parametrize("volumes", ["work", {"name": "work"}]) +def test_claim_rejects_a_bare_string_or_mapping_volume_list(node, volumes): + seen = recording_claim({"id": "sb_1", "token": "tok"}) + with pytest.raises(TypeError): + Client(node).new("rt:24.04", volumes=volumes) + with pytest.raises(TypeError): + Template(Client(node), node, "task:v1", "none", "small").new(volumes=volumes) + assert seen == [] + + def test_claim_sends_volume_mode_rw(node): seen = recording_claim( {"id": "sb_1", "token": "tok", "volumes": [{"name": "scratch", "mount": "/data", "mode": "rw"}]} diff --git a/sdk/python/tests/test_managed_volumes.py b/sdk/python/tests/test_managed_volumes.py new file mode 100644 index 00000000..d7366fd6 --- /dev/null +++ b/sdk/python/tests/test_managed_volumes.py @@ -0,0 +1,173 @@ +import threading +from http.server import HTTPServer + +import pytest +from conftest import FakeNode + +from cocoonsandbox import APIError, Client + + +class VolumeNode(FakeNode): + def do_PUT(self): + self._dispatch("PUT") + + +@pytest.fixture +def node(): + server = HTTPServer(("127.0.0.1", 0), VolumeNode) + FakeNode.routes = {} + threading.Thread(target=server.serve_forever, daemon=True).start() + yield f"127.0.0.1:{server.server_port}" + server.shutdown() + + +def test_create_inspect_and_delete_use_immutable_identity(node): + volume = { + "name": "data", + "kind": "managed", + "id": "vol_123", + "tenant": "t1", + "node": "node-1", + "state": "ready", + "created_at": "2026-10-19T00:00:00Z", + "size_bytes": 20 << 30, + "available": True, + "holders": [{"sandbox_id": "sb_1", "mode": "rw", "state": "attached"}], + } + seen = [] + + def create(body, path): + seen.append(("PUT", body, dict(FakeNode.last_headers))) + return (201 if len(seen) == 1 else 200), {"volume": volume} + + def inspect(body, path): + seen.append(("GET", body, dict(FakeNode.last_headers))) + return 200, {"volume": volume} + + def delete(body, path): + seen.append(("DELETE", body, dict(FakeNode.last_headers))) + return 204, {} + + path = "/v1/volumes/data" + FakeNode.routes[("PUT", path)] = create + FakeNode.routes[("GET", path)] = inspect + FakeNode.routes[("DELETE", path)] = delete + client = Client(node, api_token="token") + assert client.create_volume("data", 20, tenant="t1") == volume + assert client.create_volume("data", 20, tenant="t1") == volume + assert client.get_volume("data") == volume + client.delete_volume("data", volume["id"]) + client.delete_volume("data", volume["id"]) + assert [request[0] for request in seen] == ["PUT", "PUT", "GET", "DELETE", "DELETE"] + assert all(request[2]["Authorization"] == "Bearer token" for request in seen) + assert seen[0][1] == {"size_gib": 20, "tenant": "t1"} + assert seen[3][2]["If-Match"] == seen[4][2]["If-Match"] == '"vol_123"' + + +@pytest.mark.parametrize("status", [400, 401, 403, 404, 409, 412, 428, 503]) +def test_management_error_keeps_safe_details(node, status): + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: ( + status, + { + "error": "volume unavailable", + "code": "volume_busy", + "volume": "data", + "holders": [{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring", "host_path": "/secret"}], + }, + ) + with pytest.raises(APIError) as raised: + Client(node).get_volume("data") + error = raised.value + assert error.status == status and error.message == "volume unavailable" + assert error.code == "volume_busy" and error.volume == "data" + assert error.holders == [{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring"}] + assert "/secret" not in str(error) and "/secret" not in repr(error.holders) + + +def test_incomplete_reply(node): + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: (200, {}) + with pytest.raises(APIError, match="reply without volume"): + Client(node).get_volume("data") + + +def test_create_volume_forwards_invalid_name(node): + refusal = "invalid volume name or size_gib (1 through 1024)" + seen = [] + + def refuse(body, path): + seen.append((path, body)) + return 400, {"error": refusal} + + for path in ("/v1/volumes/bad%2Fname", "/v1/volumes/cocoon-data", "/v1/volumes/data"): + FakeNode.routes[("PUT", path)] = refuse + client = Client(node) + for name, size in (("bad/name", 20), ("cocoon-data", 20), ("data", 0)): + with pytest.raises(APIError) as raised: + client.create_volume(name, size) + assert raised.value.status == 400 and raised.value.message == refusal + assert seen == [ + ("/v1/volumes/bad%2Fname", {"size_gib": 20}), + ("/v1/volumes/cocoon-data", {"size_gib": 20}), + ("/v1/volumes/data", {"size_gib": 0}), + ] + + +def test_management_timeout(black_hole): + with pytest.raises(APIError) as raised: + Client(black_hole, timeout=0.05).get_volume("data") + assert raised.value.status == 0 + + +@pytest.mark.parametrize("status", [200, 202, 302]) +def test_delete_requires_confirmed_no_content(node, status): + FakeNode.routes[("DELETE", "/v1/volumes/data")] = lambda body, path: (status, {}) + with pytest.raises(APIError) as raised: + Client(node).delete_volume("data", "vol_123") + assert raised.value.status == status + + +@pytest.mark.parametrize("holders", [1, "invalid", {}, None]) +def test_malformed_holder_container_preserves_api_error(node, holders): + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: (409, {"error": "busy", "holders": holders}) + with pytest.raises(APIError) as raised: + Client(node).get_volume("data") + assert raised.value.status == 409 and raised.value.message == "busy" + assert raised.value.holders == [] + assert raised.value.reply == {"error": "busy", "holders": holders} + + +def test_grow_volume_sends_absolute_size(node): + volume = {"name": "data", "kind": "managed", "id": "vol_123", "state": "ready", "size_bytes": 40 << 30} + seen = [] + + def grow(body, path): + seen.append((body, dict(FakeNode.last_headers))) + return 200, {"volume": volume} + + FakeNode.routes[("POST", "/v1/volumes/data/grow")] = grow + assert Client(node, api_token="token").grow_volume("data", "vol_123", 40) == volume + assert len(seen) == 1 + body, headers = seen[0] + assert body == {"size_gib": 40} + assert headers["If-Match"] == '"vol_123"' and headers["Authorization"] == "Bearer token" + + +@pytest.mark.parametrize("status", [400, 401, 404, 409, 412, 428, 503]) +def test_grow_volume_preserves_errors_without_retry(node, status): + calls = [] + + def grow(body, path): + calls.append(body) + return status, {"error": "growth unavailable"} + + FakeNode.routes[("POST", "/v1/volumes/data/grow")] = grow + with pytest.raises(APIError) as raised: + Client(node).grow_volume("data", "vol_123", 40) + assert raised.value.status == status and raised.value.message == "growth unavailable" + assert len(calls) == 1 + + +def test_grow_volume_timeout(black_hole): + with pytest.raises(APIError) as raised: + Client(black_hole, timeout=0.05).grow_volume("data", "vol_123", 40) + assert raised.value.status == 0 diff --git a/sdk/python/tests/test_volumes.py b/sdk/python/tests/test_volumes.py index 3226b82d..5b834699 100644 --- a/sdk/python/tests/test_volumes.py +++ b/sdk/python/tests/test_volumes.py @@ -159,8 +159,10 @@ def test_incomplete_volume_view_preserves_handle(spawn_node, status): lambda sb: sb.set_volumes("data"), lambda sb: sb.patch_volumes(add="data"), lambda sb: sb.patch_volumes(remove="old"), + lambda sb: sb.set_volumes({"name": "data"}), + lambda sb: sb.patch_volumes(add={"name": "data"}), ], ) -def test_volume_methods_reject_a_bare_string(black_hole, call): +def test_volume_methods_reject_a_bare_string_or_mapping(black_hole, call): with pytest.raises(TypeError): call(sandbox_at(black_hole))