diff --git a/docs/index.md b/docs/index.md index 0c0e15e5..7cccc2a2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -29,6 +29,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. online pool reconfigure, metrics, the usage journal - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a read-only catalog volume on a running claim with the volume API +- [Tenant managed persistent volumes](managed-volumes.md) — runtime resource + ownership, warm attachment, recovery, feature map and verification evidence - [Go SDK](sdk.md) — connecting (single node and clusters), every option, the full sandbox surface, error handling - [Python SDK](sdk-python.md) — the guest and data-plane surface for the diff --git a/docs/managed-volumes.md b/docs/managed-volumes.md new file mode 100644 index 00000000..87fea148 --- /dev/null +++ b/docs/managed-volumes.md @@ -0,0 +1,152 @@ +# Tenant managed persistent volumes + +Managed volumes keep a tenant's data disk when a sandbox claim is released or +replaced with a different template. This phase adds node-local create, inspect, +list and conditional delete, claim-time attachment to warm VMs, and durable +single-writer exclusion. It builds on the existing +[read-only live volume API](live-volumes.md). + +This is a subset of [issue #300](https://github.com/cocoonstack/sandbox/issues/300). +Hibernate/wake, online growth, placement by volume and archive export remain +follow-up work. Attached volumes still block capture. Do not use this branch to +claim that all of #300 is complete. + +## Configure one node + +Set a nonempty `api_token` and an absolute, clean `managed_volume_dir` in the +sandboxd configuration. The managed directory must be private to the daemon, +on persistent storage, and outside disposable VM working directories. Every +existing ancestor must be owned by root or the daemon user. The daemon creates +private `disks/`, `registry.json` and `lock` entries; it exclusively locks the +registry directory. + +Startup rejects managed stores inside `data_dir/readers`, `data_dir/goldens` +and top-level `data_dir/fork-*` trees before creating files. The check includes +existing symbolic links at `data_dir`, `readers` and `goldens`. A store such as +`data_dir/managed-volumes` remains valid. Keep these directories under trusted +host control; validation does not detect bind-mount aliases or prevent a +concurrent pathname swap. + +```json +{ + "api_token": "replace-with-your-root-token", + "managed_volume_dir": "/var/lib/sandboxd/managed-volumes" +} +``` + +Keep the registry and backing files together. Do not resize, rename, replace or +hardlink a backing file by hand. Each managed resource records its immutable +ID, owner, node, capacity and backing device/inode/size. An identity mismatch +requires operator recovery and fails closed. + +The existing static catalog remains operator-owned. Its files and every path +component must be controlled by trusted operator or daemon principals. Static +catalog paths must not alias private managed storage, and a held name must +continue to identify the same backing. Stable symlink aliases and replacements +before attachment are checked. These checks precede the runtime's later path +open; they cannot defeat a concurrent host pathname swap. An untrusted actor +must not be able to change a catalog path component. + +## Manage a resource + +Use an explicit root or tenant management token. A sandbox token alone cannot +manage resources or call the existing live volume endpoints. Tenant tokens can +create and attach only their own managed resources; root must name an existing +tenant owner when creating one. + +| Request | Success | Refusal or retry behavior | +| --- | --- | --- | +| `PUT /v1/volumes/work` with `{"size_gib":20}` | `201`, `volume`, identity `ETag` | Identical retry returns `200` and the same identity; changed owner or capacity returns `409`. | +| `GET /v1/volumes` | Catalog and managed summaries visible to caller | Managed summaries include identity and owner; use individual GET for holders. | +| `GET /v1/volumes/work` | Managed detail with public holder IDs, modes and phases | Foreign and unknown names share `404`; host paths and process IDs stay private. | +| `DELETE /v1/volumes/work` with `If-Match: "vol_…"` | `204` | Missing precondition `428`; wrong identity `412`; held resource `409`. | + +Capacity is an integer from 1 through 1024 GiB. The backing is a raw sparse file; +logical capacity is not reserved physical space. Storage exhaustion can still +occur while guests write. Creation does not format or mount a filesystem. +Deletion tombstones preserve the outcome of an earlier deletion. Replaying that +old identity returns `204` without deleting a newly created same-name resource. + +The Go SDK exposes `CreateVolume`, `GetVolume` and `DeleteVolume`. Managed detail +uses `ManagedVolumeInfo`; existing `VolumeInfo` remains comparable. Structured +`ManagedVolumeError` unwraps to `APIError`. Python exposes `create_volume`, +`get_volume` and `delete_volume`; structured fields remain on `APIError`. + +## Format once, reuse across templates + +The initial raw disk requires explicit filesystem setup. This Python example +uses an attach-only claim, waits for the disk serial, formats it once and +unmounts it before release. Rerunning the example against an already formatted +resource deliberately fails the blank-disk assertion. + +```python +from cocoonsandbox import Client + +client = Client("127.0.0.1:7777", api_token="tenant-management-token") +volume = client.create_volume("work", 20) +with client.new("rt:24.04", volumes=[{"name": "work", "mode": "rw"}], mount=False) as sb: + device = sb.exec( + "sh", "-c", + 'for i in $(seq 1 200); do for p in /sys/block/*/serial; do ' + '[ "$(cat "$p")" = work ] && { echo "/dev/$(basename "$(dirname "$p")")"; exit 0; }; ' + 'done; sleep 0.01; done; exit 1', + ).strip() + sb.exec("sh", "-c", 'blkid "$1" >/dev/null 2>&1; [ "$?" = 2 ]', "probe", device) + sb.exec("mkfs.ext4", "-q", "-F", device) + sb.exec("mkdir", "-p", "/workspace") + sb.exec("mount", device, "/workspace") + sb.write_file("/workspace/marker", b"persistent") + sb.exec("sync", "-f", "/workspace/marker") + sb.exec("umount", "/workspace") + +with client.new("new-template:local", volumes=[{"name": "work", "mode": "rw", "mount": "/workspace"}]) as sb: + assert sb.read_file("/workspace/marker") == b"persistent" + +client.delete_volume("work", volume["id"]) +``` + +A reader excludes a writer, and a writer excludes both readers and other +writers. Multiple readers may share a ready disk. Live additions and removal +of managed entries are deferred, including mounted read-only managed entries. +A running managed claim can still change mounted read-only catalog entries +through the existing `GET/PUT/PATCH /v1/sandboxes/{id}/volumes` interface. +Attach-only managed entries retain their empty mount across those changes and +across daemon restart. + +## Ownership and recovery + +The managed registry supplements the existing flat `claims.json` journal. +Admission durably reserves the resource, then binds its VM process generation +before any disk effect. A live operation records the union of applied, desired +and possibly attached entries before changing a device. A confirmed retry can +trim that union only after device cleanup and claim-journal commit succeed. + +Release durably marks a hold retiring before cancelling the existing +`volumeRun` and dropping the claim. A successful runtime removal command alone +does not free the hold. Fresh runtime inspection must report absence and the +recorded boot ID/PID/start generation must be gone. Failed writes, uncertain +attachments and delayed removal retain ownership. These rules also apply when +`sync_claims` is disabled. + +A failed egress recovery fences the claim and attempts VM removal even if +managed retirement persistence fails. A stubborn VM or unavailable registry +can keep the hold for operator recovery. This phase does not promise that an +uncooperative VMM can always be killed. + +After restart, the persisted egress-quarantine failure marker resumes retirement +and removal. Ordinary failed claims keep their existing ownership until release. +Unclaimed stopped VM records with no live process are removed before startup +continues. Unreadable process metadata and unconfirmed removal still block +startup. The stale-create collector and recorded process-generation checks +remain in force. + +## Follow-up order + +1. Add quiesce/unmount/detach/capture and restore/reattach/remount with failure + rollback. Keep single-writer ownership while a claim is hibernated. +2. Add online growth with explicit block-device and guest-filesystem outcomes. +3. Add owner-node placement and a clear unavailable-owner response. +4. Add archive/export and restore of external data disks. +5. Add Kubernetes lifecycle/status ownership once the daemon contracts are stable. + +These are proposed next steps. None is implemented or qualified by this phase. diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 31a255cd..ebf4c96a 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -255,6 +255,12 @@ list; the field is emitted (as `true`) only for a writable entry this node declares and omitted otherwise (peer-only rows never carry it), so a read-only entry's response is byte-identical to v1. +When `managed_volume_dir` is enabled, this list also includes caller-visible +managed summaries and requires an explicit root or tenant token. Individual +managed GET responses carry holder details. The node-local create, inspect +and conditional-delete routes are documented in +[Tenant managed persistent volumes](managed-volumes.md#manage-a-resource). + ## GET /v1/sandboxes/{id}/volumes Returns the persisted desired set (`volumes_want`), confirmed applied set diff --git a/docs/sdk-python.md b/docs/sdk-python.md index d726d366..e168fc4f 100644 --- a/docs/sdk-python.md +++ b/docs/sdk-python.md @@ -21,6 +21,9 @@ wire is chunked, the caller's copy is not). Wire fidelity is pinned by the shared protocol fixture corpus that the Rust guest, Go, and Python all round-trip in CI. +For node-local `create_volume`, `get_volume`, `delete_volume` and the format-once +workflow, see [managed volumes](managed-volumes.md). + ## A complete example Claim a sandbox, push a project in, run a build, then freeze the built state diff --git a/docs/sdk.md b/docs/sdk.md index 2d66f2aa..c7da4017 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -9,6 +9,9 @@ module). One `Client` talks to one entry node; sandbox handles dial their owning node directly, so a client works unchanged against a single node or a cluster. +For node-local `CreateVolume`, `GetVolume`, `DeleteVolume`, managed detail +types and structured errors, see [managed volumes](managed-volumes.md). + ## A complete example Claim a sandbox, push a project into it, run a build, then freeze the built diff --git a/e2e/managed_volume_http_test.go b/e2e/managed_volume_http_test.go new file mode 100644 index 00000000..68857812 --- /dev/null +++ b/e2e/managed_volume_http_test.go @@ -0,0 +1,293 @@ +package e2e + +import ( + "encoding/json/v2" + "fmt" + "io" + "net/http" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/tenants" +) + +func TestManagedVolumeHTTPIdentityAndConditionalDelete(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":20}`, "") + if created.status != http.StatusCreated { + t.Fatalf("PV-M01 create status=%d body=%s, want 201", created.status, created.body) + } + volume := decodeManagedVolume(t, created.body) + if volume.ID == "" || volume.Name != "work" || volume.Tenant != "t1" || volume.Kind != "managed" || + volume.SizeBytes != 20<<30 || volume.State != "ready" { + t.Fatalf("PV-M01 unexpected resource projection: %+v", volume) + } + if created.etag != `"`+volume.ID+`"` { + t.Fatalf("PV-M01 etag=%q, want resource identity", created.etag) + } + for _, body := range []string{`{"size_gib":20}`, `{"tenant":"t1","size_gib":20}`} { + repeated := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", body, "") + if repeated.status != http.StatusOK || decodeManagedVolume(t, repeated.body).ID != volume.ID { + t.Fatalf("PV-M02 create retry status=%d body=%s", repeated.status, repeated.body) + } + } + changed := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":21}`, "") + if changed.status != http.StatusConflict { + t.Fatalf("PV-M03 changed capacity status=%d, want 409", changed.status) + } + got := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if got.status != http.StatusOK || got.etag != created.etag || decodeManagedVolume(t, got.body).ID != volume.ID { + t.Fatalf("PV-M02 inspect status=%d body=%s", got.status, got.body) + } + for _, tc := range []struct { + etag string + want int + }{ + {want: http.StatusPreconditionRequired}, + {etag: `"vol_00000000000000000000000000000000"`, want: http.StatusPreconditionFailed}, + } { + refused := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", tc.etag) + if refused.status != tc.want { + t.Fatalf("PV-M10 delete precondition status=%d, want %d", refused.status, tc.want) + } + } + for range 2 { + deleted := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", created.etag) + if deleted.status != http.StatusNoContent { + t.Fatalf("PV-M09/PV-M10 delete/retry status=%d body=%s", deleted.status, deleted.body) + } + } + replacement := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/work", "t1-token", `{"size_gib":20}`, "") + if replacement.status != http.StatusCreated || decodeManagedVolume(t, replacement.body).ID == volume.ID { + t.Fatalf("PV-M10 replacement lacks a fresh identity: status=%d body=%s", replacement.status, replacement.body) + } + stale := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/work", "t1-token", "", created.etag) + if stale.status != http.StatusNoContent { + t.Fatalf("PV-M10 authorized old deletion replay status=%d, want 204", stale.status) + } + survivor := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/work", "t1-token", "", "") + if survivor.status != http.StatusOK || decodeManagedVolume(t, survivor.body).ID != decodeManagedVolume(t, replacement.body).ID { + t.Fatalf("PV-M10 stale delete changed replacement: status=%d body=%s", survivor.status, survivor.body) + } +} + +func TestManagedVolumeHTTPAuthorizationAndPrivacy(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/private", "t1-token", `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("PV-M04 fixture create status=%d body=%s", created.status, created.body) + } + for _, method := range []string{http.MethodGet, http.MethodPut, http.MethodDelete} { + foreign := requestManagedVolume(t, st, method, "/v1/volumes/private", "t2-token", `{"size_gib":1}`, created.etag) + if foreign.status != http.StatusNotFound { + t.Errorf("PV-M04/PV-M08 foreign %s status=%d, want 404", method, foreign.status) + } + if method == http.MethodGet { + missing := requestManagedVolume(t, st, method, "/v1/volumes/unknown", "t2-token", "", "") + if foreign.status != missing.status || string(foreign.body) != string(missing.body) { + t.Errorf("PV-M04 foreign and unknown inspection differ: %d/%d", foreign.status, missing.status) + } + } + assertManagedReplyPrivate(t, foreign.body) + } + listing := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes", "t2-token", "", "") + if listing.status != http.StatusOK || strings.Contains(string(listing.body), `"private"`) { + t.Fatalf("PV-M04 foreign listing status=%d body=%s", listing.status, listing.body) + } + root := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/private", "root-token", "", "") + if root.status != http.StatusOK || decodeManagedVolume(t, root.body).Tenant != "t1" { + t.Fatalf("PV-M04 root inspection status=%d body=%s", root.status, root.body) + } + for _, token := range []string{"", "invalid-token"} { + for _, method := range []string{http.MethodGet, http.MethodPut, http.MethodDelete} { + reply := requestManagedVolume(t, st, method, "/v1/volumes/private", token, `{"size_gib":1}`, created.etag) + if reply.status != http.StatusUnauthorized { + t.Errorf("PV-M05 invalid credential %s status=%d, want 401", method, reply.status) + } + assertManagedReplyPrivate(t, reply.body) + } + } + claimStatus, claim := rawClaim(t, st, "") + if claimStatus != http.StatusOK { + t.Fatalf("PV-M05 sandbox credential fixture status=%d", claimStatus) + } + sandboxAttempt := requestManagedVolume(t, st, http.MethodGet, "/v1/volumes/private", claim.Token, "", "") + if sandboxAttempt.status != http.StatusUnauthorized { + t.Errorf("PV-M05 sandbox credential grants volume management: status=%d", sandboxAttempt.status) + } + released := requestManagedVolume(t, st, http.MethodPost, "/v1/sandboxes/"+claim.ID+"/release", claim.Token, "", "") + if released.status != http.StatusNoContent { + t.Fatalf("PV-M05 release credential fixture status=%d", released.status) + } + override := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/override", "t1-token", `{"size_gib":1,"tenant":"t2"}`, "") + if override.status != http.StatusForbidden { + t.Errorf("PV-M06 tenant override status=%d, want 403", override.status) + } + invalidRoot := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/root-owned", "root-token", `{"size_gib":1}`, "") + if invalidRoot.status != http.StatusBadRequest { + t.Errorf("PV-M06 root missing tenant status=%d, want 400", invalidRoot.status) + } + rootCreate := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/two", "root-token", `{"size_gib":1,"tenant":"t2"}`, "") + if rootCreate.status != http.StatusCreated || decodeManagedVolume(t, rootCreate.body).Tenant != "t2" { + t.Fatalf("PV-M06 root explicit tenant status=%d body=%s", rootCreate.status, rootCreate.body) + } + deleted := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/private", "t1-token", "", created.etag) + if deleted.status != http.StatusNoContent { + t.Fatalf("PV-M09 fixture deletion status=%d body=%s", deleted.status, deleted.body) + } + foreignTombstone := requestManagedVolume(t, st, http.MethodDelete, "/v1/volumes/private", "t2-token", "", created.etag) + if foreignTombstone.status != http.StatusNotFound { + t.Fatalf("PV-M05/PV-M10 deletion ID incorrectly grants authority: status=%d", foreignTombstone.status) + } +} + +func TestManagedVolumeHTTPRejectsInputAndSerializesCreation(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + for _, body := range []string{ + `{}`, `{"size_gib":0}`, `{"size_gib":-1}`, `{"size_gib":1025}`, + `{"size_gib":1.5}`, `{"size_gib":9223372036854775808}`, + `{"size_gib":1,"unknown":true}`, `{"size_gib":1} {}`, `{"size_gib":`, + } { + reply := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/invalid", "t1-token", body, "") + if reply.status != http.StatusBadRequest { + t.Errorf("PV-M07 input %q status=%d, want 400", body, reply.status) + } + } + oversized := `{"size_gib":1,"padding":"` + strings.Repeat("x", 1<<20) + `"}` + tooLarge := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/oversized", "t1-token", oversized, "") + if tooLarge.status != http.StatusBadRequest { + t.Errorf("PV-M07 oversized body status=%d, want 400", tooLarge.status) + } + for _, name := range []string{".bad", "cocoon-root"} { + reply := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/"+name, "t1-token", `{"size_gib":1}`, "") + if reply.status != http.StatusBadRequest { + t.Errorf("PV-M07/PV-M08 name %q status=%d, want 400", name, reply.status) + } + } + const callers = 8 + replies := make([]managedHTTPReply, callers) + var group sync.WaitGroup + for i := range replies { + group.Go(func() { + replies[i] = requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/race", "t1-token", `{"size_gib":1}`, "") + }) + } + group.Wait() + var id string + var created int + for _, reply := range replies { + if reply.status != http.StatusOK && reply.status != http.StatusCreated { + t.Fatalf("PV-M12 concurrent create status=%d body=%s", reply.status, reply.body) + } + volume := decodeManagedVolume(t, reply.body) + if id != "" && volume.ID != id { + t.Fatalf("PV-M12 competing creates returned different identities") + } + id = volume.ID + if reply.status == http.StatusCreated { + created++ + } + } + if created != 1 { + t.Fatalf("PV-M12 got %d creations, want exactly one", created) + } +} + +func TestManagedVolumeHTTPRequiresExplicitCredential(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + missing := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/implicit", "", `{"size_gib":1,"tenant":"t1"}`, "") + if missing.status != http.StatusUnauthorized { + t.Fatalf("PV-M05 legacy empty root configuration grants management without credentials: status=%d", missing.status) + } + for _, route := range []string{"/v1/volumes", "/v1/volumes/implicit"} { + for _, token := range []string{"", "invalid-token"} { + got := requestManagedVolume(t, st, http.MethodGet, route, token, "", "") + if got.status != http.StatusUnauthorized { + t.Fatalf("PV-M05 open legacy root configuration grants discovery: status=%d", got.status) + } + } + } +} + +type managedHTTPReply struct { + status int + body []byte + etag string +} + +type managedHTTPVolume struct { + ID string `json:"id"` + Name string `json:"name"` + Tenant string `json:"tenant"` + Kind string `json:"kind"` + State string `json:"state"` + SizeBytes int64 `json:"size_bytes"` +} + +func startManagedHTTPStack(t *testing.T, rootToken string) *stack { + t.Helper() + var cfg config.Config + body := fmt.Sprintf(`{"managed_volume_dir":%q}`, filepath.Join(t.TempDir(), "managed")) + if err := json.Unmarshal([]byte(body), &cfg); err != nil { + t.Fatal(err) + } + st := startConfigStack(t, rootToken, &cfg) + if rootToken == "" { + return st + } + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{ + {Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}, + }); err != nil { + t.Fatalf("configure tenants: %v", err) + } + return st +} + +func requestManagedVolume(t *testing.T, st *stack, method, route, token, body, etag string) managedHTTPReply { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), method, "http://"+st.addr+route, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + req.Header.Set("Content-Type", "application/json") + if etag != "" { + req.Header.Set("If-Match", etag) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("%s %s: %v", method, route, err) + } + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + assertManagedReplyPrivate(t, data) + return managedHTTPReply{status: resp.StatusCode, body: data, etag: resp.Header.Get("ETag")} +} + +func decodeManagedVolume(t *testing.T, body []byte) managedHTTPVolume { + t.Helper() + var reply struct { + Volume managedHTTPVolume `json:"volume"` + } + if err := json.Unmarshal(body, &reply); err != nil { + t.Fatalf("decode managed volume: %v", err) + } + return reply.Volume +} + +func assertManagedReplyPrivate(t *testing.T, body []byte) { + t.Helper() + for _, secret := range []string{"root-token", "t1-token", "t2-token", "/tmp/", "disk.raw", "registry.json"} { + if strings.Contains(string(body), secret) { + t.Errorf("managed API exposed private host or credential detail") + } + } +} diff --git a/e2e/managed_volume_privacy_test.go b/e2e/managed_volume_privacy_test.go new file mode 100644 index 00000000..3c5e3210 --- /dev/null +++ b/e2e/managed_volume_privacy_test.go @@ -0,0 +1,128 @@ +package e2e + +import ( + "encoding/json/v2" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/server" + "github.com/cocoonstack/sandbox/sandboxd/tenants" +) + +func TestLiveAttachManagedVolumePrivacy(t *testing.T) { + st := startManagedHTTPStack(t, "root-token") + for name, token := range map[string]string{"private": "t1-token", "owned": "t2-token"} { + created := requestManagedVolume(t, st, http.MethodPut, "/v1/volumes/"+name, token, `{"size_gib":1}`, "") + if created.status != http.StatusCreated { + t.Fatalf("create %s: %d %s", name, created.status, created.body) + } + } + claim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small"}`, "") + if claim.status != http.StatusOK { + t.Fatalf("claim: %d %s", claim.status, claim.body) + } + var sb server.ClaimResponse + if err := json.Unmarshal(claim.body, &sb); err != nil { + t.Fatal(err) + } + route := "/v1/sandboxes/" + sb.ID + "/volumes" + foreign := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"private","mount":"/volumes/probe"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"unknown","mount":"/volumes/probe"}]}`, "") + if foreign.status != http.StatusNotFound || unknown.status != http.StatusNotFound || string(foreign.body) != string(unknown.body) { + t.Errorf("foreign=%d %s; unknown=%d %s; want identical 404 replies", foreign.status, foreign.body, unknown.status, unknown.body) + } + owned := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"owned","mount":"/volumes/probe"}]}`, "") + if owned.status != http.StatusBadRequest || !strings.Contains(string(owned.body), "claim-time attachment only") { + t.Errorf("owned=%d %s; want unsupported live attach", owned.status, owned.body) + } +} + +type catalogPeerPlacer struct{ edgePlacer } + +func (*catalogPeerPlacer) VolumeCandidates(string, []string) []string { return []string{"peer:7777"} } + +func (*catalogPeerPlacer) VolumeOwners([]string) []string { return []string{"peer:7777"} } + +func (*catalogPeerPlacer) VolumeHolders() map[string]int { return map[string]int{"peer-only": 1} } + +func TestRootPeerCatalogRedirectWithManagedStore(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(fmt.Sprint(enabled), func(t *testing.T) { + var st *stack + if enabled { + st = startManagedHTTPStack(t, "root-token") + } else { + st = startConfigStack(t, "root-token", &config.Config{}) + } + srv := server.New(st.token, "", st.mgr, st.eng.real, &catalogPeerPlacer{}, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + defer ts.Close() + defer srv.CloseRelays() + probe := *st + probe.addr = strings.TrimPrefix(ts.URL, "http://") + response := requestManagedVolume(t, &probe, http.MethodPost, "/v1/claim", "root-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"peer-only"}]}`, "") + if response.status != http.StatusOK { + t.Fatalf("claim: %d %s", response.status, response.body) + } + var reply server.ClaimResponse + if err := json.Unmarshal(response.body, &reply); err != nil { + t.Fatal(err) + } + if !slices.Equal(reply.Redirect, []string{"peer:7777"}) || reply.ID != "" { + t.Fatalf("redirect reply = %+v", reply) + } + }) + } +} + +func TestStaticVolumePrivacyWithManagedStore(t *testing.T) { + for _, enabled := range []bool{false, true} { + t.Run(fmt.Sprint(enabled), func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "catalog.raw") + if err := os.WriteFile(path, []byte("catalog"), 0o600); err != nil { + t.Fatal(err) + } + cfg := &config.Config{Volumes: []config.VolumeSpec{{Name: "private-static", Path: path, Tenants: []string{"t1"}}}} + expected := http.StatusBadRequest + if enabled { + cfg.ManagedVolumeDir = filepath.Join(dir, "managed") + expected = http.StatusNotFound + } + st := startConfigStack(t, "root-token", cfg) + if err := st.mgr.SetTenants(t.Context(), []tenants.TenantSpec{{Name: "t1", Token: "t1-token"}, {Name: "t2", Token: "t2-token"}}); err != nil { + t.Fatal(err) + } + t.Run("claim", func(t *testing.T) { + foreign := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"private-static"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small","volumes":[{"name":"unknown"}]}`, "") + if foreign.status != expected || unknown.status != expected || string(foreign.body) != string(unknown.body) { + t.Fatalf("foreign=%d %s; unknown=%d %s; want identical %d replies", foreign.status, foreign.body, unknown.status, unknown.body, expected) + } + }) + t.Run("live-attach", func(t *testing.T) { + claim := requestManagedVolume(t, st, http.MethodPost, "/v1/claim", "t2-token", `{"template":"rt:24.04","net":"none","size":"small"}`, "") + if claim.status != http.StatusOK { + t.Fatalf("claim: %d %s", claim.status, claim.body) + } + var sb server.ClaimResponse + if err := json.Unmarshal(claim.body, &sb); err != nil { + t.Fatal(err) + } + route := "/v1/sandboxes/" + sb.ID + "/volumes" + foreign := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"private-static","mount":"/volumes/probe"}]}`, "") + unknown := requestManagedVolume(t, st, http.MethodPatch, route, "t2-token", `{"add":[{"name":"unknown","mount":"/volumes/probe"}]}`, "") + if foreign.status != expected || unknown.status != expected || string(foreign.body) != string(unknown.body) { + t.Fatalf("foreign=%d %s; unknown=%d %s; want identical %d replies", foreign.status, foreign.body, unknown.status, unknown.body, expected) + } + }) + }) + } +} diff --git a/e2e/wire_parity_test.go b/e2e/wire_parity_test.go index 012f172e..1dc2eb15 100644 --- a/e2e/wire_parity_test.go +++ b/e2e/wire_parity_test.go @@ -44,6 +44,10 @@ var wirePairs = []struct { {"checkpointRecord", false, types.Checkpoint{}, []string{"node", "key", "tenant", "policy_source", "no_egress", "guest_env", "archive"}}, {"volumeListResponse", false, server.VolumeListResponse{}, nil}, {"VolumeInfo", false, types.VolumeInfo{}, nil}, + {"ManagedVolumeInfo", false, types.ManagedVolumeInfo{}, nil}, + {"VolumeHolder", false, types.VolumeHolder{}, nil}, + {"VolumeCreate", true, types.VolumeCreate{}, nil}, + {"volumeResponse", false, server.VolumeResponse{}, nil}, {"VolumeState", false, types.VolumeState{}, nil}, {"SandboxVolumesResponse", false, types.SandboxVolumesResponse{}, nil}, {"VolumePatch", true, types.VolumePatch{}, nil}, @@ -117,6 +121,7 @@ func parseSDKFields(t *testing.T) map[string]map[string]string { t.Fatal(err) } out := map[string]map[string]string{} + embedded := map[string][]string{} for _, path := range paths { if strings.HasSuffix(path, "_test.go") { continue @@ -130,21 +135,38 @@ func parseSDKFields(t *testing.T) map[string]map[string]string { if !ok { return true } - if st, ok := spec.Type.(*ast.StructType); ok { - out[spec.Name.Name] = map[string]string{} - for _, f := range st.Fields.List { - if f.Tag == nil { - continue - } - tag, _ := strconv.Unquote(f.Tag.Value) - if name, opts := splitJSONTag(reflect.StructTag(tag).Get("json")); name != "" && name != "-" { - out[spec.Name.Name][name] = opts + st, ok := spec.Type.(*ast.StructType) + if !ok { + return true + } + out[spec.Name.Name] = map[string]string{} + for _, f := range st.Fields.List { + if len(f.Names) == 0 { + if id, ok := f.Type.(*ast.Ident); ok { + embedded[spec.Name.Name] = append(embedded[spec.Name.Name], id.Name) } } + if f.Tag == nil { + continue + } + tag, _ := strconv.Unquote(f.Tag.Value) + if name, opts := splitJSONTag(reflect.StructTag(tag).Get("json")); name != "" && name != "-" { + out[spec.Name.Name][name] = opts + } } return true }) } + var flatten func(string) map[string]string + flatten = func(name string) map[string]string { + for _, base := range embedded[name] { + maps.Copy(out[name], flatten(base)) + } + return out[name] + } + for name := range out { + flatten(name) + } return out } diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index fba3efba..63e9d4aa 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -274,9 +274,10 @@ type Config struct { DiskReserve *DiskReserve `json:"disk_reserve,omitempty"` - Volumes []VolumeSpec `json:"volumes,omitempty"` - Mesh *MeshConfig `json:"mesh,omitempty"` - Pools []PoolSpec `json:"pools"` + ManagedVolumeDir string `json:"managed_volume_dir,omitempty"` + Volumes []VolumeSpec `json:"volumes,omitempty"` + Mesh *MeshConfig `json:"mesh,omitempty"` + Pools []PoolSpec `json:"pools"` } // HasEgress reports whether the node can attach egress-lane VMs. @@ -398,6 +399,9 @@ func (c *Config) validateReplication() error { } func (c *Config) validateVolumes() error { + if err := c.ValidateManagedVolumeStore(); err != nil { + return err + } names := make(map[string]struct{}, len(c.Volumes)) paths := make(map[string]string, len(c.Volumes)) for _, volume := range c.Volumes { @@ -422,6 +426,58 @@ func (c *Config) validateVolumes() error { return nil } +func (c *Config) ValidateManagedVolumeStore() error { + if c.ManagedVolumeDir == "" { + return nil + } + if !filepath.IsAbs(c.ManagedVolumeDir) || filepath.Clean(c.ManagedVolumeDir) != c.ManagedVolumeDir || c.ManagedVolumeDir == "/" { + return fmt.Errorf("managed_volume_dir must be a clean absolute non-root path") + } + if c.APIToken == "" { + return fmt.Errorf("managed_volume_dir requires a nonempty api_token") + } + dataDir, err := filepath.Abs(c.DataDir) + if err != nil { + return fmt.Errorf("resolve data_dir: %w", err) + } + resolved, err := resolveExistingAncestors(dataDir) + if err != nil { + return fmt.Errorf("resolve data_dir: %w", err) + } + for _, root := range []string{dataDir, resolved} { + relative, err := filepath.Rel(root, c.ManagedVolumeDir) + if err != nil { + return fmt.Errorf("compare managed_volume_dir with data_dir: %w", err) + } + first, _, _ := strings.Cut(relative, string(filepath.Separator)) + if first == "readers" || first == "goldens" || strings.HasPrefix(first, "fork-") { + return fmt.Errorf("managed_volume_dir must be outside data_dir startup cleanup paths (readers, goldens, fork-*)") + } + } + for _, namespace := range []string{"readers", "goldens"} { + swept, err := resolveExistingAncestors(filepath.Join(dataDir, namespace)) + if err != nil { + return fmt.Errorf("resolve data_dir cleanup path: %w", err) + } + relative, err := filepath.Rel(swept, c.ManagedVolumeDir) + if err != nil { + return fmt.Errorf("compare managed_volume_dir with cleanup path: %w", err) + } + if relative != ".." && !strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return fmt.Errorf("managed_volume_dir must be outside data_dir %s cleanup path", namespace) + } + relative, err = filepath.Rel(c.ManagedVolumeDir, swept) + if err != nil { + return fmt.Errorf("compare cleanup path with managed_volume_dir: %w", err) + } + first, _, _ := strings.Cut(relative, string(filepath.Separator)) + if first == "disks" { + return fmt.Errorf("data_dir %s cleanup path must be outside managed volume backing storage", namespace) + } + } + return nil +} + func (c *Config) validatePreview() error { if c.PreviewListen == "" { return nil @@ -663,6 +719,25 @@ func StorageBytes(s string) (int64, error) { return units.RAMInBytes(s) } +func resolveExistingAncestors(path string) (string, error) { + suffix := "" + for { + _, err := os.Lstat(path) + if err == nil { + resolved, resolveErr := filepath.EvalSymlinks(path) + if resolveErr != nil { + return "", resolveErr + } + return filepath.Join(resolved, suffix), nil + } + if !os.IsNotExist(err) || filepath.Dir(path) == path { + return "", err + } + suffix = filepath.Join(filepath.Base(path), suffix) + path = filepath.Dir(path) + } +} + func validatePolicy(p *egress.Policy, secrets map[string]struct{}) error { if p == nil { return nil diff --git a/sandboxd/config/managed_volume_test.go b/sandboxd/config/managed_volume_test.go new file mode 100644 index 00000000..5bd02a80 --- /dev/null +++ b/sandboxd/config/managed_volume_test.go @@ -0,0 +1,137 @@ +package config + +import ( + "os" + "path/filepath" + "testing" +) + +func TestManagedVolumeStoreExcludesStartupCleanupPaths(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + for _, tc := range []struct { + path string + allow bool + }{ + {filepath.Join(data, "managed"), true}, + {filepath.Join(root, "outside"), true}, + {filepath.Join(data, "readers-backup", "managed"), true}, + {filepath.Join(data, "forks", "managed"), true}, + {filepath.Join(data, "readers"), false}, + {filepath.Join(data, "readers", "managed"), false}, + {filepath.Join(data, "fork-state"), false}, + {filepath.Join(data, "fork-state", "managed"), false}, + {filepath.Join(data, "goldens"), false}, + {filepath.Join(data, "goldens", "managed"), false}, + } { + t.Run(tc.path, func(t *testing.T) { + cfg := Config{DataDir: data, ManagedVolumeDir: tc.path, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Fatalf("path %q accepted=%v, want %v: %v", tc.path, err == nil, tc.allow, err) + } + }) + } +} + +func TestManagedVolumeStoreResolvesDataDirectoryAliases(t *testing.T) { + root := t.TempDir() + realDir := filepath.Join(root, "real") + alias := filepath.Join(root, "alias") + if err := os.Mkdir(realDir, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Symlink(realDir, alias); err != nil { + t.Fatal(err) + } + t.Chdir(root) + for _, tc := range []struct { + name, data, managed string + allow bool + }{ + {"alias-readers", alias, filepath.Join(realDir, "readers", "managed"), false}, + {"alias-goldens", alias, filepath.Join(realDir, "goldens"), false}, + {"alias-fork", alias, filepath.Join(realDir, "fork-persistent"), false}, + {"alias-new-suffix", filepath.Join(alias, "new", "data"), filepath.Join(realDir, "new", "data", "readers", "managed"), false}, + {"alias-safe", alias, filepath.Join(realDir, "managed"), true}, + {"relative", "real", filepath.Join(realDir, "readers", "managed"), false}, + {"relative-safe", "real", filepath.Join(realDir, "readers-safe"), true}, + {"empty", "", filepath.Join(root, "fork-store"), false}, + {"empty-safe", "", filepath.Join(root, "managed"), true}, + {"ancestor", filepath.Join(realDir, "data"), realDir, true}, + {"same-root", realDir, realDir, true}, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := Config{DataDir: tc.data, ManagedVolumeDir: tc.managed, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Fatalf("data=%q managed=%q accepted=%v, want %v: %v", tc.data, tc.managed, err == nil, tc.allow, err) + } + }) + } + if _, err := os.Stat(filepath.Join(realDir, "new")); !os.IsNotExist(err) { + t.Fatalf("validation created data directories: %v", err) + } +} + +func TestManagedVolumeStoreRejectsCleanupDirectoryAliases(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + t.Run(namespace, func(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + external := filepath.Join(root, "external") + for _, dir := range []string{data, external} { + if err := os.Mkdir(dir, 0o700); err != nil { + t.Fatal(err) + } + } + if err := os.Symlink(external, filepath.Join(data, namespace)); err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + path string + allow bool + }{ + {external, false}, + {filepath.Join(external, "managed"), false}, + {filepath.Join(data, "managed"), true}, + {root, true}, + } { + cfg := Config{DataDir: data, ManagedVolumeDir: tc.path, APIToken: "root"} + err := cfg.ValidateManagedVolumeStore() + if (err == nil) != tc.allow { + t.Errorf("%s alias target %q accepted=%v, want %v: %v", namespace, tc.path, err == nil, tc.allow, err) + } + } + if _, err := os.Stat(filepath.Join(external, "managed")); !os.IsNotExist(err) { + t.Fatalf("validation created managed directory: %v", err) + } + }) + } +} + +func TestManagedVolumeStoreProtectsBackingDirectoryFromCleanupAliases(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + for _, target := range []string{"disks", "disks/nested", "disks-backup", "work/readers"} { + t.Run(namespace+"/"+target, func(t *testing.T) { + root := t.TempDir() + data := filepath.Join(root, "data") + managed := filepath.Join(root, "managed") + aliasTarget := filepath.Join(managed, target) + for _, dir := range []string{data, aliasTarget} { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + } + if err := os.Symlink(aliasTarget, filepath.Join(data, namespace)); err != nil { + t.Fatal(err) + } + cfg := Config{DataDir: data, ManagedVolumeDir: managed, APIToken: "root"} + allow := target == "disks-backup" || target == "work/readers" + if err := cfg.ValidateManagedVolumeStore(); (err == nil) != allow { + t.Fatalf("cleanup alias %q accepted=%v, want %v: %v", target, err == nil, allow, err) + } + }) + } + } +} diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 683074ee..deefbf81 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -55,7 +55,7 @@ func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, o ClaimOptio } // holds belong to this path until finalize; past it the sandbox carries them. reserved := applied - defer func() { m.unreserveVolumes(reserved) }() + defer func() { m.releaseUnbound(volumeSpecs, reserved) }() m.mu.Lock() var sb *types.Sandbox if p := m.pools[key]; p != nil { @@ -68,7 +68,7 @@ func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, o ClaimOptio return nil, ErrNoWarm } if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs, applied); volumeErr != nil { - m.abortVolumeClaim(ctx, sb.VMName, &reserved) + m.abortVolumeClaim(ctx, sb.VMName, &reserved, volumeSpecs) return nil, volumeErr } o.apply(sb) @@ -89,7 +89,7 @@ func (m *Manager) ClaimProvision(ctx context.Context, key types.PoolKey, o Claim return nil, err } reserved := applied - defer func() { m.unreserveVolumes(reserved) }() + defer func() { m.releaseUnbound(volumeSpecs, reserved) }() golden, err := m.resolveGolden(ctx, key, o.Tenant) if err != nil { return nil, fmt.Errorf("resolve template: %w", err) @@ -104,7 +104,7 @@ func (m *Manager) ClaimProvision(ctx context.Context, key types.PoolKey, o Claim return nil, err } if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs, applied); volumeErr != nil { - m.abortVolumeClaim(ctx, sb.VMName, &reserved) + m.abortVolumeClaim(ctx, sb.VMName, &reserved, volumeSpecs) return nil, volumeErr } sb.TemplateDigest = golden.templateDigest @@ -309,6 +309,10 @@ func (m *Manager) releaseResolved(ctx context.Context, id string, sb *types.Sand marked = ck m.mu.Lock() } + if err := m.retireManaged(id); err != nil { + m.mu.Unlock() + return err + } volumeHolds := m.cancelVolumeRun(sb) snap, ck, vmName := sb.HibernateSnap, sb.ArchiveCk, sb.VMName if ck != "" { @@ -454,13 +458,26 @@ func (m *Manager) finalize(ctx context.Context, sb *types.Sandbox, ttl time.Dura func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl time.Duration, claimRefPrefix string, inherited bool) error { now := time.Now() for _, sb := range sbs { + id := sb.ID + _, managed := m.managedHold(id) stampIdentity(sb, clampTTL(ttl)) + if managed { + sb.ID = id + } if claimRefPrefix != "" { sb.ClaimRef = claimRefPrefix + sb.ID } sb.TouchAt(now) } - err := m.checkInjects(m.view.Load(), sbs) + var err error + for _, sb := range sbs { + if err = m.activateManaged(sb.ID); err != nil { + break + } + } + if err == nil { + err = m.checkInjects(m.view.Load(), sbs) + } if err == nil { err = m.deliverEnv(ctx, sbs, inherited) } @@ -510,6 +527,9 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t // rollbackClaim unwinds a claim batch: a NIC that cannot be locked is never handed out. func (m *Manager) rollbackClaim(ctx context.Context, sbs []*types.Sandbox) { + for _, sb := range sbs { + _ = m.retireManaged(sb.ID) + } m.mu.Lock() ids := make([]string, len(sbs)) holds := make([][]types.Volume, len(sbs)) @@ -529,7 +549,13 @@ func (m *Manager) rollbackClaim(ctx context.Context, sbs []*types.Sandbox) { } // abortVolumeClaim removes a failed claim's VM and clears reserved against a double release. -func (m *Manager) abortVolumeClaim(ctx context.Context, vmName string, reserved *[]types.Volume) { +func (m *Manager) abortVolumeClaim(ctx context.Context, vmName string, reserved *[]types.Volume, specs []resolvedVolume) { + if len(specs) > 0 && specs[0].holdID != "" { + if h, ok := m.managedHold(specs[0].holdID); ok && h.VMName == "" { + m.releaseUnbound(specs, *reserved) + *reserved = nil + } + } m.removeOrRetry(ctx, vmName, "", volumeTeardown{holds: *reserved}) *reserved = nil } @@ -572,6 +598,9 @@ func (m *Manager) reapOnce(ctx context.Context) { } expired = append(expired, victim{action: action, id: id, sb: sb}) default: + if err := m.retireManaged(id); err != nil { + continue + } expired = append(expired, victim{action: reapDestroy, id: id, vmName: sb.VMName, snap: sb.HibernateSnap, tenant: sb.Tenant, sb: sb, holds: m.cancelVolumeRun(sb)}) m.dropClaimLocked(sb) dropped = append(dropped, id) @@ -750,9 +779,18 @@ func (m *Manager) admitOptions(ctx context.Context, key types.PoolKey, o ClaimOp if err := m.checkEnv(o.Env); err != nil { return nil, nil, err } - volumeSpecs, err := m.resolveVolumes(ctx, o.Tenant, o.Volumes) - if err != nil { - return nil, nil, err + volumeSpecs, resolveErr := m.resolveVolumes(ctx, o.Tenant, o.Volumes) + if resolveErr != nil { + return nil, nil, resolveErr + } + for _, v := range volumeSpecs { + if v.managedID != "" { + if err := m.tenantRemoved(ctx, o.Tenant); err != nil { + return nil, nil, err + } + applied, err := m.admitManaged(o.Tenant, volumeSpecs) + return volumeSpecs, applied, err + } } applied, err := m.admitVolumes(o.Tenant, volumeSpecs) return volumeSpecs, applied, err diff --git a/sandboxd/pool/failover.go b/sandboxd/pool/failover.go index dd0896f4..926a8404 100644 --- a/sandboxd/pool/failover.go +++ b/sandboxd/pool/failover.go @@ -411,6 +411,13 @@ func (m *Manager) forgetClaim(ctx context.Context, sb *types.Sandbox) bool { m.mu.Unlock() return false } + if _, managed := m.managedHold(sb.ID); managed { + if err := m.retireManaged(sb.ID); err != nil { + m.mu.Unlock() + return false + } + m.cancelVolumeRun(sb) + } m.dropClaimLocked(sb) js := m.store.del(sb.ID) m.mu.Unlock() diff --git a/sandboxd/pool/managed_attachment.go b/sandboxd/pool/managed_attachment.go new file mode 100644 index 00000000..df82e5fe --- /dev/null +++ b/sandboxd/pool/managed_attachment.go @@ -0,0 +1,392 @@ +package pool + +import ( + "context" + "errors" + "fmt" + "io/fs" + "os" + "strconv" + "strings" + "syscall" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func readProcessIdentity(pid int) (processIdentity, error) { + return readProcessIdentityAt("/proc", pid) +} + +func readProcessIdentityAt(procDir string, pid int) (processIdentity, error) { + if pid <= 0 { + return processIdentity{}, errors.New("VMM PID unavailable") + } + proc := os.DirFS(procDir) + boot, err := fs.ReadFile(proc, "sys/kernel/random/boot_id") + if err != nil { + // Missing boot metadata cannot prove that the PID is gone. + return processIdentity{}, fmt.Errorf("read VMM boot identity: %v", err) + } + bootID := strings.TrimSpace(string(boot)) + if bootID == "" { + return processIdentity{}, errors.New("empty VMM boot identity") + } + stat, err := fs.ReadFile(proc, strconv.Itoa(pid)+"/stat") + if err != nil { + return processIdentity{}, err + } + end := strings.LastIndexByte(string(stat), ')') + if end < 0 { + return processIdentity{}, errors.New("invalid process identity") + } + fields := strings.Fields(string(stat[end+1:])) + if len(fields) < 20 { + return processIdentity{}, errors.New("invalid process identity") + } + ticks, err := strconv.ParseUint(fields[19], 10, 64) + if err != nil { + return processIdentity{}, err + } + return processIdentity{BootID: bootID, PID: pid, Start: ticks}, nil +} + +func (r *volumeRegistry) processGone(p processIdentity) bool { + if p.PID <= 0 || p.Start == 0 || p.BootID == "" { + return false + } + current, err := r.process(p.PID) + if errors.Is(err, os.ErrNotExist) { + return true + } + return err == nil && current != p +} + +func (m *Manager) admitManaged(tenant string, volumes []resolvedVolume) ([]types.Volume, error) { + applied := appliedVolumes(volumes) + m.mu.Lock() + defer m.mu.Unlock() + if err := m.quotaErr(1, tenant); err != nil { + return nil, err + } + r := m.managed + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return nil, managedStorageError() + } + h := volumeHold{SandboxID: "sb_" + randHex(8), Tenant: tenant, State: holdAttaching} + for _, volume := range volumes { + ref, err := r.referenceForAdmission(tenant, volume) + if err != nil { + return nil, err + } + h.Refs = append(h.Refs, ref) + } + if err := m.reserveVolumes(applied); err != nil { + return nil, err + } + if err := m.confirmVolumesClean(volumes); err != nil { + m.releaseVolumes(applied) + return nil, err + } + r.data.Holds[h.SandboxID] = h + for i := range volumes { + volumes[i].holdID = h.SandboxID + } + if err := r.commit(); err != nil { + return nil, err + } + return applied, nil +} + +func (r *volumeRegistry) referenceForAdmission(tenant string, volume resolvedVolume) (volumeRef, error) { + if volume.managedID == "" { + identity, err := catalogBacking(volume.disk.Path) + if err != nil { + return volumeRef{}, ErrVolumeUnavailable + } + return volumeRef{Volume: volume.applied, Path: volume.disk.Path, Backing: identity}, nil + } + v, ok := r.data.Volumes[volume.applied.Name] + if !ok || v.ID != volume.managedID || (tenant != "" && v.Tenant != tenant) { + return volumeRef{}, managedNotFound() + } + if v.State != volumeReady || r.verify(v) != nil { + return volumeRef{}, managedStorageError() + } + for _, other := range r.data.Holds { + for _, ref := range other.Refs { + if ref.ID == v.ID && (ref.Volume.RW() || volume.applied.RW()) { + return volumeRef{}, &VolumeError{Status: 409, Code: volumeBusyCode, Message: "volume is held by a sandbox", Volume: v.Name, Holders: r.holders(v.ID)} + } + } + } + return volumeRef{Volume: volume.applied, ID: v.ID, Path: r.path(v.ID), Backing: v.Backing}, nil +} + +func (m *Manager) releaseUnbound(volumes []resolvedVolume, reserved []types.Volume) { + if len(reserved) == 0 { + return + } + if len(volumes) == 0 || volumes[0].holdID == "" { + m.unreserveVolumes(reserved) + return + } + r := m.managed + m.mu.Lock() + defer m.mu.Unlock() + r.mu.Lock() + defer r.mu.Unlock() + id := volumes[0].holdID + h, ok := r.data.Holds[id] + if !ok || h.VMName != "" || r.fault != nil { + return + } + delete(r.data.Holds, id) + if r.commit() == nil { + m.releaseVolumes(reserved) + } +} + +func (m *Manager) bindManaged(ctx context.Context, sb *types.Sandbox, volumes []resolvedVolume) error { + if len(volumes) == 0 || volumes[0].holdID == "" { + return nil + } + id := volumes[0].holdID + sb.ID = id + rec, present, err := m.eng.Inspect(ctx, sb.VMName) + if err != nil || !present { + return managedStorageError() + } + p, err := m.managed.process(rec.PID) + if err != nil { + return managedStorageError() + } + r := m.managed + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return managedStorageError() + } + h, ok := r.data.Holds[id] + if !ok || h.State != holdAttaching || h.VMName != "" { + return managedStorageError() + } + h.VMName = sb.VMName + h.Process = p + r.data.Holds[id] = h + return r.commit() +} + +func (m *Manager) activateManaged(id string) error { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + if !ok { + return nil + } + if r.fault != nil || h.State == holdRetiring { + return managedStorageError() + } + h.State = holdClaimed + r.data.Holds[id] = h + return r.commit() +} + +func (m *Manager) finishManaged(id string, clears []string) bool { + r := m.managed + m.mu.Lock() + defer m.mu.Unlock() + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + if !ok { + return true + } + if r.fault != nil { + return false + } + if err := m.store.commit(m.store.del(id)); err != nil { + return false + } + for _, path := range clears { + for _, ref := range h.Refs { + if ref.Path == path { + if identity, err := catalogBacking(path); err == nil && identity == ref.Backing { + _ = clearVolumeDirty(path) + } + break + } + } + } + delete(r.data.Holds, id) + if r.commit() != nil { + r.data.Holds[id] = h + return false + } + refs := make([]types.Volume, len(h.Refs)) + for i, ref := range h.Refs { + refs[i] = ref.Volume + } + m.releaseVolumes(refs) + return true +} + +func (m *Manager) recoverManaged(ctx context.Context, claims map[string]*types.Sandbox, live map[string]types.VMRecord) (map[string]bool, error) { + owned := map[string]bool{} + r := m.managed + if r == nil { + return owned, nil + } + r.mu.Lock() + if r.fault != nil { + r.mu.Unlock() + return owned, managedStorageError() + } + holds := make([]volumeHold, 0, len(r.data.Holds)) + for _, h := range r.data.Holds { + holds = append(holds, h) + } + r.mu.Unlock() + for _, h := range holds { + m.mu.Lock() + for _, ref := range h.Refs { + m.adoptVolumes([]types.Volume{ref.Volume}) + } + m.mu.Unlock() + sb, exists := claims[h.SandboxID] + rec, present := live[h.VMName] + valid := exists && sb.VMName == h.VMName && sb.Tenant == h.Tenant && sb.Failed != egressQuarantineFailure && h.State != holdRetiring && present + if valid { + identity, err := r.process(rec.PID) + if err != nil && !errors.Is(err, os.ErrNotExist) { + return owned, managedStorageError() + } + valid = err == nil && identity == h.Process + } + if valid { + for _, volume := range heldVolumes(sb) { + matched := false + for _, ref := range h.Refs { + if ref.Volume == volume { + matched = true + break + } + } + if !matched { + return owned, managedStorageError() + } + } + owned[h.VMName] = true + if err := m.activateManaged(h.SandboxID); err != nil { + return owned, err + } + continue + } + delete(claims, h.SandboxID) + if err := m.retireManaged(h.SandboxID); err != nil { + return owned, err + } + if h.VMName == "" { + if !m.finishManaged(h.SandboxID, nil) { + return owned, managedStorageError() + } + continue + } + owned[h.VMName] = true + m.removeOrRetry(ctx, h.VMName, h.SandboxID, volumeTeardown{managedID: h.SandboxID}) + } + for _, sb := range claims { + for _, v := range heldVolumes(sb) { + r.mu.Lock() + _, managed := r.data.Volumes[v.Name] + _, held := r.data.Holds[sb.ID] + r.mu.Unlock() + if managed && !held { + return owned, managedStorageError() + } + } + } + return owned, nil +} + +func (m *Manager) managedLiveAdd(id string, volumes []resolvedVolume) error { + if len(volumes) == 0 { + return nil + } + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + if !ok { + return nil + } + if r.fault != nil || h.State != holdClaimed { + return managedStorageError() + } + for _, v := range volumes { + for _, ref := range h.Refs { + if ref.Volume.Name == v.applied.Name { + return volumeError(409, "volume_attachment_uncertain", "previous attachment requires sandbox release") + } + } + identity, err := catalogBacking(v.disk.Path) + if err != nil { + return ErrVolumeUnavailable + } + h.Refs = append(h.Refs, volumeRef{Volume: v.applied, Path: v.disk.Path, Backing: identity}) + } + r.data.Holds[id] = h + return r.commit() +} + +func (m *Manager) managedLiveRemove(id string, volumes []types.Volume) error { + if len(volumes) == 0 { + return nil + } + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + if !ok { + return nil + } + if r.fault != nil || h.State != holdClaimed { + return managedStorageError() + } + remove := map[string]bool{} + for _, v := range volumes { + remove[v.Name] = true + } + refs := make([]volumeRef, 0, len(h.Refs)) + for _, ref := range h.Refs { + if !remove[ref.Volume.Name] { + refs = append(refs, ref) + } + } + h.Refs = refs + r.data.Holds[id] = h + return r.commit() +} + +func catalogBacking(path string) (backingIdentity, error) { + st, err := os.Stat(path) + if err != nil { + return backingIdentity{}, err + } + if !st.Mode().IsRegular() { + return backingIdentity{}, errors.New("catalog backing is not a regular file") + } + raw := st.Sys().(*syscall.Stat_t) + return backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()}, nil +} diff --git a/sandboxd/pool/managed_integration_test.go b/sandboxd/pool/managed_integration_test.go new file mode 100644 index 00000000..ef0c4bb9 --- /dev/null +++ b/sandboxd/pool/managed_integration_test.go @@ -0,0 +1,284 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestManagedStoreRequiresAuthenticatedRootConfiguration(t *testing.T) { + cfg := managedConfig(t) + cfg.APIToken = "" + _, err := NewManager(t.Context(), cfg, newManagedEngine(), testSecrets(t), Backends{}) + if err == nil || !strings.Contains(err.Error(), "api_token") { + t.Fatalf("anonymous managed store opened: %v", err) + } +} + +func TestManagedNamesAndCatalogSymlinksSurviveRestart(t *testing.T) { + cfg := managedConfig(t) + image := writeVolumeImage(t, "catalog.img", "catalog") + link := filepath.Join(t.TempDir(), "catalog-link") + if err := os.Symlink(image, link); err != nil { + t.Fatal(err) + } + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: link, Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "kb-data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("kb-data", "catalog")) + if err != nil { + t.Fatal(err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if got, err := m.GetVolume("kb-data", "t1"); err != nil || got.ID != v.ID { + t.Fatalf("reopened name: %+v %v", got, err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } +} + +func TestManagedRejectsCatalogSymlinkToManagedBacking(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + link := filepath.Join(t.TempDir(), "catalog-link") + if err := os.Symlink(m.managed.path(volumeID(v.ID)), link); err != nil { + t.Fatal(err) + } + m.Close() + cfg.Volumes = []config.VolumeSpec{{Name: "alias", Path: link, Writable: true}} + if next, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + next.Close() + t.Fatal("managed backing alias accepted as an independent catalog writer") + } +} + +func TestManagedRejectsCatalogReplacementBeforeAttach(t *testing.T) { + for _, alias := range []bool{false, true} { + t.Run(map[bool]string{false: "different inode", true: "managed alias"}[alias], func(t *testing.T) { + cfg := managedConfig(t) + path := writeVolumeImage(t, "catalog.img", "original") + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: path, Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + specs, _, err := m.admitOptions(t.Context(), testKey, managedOptions("data", "catalog")) + if err != nil { + t.Fatal(err) + } + rec, err := e.RunCold(t.Context(), "sbx-replaced", testKey, "") + if err != nil { + t.Fatal(err) + } + sb := &types.Sandbox{VMName: rec.Config.Name, VsockSocket: rec.VsockSocket, Key: testKey} + if err := m.bindManaged(t.Context(), sb, specs); err != nil { + t.Fatal(err) + } + if err := os.Rename(path, path+".old"); err != nil { + t.Fatal(err) + } + if alias { + if err := os.Symlink(m.managed.path(volumeID(v.ID)), path); err != nil { + t.Fatal(err) + } + } else if err := os.WriteFile(path, []byte("replacement"), 0o600); err != nil { + t.Fatal(err) + } + for _, spec := range specs { + if spec.applied.Name != "catalog" { + continue + } + if err := m.applyVolume(t.Context(), sb, spec, false); err == nil { + t.Fatal("replacement attached after admission pinned another backing") + } + } + e.mu.Lock() + defer e.mu.Unlock() + if len(e.volumeSpecs) != 0 { + t.Fatalf("device effects before replacement rejection: %v", e.volumeOps) + } + }) + } +} + +func TestManagedReleaseDoesNotClearReplacementCatalogDirtyMarker(t *testing.T) { + cfg := managedConfig(t) + path := writeVolumeImage(t, "catalog.img", "old") + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: path, Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "t1", Volumes: []types.Volume{ + {Name: "data", Mode: "rw"}, {Name: "catalog", Mode: "rw"}, + }}) + if claimErr != nil { + t.Fatal(claimErr) + } + if err := os.Rename(path, path+".old"); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("replacement"), 0o600); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + if !volumeDirty(path) { + t.Fatal("release cleared a replacement backing's dirty marker") + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "t1", Volumes: []types.Volume{{Name: "catalog"}}}) + if !errors.Is(claimErr, ErrVolumeNeedsRecovery) { + t.Fatalf("replacement reader admitted: %v", claimErr) + } +} + +func TestManagedSettledRetryDrainsPreviouslyUncertainCatalogRef(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog"), Writable: true}} + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + e.diskAttachErrFor, e.diskAttachErr = "catalog", errors.New("uncertain attach") + _, claimErr = m.PatchVolumes(t.Context(), sb.ID, "", types.VolumePatch{Add: []types.Volume{{Name: "catalog"}}}) + if claimErr == nil { + t.Fatal("injected attach succeeded") + } + e.diskAttachErr, e.diskAttachErrFor = nil, "" + if err := os.Mkdir(m.store.path+".tmp", 0o700); err != nil { + t.Fatal(err) + } + _, claimErr = m.PatchVolumes(t.Context(), sb.ID, "", types.VolumePatch{Remove: []string{"catalog"}}) + if claimErr == nil { + t.Fatal("injected claim commit succeeded") + } + if err := os.Remove(m.store.path + ".tmp"); err != nil { + t.Fatal(err) + } + waitFor(t, m.store.synced) + if _, err := m.PatchVolumes(t.Context(), sb.ID, "", types.VolumePatch{}); err != nil { + t.Fatal(err) + } + h, ok := m.managedHold(sb.ID) + if !ok || len(h.Refs) != 1 { + t.Fatalf("settled retry leaked uncertain reference: %+v", h) + } + writer, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("catalog")) + if claimErr != nil { + t.Fatalf("confirmed detached catalog still busy: %v", claimErr) + } + if err := m.Release(t.Context(), writer.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } +} + +type managedCancelEngine struct { + *managedTestEngine + entered chan struct{} +} + +func (e *managedCancelEngine) DiskAttach(ctx context.Context, name string, spec engine.VolumeSpec) error { + if spec.Name == "catalog" { + close(e.entered) + <-ctx.Done() + return ctx.Err() + } + return e.managedTestEngine.DiskAttach(ctx, name, spec) +} + +func TestManagedReleaseCancelsExistingVolumeRunAndRetainsFullUnion(t *testing.T) { + cfg := managedConfig(t) + cfg.Volumes = []config.VolumeSpec{{Name: "catalog", Path: writeVolumeImage(t, "catalog.img", "catalog"), Writable: true}} + e := &managedCancelEngine{managedTestEngine: newManagedEngine(), entered: make(chan struct{})} + m, openErr := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if openErr != nil { + t.Fatal(openErr) + } + t.Cleanup(func() { m.Close() }) + m.managed.process = e.identity + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + createManagedTest(t, m, "data") + sb, openErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if openErr != nil { + t.Fatal(openErr) + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, err := m.PatchVolumes(ctx, sb.ID, "", types.VolumePatch{Add: []types.Volume{{Name: "catalog"}}}) + done <- err + }() + <-e.entered + e.falseRemove.Store(true) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err == nil { + t.Fatal("surviving VMM reported removed") + } + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("mutation cancellation: %v", err) + } + case <-time.After(time.Second): + t.Fatal("release did not cancel the live volume run") + } + h, ok := m.managedHold(sb.ID) + if !ok || h.State != holdRetiring || len(h.Refs) != 2 { + t.Fatalf("retirement lost possible device references: %+v", h) + } + _, openErr = m.ClaimProvision(t.Context(), testKey, managedOptions("catalog")) + if !errors.Is(openErr, ErrVolumeBusy) { + t.Fatalf("uncertain catalog writer admitted: %v", openErr) + } + e.falseRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if _, held := m.managedHold(sb.ID); held { + t.Fatal("hold retained after confirmed VMM removal") + } +} + +func TestManagedQuarantineStillRemovesVMWhenRetirementCannotCommit(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + m.managed.commitFile = func(string, []byte) error { return errors.New("retirement unavailable") } + if m.quarantineClaim(t.Context(), sb) { + t.Fatal("uncommitted retirement reported fully cleaned") + } + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || present { + t.Fatalf("quarantined VM remains alive: %v %v", present, err) + } + current, ok := m.Sandbox(sb.ID) + if !ok || current.Failed == "" { + t.Fatalf("quarantined row remained serviceable: %+v", current) + } + if _, held := m.managedHold(sb.ID); !held { + t.Fatal("failed retirement discarded disk ownership") + } +} diff --git a/sandboxd/pool/managed_live.go b/sandboxd/pool/managed_live.go new file mode 100644 index 00000000..7e5396c5 --- /dev/null +++ b/sandboxd/pool/managed_live.go @@ -0,0 +1,32 @@ +package pool + +import ( + "fmt" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func (m *Manager) managedVolumesHeld(id string) ([]types.Volume, bool) { + h, ok := m.managedHold(id) + if !ok { + return nil, false + } + volumes := make([]types.Volume, len(h.Refs)) + for i, ref := range h.Refs { + volumes[i] = ref.Volume + } + return volumes, true +} + +func (m *Manager) validateManagedTarget(id string, target []types.Volume) error { + h, ok := m.managedHold(id) + if !ok { + return nil + } + for _, ref := range h.Refs { + if ref.ID != "" && volumeIndex(target, ref.Volume.Name) < 0 { + return fmt.Errorf("%w: managed volumes cannot be removed live", ErrBadVolume) + } + } + return nil +} diff --git a/sandboxd/pool/managed_recovery.go b/sandboxd/pool/managed_recovery.go new file mode 100644 index 00000000..00f3e6d6 --- /dev/null +++ b/sandboxd/pool/managed_recovery.go @@ -0,0 +1,92 @@ +package pool + +import ( + "context" + "errors" + "maps" + "os" + "strings" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func (m *Manager) recoverManagedOrphans(ctx context.Context, claims map[string]*types.Sandbox, live map[string]types.VMRecord, owned map[string]bool) error { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + if len(r.data.Volumes) == 0 && len(r.data.Orphans) == 0 { + r.mu.Unlock() + return nil + } + orphans := maps.Clone(r.data.Orphans) + if orphans == nil { + orphans = map[string]processIdentity{} + } + r.mu.Unlock() + known := maps.Clone(owned) + for _, sb := range claims { + known[sb.VMName] = true + } + for name, rec := range live { + if !strings.HasPrefix(name, vmPrefix) || known[name] { + continue + } + if rec.State == vmStateCreating { + outcome, err := m.eng.ReconcileStaleCreate(context.WithoutCancel(ctx), name) + if err != nil { + return managedStorageError() + } + switch outcome { + case engine.StaleCreateCollected, engine.StaleCreateNotFound: + delete(live, name) + continue + default: + return managedStorageError() + } + } + if _, tracked := orphans[name]; tracked { + continue + } + identity, err := r.process(rec.PID) + if (rec.PID > 0 && errors.Is(err, os.ErrNotExist)) || (rec.PID == 0 && rec.State == "stopped") { + _ = m.eng.Remove(context.WithoutCancel(ctx), name) + if !m.confirmGone(ctx, name) { + return managedStorageError() + } + delete(live, name) + continue + } + if err != nil { + return managedStorageError() + } + orphans[name] = identity + } + if len(orphans) == 0 { + return nil + } + r.mu.Lock() + r.data.Orphans = maps.Clone(orphans) + err := r.commit() + r.mu.Unlock() + if err != nil { + return err + } + for name, p := range orphans { + _ = m.eng.Remove(context.WithoutCancel(ctx), name) + if !m.confirmGone(ctx, name) || !r.processGone(p) { + return managedStorageError() + } + r.mu.Lock() + delete(r.data.Orphans, name) + err := r.commit() + r.mu.Unlock() + if err != nil { + return err + } + delete(live, name) + } + return nil +} diff --git a/sandboxd/pool/managed_recovery_regression_test.go b/sandboxd/pool/managed_recovery_regression_test.go new file mode 100644 index 00000000..8bc9d66d --- /dev/null +++ b/sandboxd/pool/managed_recovery_regression_test.go @@ -0,0 +1,225 @@ +package pool + +import ( + "errors" + "os" + "path/filepath" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestManagedQuarantineRemovalSurvivesFailedRetirementRestart(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + for _, published := range []bool{false, true} { + name := "async" + if syncClaims { + name = "sync" + } + if published { + name += "/published" + } else { + name += "/unpublished" + } + t.Run(name, func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + sb.Key.Net = types.NetEgress + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + e.falseRemove.Store(true) + m.managed.commitFile = func(path string, raw []byte) error { + if published { + if err := writeVolumeSnapshot(path, raw); err != nil { + return err + } + } + return errors.New("retirement persistence fault") + } + if m.quarantineClaim(t.Context(), sb) { + t.Fatal("ineffective removal reported success") + } + waitFor(t, m.store.synced) + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, claimErr, 409) + e.falseRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || present { + t.Fatalf("quarantined VM survived removal retry after restart: present=%v err=%v", present, err) + } + replacement, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatalf("confirmed removal did not free managed disk: %v", claimErr) + } + if err := m.Release(t.Context(), replacement.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + }) + } + } +} + +func TestManagedOrdinaryFailedClaimRetainsOwnershipOnRestart(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + sb.Failed = "ordinary failure requiring explicit release" + if err := m.store.commit(m.store.set(sb)); err != nil { + t.Fatal(err) + } + if err := m.Close(); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + m.retryRemovals(t.Context()).Wait() + if _, present, err := e.Inspect(t.Context(), sb.VMName); err != nil || !present { + t.Fatalf("ordinary failed claim removed: %v %v", present, err) + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, claimErr, 409) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } +} + +func TestManagedStartupCleansDeadOrphansAndRefusesUnknownProcess(t *testing.T) { + for _, tc := range []struct { + name string + pid int + stopped bool + processError error + allow bool + ineffectiveRemove bool + }{ + {name: "stopped-absent-pid", pid: 12345, stopped: true, processError: os.ErrNotExist, allow: true}, + {name: "running-absent-pid", pid: 12345, processError: os.ErrNotExist, allow: true}, + {name: "stopped-zero-pid", stopped: true, allow: true}, + {name: "running-zero-pid"}, + {name: "unreadable-pid", pid: 12345, stopped: true, processError: os.ErrPermission}, + {name: "ineffective-remove", pid: 12345, stopped: true, processError: os.ErrNotExist, ineffectiveRemove: true}, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + createManagedTest(t, m, "data") + const name = "sbx-stale-orphan" + e.mu.Lock() + e.vms[name] = "/fake/vsock" + e.stopped[name] = tc.stopped + e.pids[name] = tc.pid + e.mu.Unlock() + if tc.processError != nil { + m.managed.process = func(int) (processIdentity, error) { return processIdentity{}, tc.processError } + } else { + m.managed.process = readProcessIdentity + } + e.falseRemove.Store(tc.ineffectiveRemove) + reconcileErr := m.Reconcile(t.Context()) + if tc.allow { + if reconcileErr != nil { + t.Fatalf("dead orphan blocked startup: %v", reconcileErr) + } + if _, present, err := e.Inspect(t.Context(), name); err != nil || present { + t.Fatalf("dead orphan survived cleanup: %v %v", present, err) + } + } else { + requireVolumeStatus(t, reconcileErr, 503) + } + if tc.processError == os.ErrPermission || (!tc.stopped && tc.pid == 0) { + e.mu.Lock() + defer e.mu.Unlock() + if len(e.removes) != 0 { + t.Fatalf("unproven process removed: %v", e.removes) + } + } + }) + } +} + +func TestManagedStoreRejectsSweepOwnedPathsBeforeCreatingFiles(t *testing.T) { + for _, rel := range []string{"readers", "readers/managed", "fork-persistent", "fork-persistent/managed", "goldens", "goldens/managed"} { + t.Run(rel, func(t *testing.T) { + cfg := managedConfig(t) + cfg.ManagedVolumeDir = filepath.Join(cfg.DataDir, rel) + m, err := NewManager(t.Context(), cfg, newManagedEngine(), testSecrets(t), testBackends(t, cfg)) + if err == nil { + m.Close() + t.Fatal("accepted managed store inside startup cleanup namespace") + } + if _, err := os.Stat(cfg.ManagedVolumeDir); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("unsafe store modified before rejection: %v", err) + } + }) + } +} + +func TestManagedStorePreservesBackingWhenCleanupAliasesDisks(t *testing.T) { + for _, namespace := range []string{"readers", "goldens"} { + t.Run(namespace, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + v := createManagedTest(t, m, "data") + backing := m.managed.path(volumeID(v.ID)) + f, openErr := os.OpenFile(backing, os.O_WRONLY, 0) + if openErr != nil { + t.Fatal(openErr) + } + _, writeErr := f.WriteAt([]byte("preserved"), 4096) + if err := errors.Join(writeErr, f.Close(), m.Close()); err != nil { + t.Fatal(err) + } + cleanup := filepath.Join(cfg.DataDir, namespace) + if err := os.Remove(cleanup); err != nil && !errors.Is(err, os.ErrNotExist) { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(cfg.ManagedVolumeDir, "disks"), cleanup); err != nil { + t.Fatal(err) + } + next, openErr := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if openErr == nil { + t.Error("constructor accepted cleanup alias into managed backing directory") + if err := next.Reconcile(t.Context()); err != nil { + t.Errorf("reconcile: %v", err) + } + next.Close() + } + f, openErr = os.Open(backing) + if openErr != nil { + t.Fatalf("managed backing lost: %v", openErr) + } + defer f.Close() + marker := make([]byte, len("preserved")) + if _, err := f.ReadAt(marker, 4096); err != nil || string(marker) != "preserved" { + t.Fatalf("managed bytes changed: %q %v", marker, err) + } + }) + } +} diff --git a/sandboxd/pool/managed_startup_boundary_test.go b/sandboxd/pool/managed_startup_boundary_test.go new file mode 100644 index 00000000..e438e0f6 --- /dev/null +++ b/sandboxd/pool/managed_startup_boundary_test.go @@ -0,0 +1,102 @@ +package pool + +import ( + "errors" + "os" + "path/filepath" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/engine" +) + +func TestManagedStartupCollectsUnclaimedCreatingVMBeforeProcessProof(t *testing.T) { + for _, tc := range []struct { + name string + outcome engine.StaleCreateOutcome + collector error + available bool + }{ + {name: "collected", outcome: engine.StaleCreateCollected, available: true}, + {name: "not-found", outcome: engine.StaleCreateNotFound, available: true}, + {name: "busy", outcome: engine.StaleCreateBusy}, + {name: "not-creating", outcome: engine.StaleCreateNotCreating}, + {name: "collector-error", collector: errors.New("collector unavailable")}, + } { + t.Run(tc.name, func(t *testing.T) { + e := newManagedEngine() + m := openManagedTest(t, managedConfig(t), e) + createManagedTest(t, m, "data") + name := "sbx-unclaimed-create" + e.mu.Lock() + e.vms[name] = "/fake/vsock" + e.creating[name] = true + e.staleOutcome = tc.outcome + e.staleErr = tc.collector + e.mu.Unlock() + + err := m.Reconcile(t.Context()) + if tc.available { + if err != nil { + t.Fatalf("proven stale create blocked startup: %v", err) + } + } else { + requireVolumeStatus(t, err, 503) + } + e.mu.Lock() + defer e.mu.Unlock() + if len(e.staleReconciles) != 1 || e.staleReconciles[0] != name { + t.Fatalf("collector calls=%v, want %s", e.staleReconciles, name) + } + if len(e.removes) != 0 { + t.Fatalf("unsafe Remove fallback called: %v", e.removes) + } + _, present := e.vms[name] + if present == tc.available { + t.Fatalf("creating VM present=%v after outcome %q", present, tc.outcome) + } + }) + } +} + +func TestManagedDirectoryRequiresTrustedAncestorOwner(t *testing.T) { + if os.Geteuid() != 0 { + t.Skip("fixture needs root to create a root-owned child below a different owner") + } + for _, tc := range []struct { + name string + owner int + sticky bool + allow bool + }{ + {name: "untrusted-0755", owner: 65534}, + {name: "untrusted-sticky", owner: 65534, sticky: true}, + {name: "root-0755", owner: 0, allow: true}, + {name: "root-sticky", owner: 0, sticky: true, allow: true}, + } { + t.Run(tc.name, func(t *testing.T) { + parent := filepath.Join(t.TempDir(), "parent") + if err := os.Mkdir(parent, 0o700); err != nil { + t.Fatal(err) + } + mode := os.FileMode(0o755) + if tc.sticky { + mode = 0o777 | os.ModeSticky + } + if err := os.Chmod(parent, mode); err != nil { + t.Fatal(err) + } + managed := filepath.Join(parent, "managed") + if err := os.Mkdir(managed, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Chown(parent, tc.owner, tc.owner); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chown(parent, 0, 0) }) + err := privateVolumeDir(managed) + if (err == nil) != tc.allow { + t.Fatalf("trusted=%v, validation error=%v", tc.allow, err) + } + }) + } +} diff --git a/sandboxd/pool/managed_volume.go b/sandboxd/pool/managed_volume.go new file mode 100644 index 00000000..01d4fe8f --- /dev/null +++ b/sandboxd/pool/managed_volume.go @@ -0,0 +1,411 @@ +package pool + +import ( + "context" + "errors" + "os" + "path/filepath" + "slices" + "strings" + "syscall" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +// VolumeError carries only caller-safe details. Storage paths and process IDs +// remain in the registry and never become API diagnostics. +const volumeBusyCode = "volume_busy" + +type VolumeError struct { + Status int `json:"-"` + Message string `json:"error"` + Code string `json:"code"` + Volume string `json:"volume,omitempty"` + Holders []types.VolumeHolder `json:"holders,omitempty"` +} + +func (e *VolumeError) Error() string { return e.Message } +func (e *VolumeError) Is(target error) bool { + return target == ErrVolumeBusy && e.Code == volumeBusyCode +} + +func volumeError(status int, code, message string) *VolumeError { + return &VolumeError{Status: status, Code: code, Message: message} +} + +func managedStorageError() *VolumeError { + return volumeError(503, "volume_storage_unavailable", "volume storage requires recovery; retry after recovery") +} + +func managedNotFound() *VolumeError { return volumeError(404, "volume_not_found", "unknown volume") } + +func (m *Manager) CreateVolume(ctx context.Context, name string, req types.VolumeCreate, tenant string) (types.ManagedVolumeInfo, bool, error) { + if !types.ValidVolumeName(name) || req.SizeGiB < 1 || req.SizeGiB > 1024 { + return types.ManagedVolumeInfo{}, false, volumeError(400, "invalid_volume", "invalid volume name or size_gib (1 through 1024)") + } + if tenant != "" && req.Tenant != "" && tenant != req.Tenant { + return types.ManagedVolumeInfo{}, false, volumeError(403, "volume_owner_forbidden", "tenant cannot select another volume owner") + } + owner := req.Tenant + if tenant != "" { + owner = tenant + } + if !types.NameRe.MatchString(owner) { + return types.ManagedVolumeInfo{}, false, volumeError(400, "invalid_volume_owner", "an existing tenant owner is required") + } + if _, ok, err := m.tenants.Lookup(ctx, owner); err != nil { + return types.ManagedVolumeInfo{}, false, managedStorageError() + } else if !ok { + return types.ManagedVolumeInfo{}, false, volumeError(400, "invalid_volume_owner", "an existing tenant owner is required") + } + r := m.managed + if r == nil { + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + if _, ok := m.volumes[name]; ok { + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_name_conflict", "volume name is unavailable") + } + v, exists := r.data.Volumes[name] + if exists && tenant != "" && v.Tenant != tenant { + return types.ManagedVolumeInfo{}, false, managedNotFound() + } + size := req.SizeGiB << 30 + if exists && (v.Size != size || v.Tenant != owner) { + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_create_conflict", "volume already exists with different size or owner") + } + if !exists { + v = managedVolume{ID: volumeID("vol_" + randHex(16)), Name: name, Tenant: owner, Node: r.node, Size: size, State: volumeCreating, CreatedAt: time.Now().UTC()} + r.data.Volumes[name] = v + if err := r.commit(); err != nil { + return types.ManagedVolumeInfo{}, false, err + } + } + switch v.State { + case volumeCreating: + if err := r.completeCreate(v); err != nil { + return types.ManagedVolumeInfo{}, false, err + } + v = r.data.Volumes[name] + case volumeReady: + if err := r.verify(v); err != nil { + return types.ManagedVolumeInfo{}, false, r.markRecovery(v) + } + case volumeDeleting: + return types.ManagedVolumeInfo{}, false, volumeError(409, "volume_deleting", "volume deletion is pending") + default: + return types.ManagedVolumeInfo{}, false, managedStorageError() + } + return r.info(v), !exists, nil +} + +func (r *volumeRegistry) completeCreate(v managedVolume) error { + if err := r.verifyDirectories(); err != nil { + return managedStorageError() + } + path := r.path(v.ID) + f, err := openPrivateFile(path, syscall.O_CREAT|syscall.O_EXCL|syscall.O_RDWR) + if err != nil { + if errors.Is(err, os.ErrExist) { + return volumeError(503, "volume_create_incomplete", "incomplete volume allocation requires operator quarantine before retry") + } + return managedStorageError() + } + if err = f.Truncate(v.Size); err == nil { + err = f.Sync() + } + st, statErr := f.Stat() + closeErr := f.Close() + if err == nil { + err = errors.Join(statErr, closeErr, syncDir(r.dir+"/disks")) + } + if err != nil { + return managedStorageError() + } + raw := st.Sys().(*syscall.Stat_t) + v.Backing = backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()} + v.State = volumeReady + r.data.Volumes[v.Name] = v + return r.commit() +} + +func (r *volumeRegistry) markRecovery(v managedVolume) error { + v.State = volumeRecovery + r.data.Volumes[v.Name] = v + if err := r.commit(); err != nil { + return err + } + return managedStorageError() +} + +func (m *Manager) GetVolume(name, tenant string) (types.ManagedVolumeInfo, error) { + r := m.managed + if r == nil { + return types.ManagedVolumeInfo{}, managedNotFound() + } + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return types.ManagedVolumeInfo{}, managedStorageError() + } + v, ok := r.data.Volumes[name] + if !ok || (tenant != "" && v.Tenant != tenant) { + return types.ManagedVolumeInfo{}, managedNotFound() + } + if v.State == volumeReady && r.verify(v) != nil { + if err := r.markRecovery(v); r.fault != nil { + return types.ManagedVolumeInfo{}, err + } + v = r.data.Volumes[name] + } + return r.info(v), nil +} + +func (m *Manager) DeleteVolume(ctx context.Context, name, expected, tenant string) error { + if err := ctx.Err(); err != nil { + return err + } + r := m.managed + if r == nil { + return managedNotFound() + } + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return managedStorageError() + } + if expected == "" { + return volumeError(428, "volume_precondition_required", "If-Match must identify the volume") + } + id := volumeID(expected) + if t, ok := r.data.Deleted[id]; ok && t.Name == name && (tenant == "" || tenant == t.Tenant) { + return nil + } + v, ok := r.data.Volumes[name] + if !ok || (tenant != "" && v.Tenant != tenant) { + return managedNotFound() + } + if v.ID != id { + return volumeError(412, "volume_precondition_failed", "volume identity does not match") + } + if holders := r.holders(v.ID); len(holders) > 0 { + return &VolumeError{Status: 409, Code: volumeBusyCode, Message: "volume is held by a sandbox", Volume: name, Holders: holders} + } + if err := r.verifyDirectories(); err != nil { + return managedStorageError() + } + path := r.path(id) + actual, err := inspectBacking(path) + missing := errors.Is(err, os.ErrNotExist) + if missing && v.State != volumeCreating && v.State != volumeDeleting { + return r.markRecovery(v) + } + if !missing && (err != nil || v.Backing.Inode == 0 || actual != v.Backing) { + return volumeError(503, "volume_backing_mismatch", "volume backing requires operator recovery") + } + if v.State != volumeDeleting { + v.State = volumeDeleting + r.data.Volumes[name] = v + if err := r.commit(); err != nil { + return err + } + } + if !missing { + if err := os.Remove(path); err != nil { + return managedStorageError() + } + } + if err := syncDir(r.dir + "/disks"); err != nil { + return managedStorageError() + } + if err := clearVolumeDirty(path); err != nil { + return managedStorageError() + } + r.data.Deleted[id] = volumeTombstone{ID: id, Name: name, Tenant: v.Tenant} + delete(r.data.Volumes, name) + return r.commit() +} + +func (r *volumeRegistry) holders(id volumeID) []types.VolumeHolder { + holders := []types.VolumeHolder{} + for _, h := range r.data.Holds { + for _, ref := range h.Refs { + if ref.ID == id { + mode := "ro" + if ref.Volume.RW() { + mode = "rw" + } + holders = append(holders, types.VolumeHolder{SandboxID: h.SandboxID, Mode: mode, State: string(h.State)}) + } + } + } + slices.SortFunc(holders, func(a, b types.VolumeHolder) int { return strings.Compare(a.SandboxID, b.SandboxID) }) + return holders +} + +func (r *volumeRegistry) info(v managedVolume) types.ManagedVolumeInfo { + return types.ManagedVolumeInfo{Name: v.Name, DefaultMount: types.DefaultVolumeMount(v.Name), SizeBytes: v.Size, Available: v.State == volumeReady, Nodes: 1, Writable: true, Kind: "managed", ID: string(v.ID), Tenant: v.Tenant, Node: v.Node, State: string(v.State), CreatedAt: v.CreatedAt, Holders: r.holders(v.ID)} +} + +// ManagedVolumeStatus lets listing fail clearly rather than presenting a corrupt +// registry as an empty managed catalog. A disabled store has no catalog cost. +func (m *Manager) ManagedVolumeStatus() error { + if m.managed == nil { + return nil + } + m.managed.mu.Lock() + defer m.managed.mu.Unlock() + if m.managed.fault != nil { + return managedStorageError() + } + return nil +} + +func (m *Manager) managedInfos(tenant string) []types.VolumeInfo { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return nil + } + infos := []types.VolumeInfo{} + for _, v := range r.data.Volumes { + if tenant == "" || tenant == v.Tenant { + info := r.info(v) + if v.State == volumeReady && r.verify(v) != nil { + info.State = string(volumeRecovery) + info.Available = false + } + infos = append(infos, info.VolumeInfo) + } + } + return infos +} + +func (m *Manager) lookupManaged(name string) (catalogVolume, bool, error) { + r := m.managed + if r == nil { + return catalogVolume{}, false, nil + } + r.mu.Lock() + defer r.mu.Unlock() + if r.fault != nil { + return catalogVolume{}, false, managedStorageError() + } + v, ok := r.data.Volumes[name] + if !ok { + return catalogVolume{}, false, nil + } + entry := catalogVolume{disk: engine.VolumeSpec{Name: name, Path: r.path(v.ID), DirectIO: types.DirectIOOff}, tenants: []string{v.Tenant}, writable: true, managedID: v.ID} + if v.State != volumeReady { + return entry, true, managedStorageError() + } + if err := r.verify(v); err != nil { + return entry, true, r.markRecovery(v) + } + return entry, true, nil +} + +func (m *Manager) managedHold(id string) (volumeHold, bool) { + r := m.managed + if r == nil { + return volumeHold{}, false + } + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + h.Refs = slices.Clone(h.Refs) + return h, ok +} + +func (m *Manager) managedHoldByVM(name string) (volumeHold, bool) { + r := m.managed + if r == nil { + return volumeHold{}, false + } + r.mu.Lock() + defer r.mu.Unlock() + for _, h := range r.data.Holds { + if h.VMName == name { + h.Refs = slices.Clone(h.Refs) + return h, true + } + } + return volumeHold{}, false +} + +func (m *Manager) retireManaged(id string) error { + r := m.managed + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + h, ok := r.data.Holds[id] + if !ok { + return nil + } + if r.fault != nil { + return managedStorageError() + } + if h.State == holdRetiring { + return nil + } + h.State = holdRetiring + r.data.Holds[id] = h + return r.commit() +} + +func (m *Manager) managedRefPath(id, name string) (string, bool) { + h, ok := m.managedHold(id) + if !ok { + return "", false + } + for _, ref := range h.Refs { + if ref.Volume.Name == name { + return ref.Path, true + } + } + return "", false +} + +func (m *Manager) ManagedVolumesEnabled() bool { return m.managed != nil } + +func (m *Manager) managedCatalogAlias(path string) bool { + r := m.managed + if r == nil { + return false + } + clean := filepath.Clean(path) + if canonical, err := filepath.EvalSymlinks(path); err == nil { + clean = canonical + } + if clean == r.dir || strings.HasPrefix(clean, r.dir+string(filepath.Separator)) { + return true + } + info, err := os.Stat(path) + if err != nil { + return false + } + identity := info.Sys().(*syscall.Stat_t) + r.mu.Lock() + defer r.mu.Unlock() + for _, v := range r.data.Volumes { + if v.Backing.Inode != 0 && identity.Ino == v.Backing.Inode && backingDeviceID(identity.Dev) == v.Backing.Device { + return true + } + if backing, err := os.Lstat(r.path(v.ID)); err == nil && os.SameFile(info, backing) { + return true + } + } + return false +} diff --git a/sandboxd/pool/managed_volume_test.go b/sandboxd/pool/managed_volume_test.go new file mode 100644 index 00000000..04c6715a --- /dev/null +++ b/sandboxd/pool/managed_volume_test.go @@ -0,0 +1,595 @@ +package pool + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "syscall" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/tenants" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type managedTestEngine struct { + *fakeEngine + next atomic.Int64 + falseRemove atomic.Bool + missingInventory atomic.Bool + processLives atomic.Bool + processUnknown atomic.Bool + attachEntered chan struct{} + attachResume chan struct{} +} + +func newManagedEngine() *managedTestEngine { return &managedTestEngine{fakeEngine: newFakeEngine()} } + +func (e *managedTestEngine) RunCold(ctx context.Context, name string, key types.PoolKey, storage string) (types.VMRecord, error) { + e.mu.Lock() + e.pids[name] = 1000 + int(e.next.Add(1)) + e.mu.Unlock() + return e.fakeEngine.RunCold(ctx, name, key, storage) +} + +func (e *managedTestEngine) Remove(ctx context.Context, name string) error { + if e.falseRemove.Load() { + return nil + } + return e.fakeEngine.Remove(ctx, name) +} + +func (e *managedTestEngine) Inspect(ctx context.Context, name string) (types.VMRecord, bool, error) { + if e.missingInventory.Load() { + return types.VMRecord{}, false, nil + } + return e.fakeEngine.Inspect(ctx, name) +} + +func (e *managedTestEngine) DiskAttach(ctx context.Context, name string, spec engine.VolumeSpec) error { + if e.attachEntered != nil { + e.attachEntered <- struct{}{} + <-e.attachResume + } + return e.fakeEngine.DiskAttach(ctx, name, spec) +} + +func (e *managedTestEngine) identity(pid int) (processIdentity, error) { + if e.processUnknown.Load() { + return processIdentity{}, os.ErrPermission + } + e.mu.Lock() + defer e.mu.Unlock() + for name, p := range e.pids { + if p == pid { + if _, ok := e.vms[name]; ok || e.processLives.Load() { + return processIdentity{PID: pid, Start: uint64(pid), BootID: "test-boot"}, nil + } + } + } + return processIdentity{}, os.ErrNotExist +} + +func managedConfig(t *testing.T) *config.Config { + t.Helper() + dir := t.TempDir() + return &config.Config{DataDir: filepath.Join(dir, "data"), ManagedVolumeDir: filepath.Join(dir, "managed"), APIToken: "root", Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 1}}} +} + +func openManagedTest(t *testing.T, cfg *config.Config, e *managedTestEngine) *Manager { + t.Helper() + m, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { m.Close() }) + m.managed.process = e.identity + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t1", Token: "t1-token"}); err != nil { + t.Fatal(err) + } + if err := m.PutTenant(t.Context(), tenants.TenantSpec{Name: "t2", Token: "t2-token"}); err != nil { + t.Fatal(err) + } + return m +} + +func createManagedTest(t *testing.T, m *Manager, name string) types.ManagedVolumeInfo { + t.Helper() + v, created, err := m.CreateVolume(t.Context(), name, types.VolumeCreate{SizeGiB: 1}, "t1") + if err != nil || !created { + t.Fatalf("create: created=%v err=%v", created, err) + } + return v +} + +func managedOptions(names ...string) ClaimOptions { + o := ClaimOptions{Tenant: "t1"} + for _, name := range names { + o.Volumes = append(o.Volumes, types.Volume{Name: name, Mode: types.VolumeModeRW, AttachOnly: true}) + } + return o +} + +func requireVolumeStatus(t *testing.T, err error, status int) *VolumeError { + t.Helper() + var v *VolumeError + if !errors.As(err, &v) || v.Status != status { + t.Fatalf("error=%v, want volume status %d", err, status) + } + return v +} + +func TestManagedCreateReplaySparseOwnershipAndDeleteIdentity(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + st, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if st.Size() != 1<<30 || st.Sys().(*syscall.Stat_t).Blocks*512 >= st.Size() { + t.Fatal("backing is not a sparse GiB raw image") + } + f, err := os.OpenFile(path, os.O_WRONLY, 0o600) + if err != nil { + t.Fatal(err) + } + _, err = f.WriteAt([]byte("preserved"), 4096) + if err != nil { + t.Fatal(err) + } + f.Close() + if v.Kind != "managed" || v.State != "ready" || v.Tenant != "t1" || v.Node == "" || v.CreatedAt.IsZero() { + t.Fatalf("projection=%+v", v) + } + _, _, err = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 2}, "t1") + requireVolumeStatus(t, err, 409) + _, _, err = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t2") + requireVolumeStatus(t, err, 404) + _, err = m.GetVolume("data", "t2") + requireVolumeStatus(t, err, 404) + m.Close() + m = openManagedTest(t, cfg, e) + repeat, created, err := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1, Tenant: "t1"}, "t1") + if err != nil || created || repeat.ID != v.ID { + t.Fatalf("replay=%+v created=%v err=%v", repeat, created, err) + } + f, err = os.Open(path) + if err != nil { + t.Fatal(err) + } + buf := make([]byte, 9) + _, err = f.ReadAt(buf, 4096) + f.Close() + if err != nil || string(buf) != "preserved" { + t.Fatalf("replay changed bytes: %q %v", buf, err) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", "", "t1"), 428) + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", "wrong", "t1"), 412) + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + newer := createManagedTest(t, m, "data") + if newer.ID == v.ID { + t.Fatal("identity reused") + } + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t2"), 404) + if _, err := os.Stat(m.managed.path(volumeID(newer.ID))); err != nil { + t.Fatal("stale delete removed replacement", err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if got, err := m.GetVolume("data", "t1"); err != nil || got.ID != newer.ID { + t.Fatalf("restart=%+v %v", got, err) + } +} + +func TestManagedConcurrentCreateAndWriterAdmission(t *testing.T) { + m := openManagedTest(t, managedConfig(t), newManagedEngine()) + var created atomic.Int32 + var wg sync.WaitGroup + ids := make(chan string, 12) + for range 12 { + wg.Go(func() { + v, wasCreated, err := m.CreateVolume(t.Context(), "race", types.VolumeCreate{SizeGiB: 1}, "t1") + if err != nil { + t.Error(err) + return + } + if wasCreated { + created.Add(1) + } + ids <- v.ID + }) + } + wg.Wait() + close(ids) + first := "" + for id := range ids { + if first == "" { + first = id + } + if id != first { + t.Error("concurrent identity changed") + } + } + if created.Load() != 1 { + t.Fatalf("created=%d", created.Load()) + } + successes := make(chan *types.Sandbox, 8) + for range 8 { + wg.Go(func() { + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("race")) + if err == nil { + successes <- sb + return + } + var ve *VolumeError + if !errors.As(err, &ve) || ve.Status != 409 || len(ve.Holders) != 1 || ve.Holders[0].SandboxID == "" { + t.Errorf("conflict=%+v", err) + } + }) + } + wg.Wait() + close(successes) + var winner *types.Sandbox + count := 0 + for sb := range successes { + winner = sb + count++ + } + if count != 1 { + t.Fatalf("writers=%d", count) + } + if err := m.Release(t.Context(), winner.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + if err := m.DeleteVolume(t.Context(), "race", first, "t1"); err != nil { + t.Fatal(err) + } +} + +func TestManagedRetirementSurvivesRestartAndStaleClaim(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + t.Run(fmt.Sprint(syncClaims), func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + sb, claimErr := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if claimErr != nil { + t.Fatal(claimErr) + } + e.falseRemove.Store(true) + if err := m.retireManaged(sb.ID); err != nil { + t.Fatal(err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if _, ok := m.Sandbox(sb.ID); ok { + t.Fatal("retiring stale claim was adopted") + } + _, claimErr = m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + busy := requireVolumeStatus(t, claimErr, 409) + if busy.Holders[0].State != "retiring" || busy.Holders[0].SandboxID != sb.ID { + t.Fatalf("holder=%+v", busy) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + e.falseRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + if len(m.managed.data.Holds) != 0 { + t.Fatal("retirement leaked a hold") + } + }) + } +} + +func TestManagedActiveClaimAdoptionAndProcessProof(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + t.Run(fmt.Sprint(syncClaims), func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + sb, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + if err != nil { + t.Fatal(err) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + adopted, ok := m.Sandbox(sb.ID) + if !ok || adopted.Tenant != "t1" { + t.Fatal("writer not adopted") + } + e.processLives.Store(true) + if err := m.Release(t.Context(), sb.ID, Cred{Operator: true}); err == nil { + t.Fatal("absence alone released a live process") + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + e.processUnknown.Store(true) + m.retryRemovals(t.Context()).Wait() + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + e.processUnknown.Store(false) + e.processLives.Store(false) + m.retryRemovals(t.Context()).Wait() + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + }) + } +} + +func TestManagedPreclaimCrashRetainsOwnership(t *testing.T) { + for _, syncClaims := range []bool{false, true} { + for _, boundary := range []string{"reserved", "bound", "attached", "mounted"} { + t.Run(fmt.Sprintf("%s/sync=%v", boundary, syncClaims), func(t *testing.T) { + cfg := managedConfig(t) + cfg.SyncClaims = syncClaims + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + options := managedOptions("data") + if boundary == "mounted" { + options.Volumes[0].AttachOnly = false + } + specs, _, err := m.admitOptions(t.Context(), testKey, options) + if err != nil { + t.Fatal(err) + } + if boundary != "reserved" { + rec, err := e.RunCold(t.Context(), "sbx-interrupted", testKey, "") + if err != nil { + t.Fatal(err) + } + sb := &types.Sandbox{VMName: rec.Config.Name, VsockSocket: rec.VsockSocket, Key: testKey} + if err := m.bindManaged(t.Context(), sb, specs); err != nil { + t.Fatal(err) + } + var applyErr error + if boundary == "attached" || boundary == "mounted" { + applyErr = m.applyVolume(t.Context(), sb, specs[0], false) + } + if applyErr != nil { + t.Fatal(applyErr) + } + e.falseRemove.Store(true) + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if boundary != "reserved" { + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + } + e.falseRemove.Store(false) + m.retryRemovals(t.Context()).Wait() + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } + }) + } + } +} + +func TestManagedAmbiguousCommitFailsClosedAndRecovers(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + writes := 0 + m.managed.commitFile = func(path string, raw []byte) error { + writes++ + if err := writeVolumeSnapshot(path, raw); err != nil { + return err + } + return errors.New("directory sync uncertain /private/path") + } + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + ve := requireVolumeStatus(t, err, 503) + if strings.Contains(ve.Error(), "private") { + t.Fatal("storage error leaked a path") + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 503) + if writes != 1 || len(e.volumeSpecs) != 0 { + t.Fatal("mutation or attach continued after uncertain commit") + } + m.Close() + m = openManagedTest(t, cfg, e) + if err := m.Reconcile(t.Context()); err != nil { + t.Fatal(err) + } + if err := m.DeleteVolume(t.Context(), "data", v.ID, "t1"); err != nil { + t.Fatal(err) + } +} + +func TestManagedCreateUnrecordedInodeRequiresQuarantine(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + writes := 0 + m.managed.commitFile = func(path string, raw []byte) error { + writes++ + if writes == 2 { + return errors.New("ready metadata failed") + } + return writeVolumeSnapshot(path, raw) + } + _, _, createErr := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + requireVolumeStatus(t, createErr, 503) + m.Close() + m = openManagedTest(t, cfg, e) + v, createErr := m.GetVolume("data", "t1") + if createErr != nil || v.State != "creating" { + t.Fatalf("interrupted create=%+v %v", v, createErr) + } + path := m.managed.path(volumeID(v.ID)) + before, createErr := os.Stat(path) + if createErr != nil { + t.Fatal(createErr) + } + _, _, createErr = m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + ve := requireVolumeStatus(t, createErr, 503) + if ve.Code != "volume_create_incomplete" { + t.Fatal(ve) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 503) + after, createErr := os.Stat(path) + if createErr != nil || !os.SameFile(before, after) { + t.Fatal("unknown inode modified") + } + m.Close() + if err := os.Rename(path, filepath.Join(filepath.Dir(cfg.ManagedVolumeDir), "quarantine.raw")); err != nil { + t.Fatal(err) + } + m = openManagedTest(t, cfg, e) + repeat, created, createErr := m.CreateVolume(t.Context(), "data", types.VolumeCreate{SizeGiB: 1}, "t1") + if createErr != nil || created || repeat.ID != v.ID { + t.Fatalf("quarantine repair=%+v %v %v", repeat, created, createErr) + } +} + +func TestManagedCorruptAndUnsafeStorage(t *testing.T) { + for _, fault := range []string{"truncated", "version", "missing-registry", "replaced", "symlink", "missing", "permissions"} { + t.Run(fault, func(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + path := m.managed.path(volumeID(v.ID)) + m.Close() + switch fault { + case "truncated": + if err := os.WriteFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json"), []byte("{"), 0o600); err != nil { + t.Fatal(err) + } + case "version": + raw, err := os.ReadFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json")) + if err != nil { + t.Fatal(err) + } + var data volumeRegistryData + if err := json.Unmarshal(raw, &data); err != nil { + t.Fatal(err) + } + data.Version = 99 + raw, _ = json.Marshal(data) + if err := os.WriteFile(filepath.Join(cfg.ManagedVolumeDir, "registry.json"), raw, 0o600); err != nil { + t.Fatal(err) + } + case "missing-registry": + if err := os.Remove(filepath.Join(cfg.ManagedVolumeDir, "registry.json")); err != nil { + t.Fatal(err) + } + case "replaced": + if err := os.Rename(path, path+".original"); err != nil { + t.Fatal(err) + } + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0o600) + if err != nil { + t.Fatal(err) + } + if err := f.Truncate(1 << 30); err != nil { + t.Fatal(err) + } + f.Close() + case "symlink": + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + if err := os.Symlink("/etc/passwd", path); err != nil { + t.Fatal(err) + } + case "missing": + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + case "permissions": + if err := os.Chmod(path, 0o000); err != nil { + t.Fatal(err) + } + } + m = openManagedTest(t, cfg, e) + _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")) + requireVolumeStatus(t, err, 503) + if len(e.volumeSpecs) != 0 { + t.Fatal("unsafe backing attached") + } + if fault == "replaced" || fault == "symlink" || fault == "permissions" { + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 503) + } + }) + } +} + +func TestManagedStoreExclusiveAndDirectoryPermissions(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + if _, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("second daemon accepted locked store") + } + m.Close() + if err := os.Chmod(cfg.ManagedVolumeDir, 0o755); err != nil { + t.Fatal(err) + } + if _, err := NewManager(t.Context(), cfg, e, testSecrets(t), testBackends(t, cfg)); err == nil { + t.Fatal("public managed directory accepted") + } +} + +func TestManagedStalledAttachDoesNotBlockOtherClaims(t *testing.T) { + cfg := managedConfig(t) + e := newManagedEngine() + m := openManagedTest(t, cfg, e) + v := createManagedTest(t, m, "data") + unrelated, claimErr := m.ClaimProvision(t.Context(), testKey, ClaimOptions{}) + if claimErr != nil { + t.Fatal(claimErr) + } + e.attachEntered = make(chan struct{}, 1) + e.attachResume = make(chan struct{}) + done := make(chan error, 1) + go func() { _, err := m.ClaimProvision(t.Context(), testKey, managedOptions("data")); done <- err }() + <-e.attachEntered + if _, err := m.Renew(t.Context(), unrelated.ID, Cred{Operator: true}, time.Minute, ""); err != nil { + t.Fatal(err) + } + if err := m.Release(t.Context(), unrelated.ID, Cred{Operator: true}); err != nil { + t.Fatal(err) + } + got, claimErr := m.GetVolume("data", "t1") + if claimErr != nil || len(got.Holders) != 1 || got.Holders[0].State != "attaching" { + t.Fatalf("attaching projection=%+v %v", got, claimErr) + } + requireVolumeStatus(t, m.DeleteVolume(t.Context(), "data", v.ID, "t1"), 409) + close(e.attachResume) + if err := <-done; err != nil { + t.Fatal(err) + } +} diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 9e0124cb..3d49c116 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -329,6 +329,7 @@ type Manager struct { volumes map[string]catalogVolume // volumeAdmission counts the live holders of each volume name; guarded by m.mu. volumeAdmission map[string]volumeHolders + managed *volumeRegistry volumeRuns map[*types.Sandbox]*volumeRun volumeErrors map[*types.Sandbox]map[string]string @@ -438,6 +439,9 @@ type Manager struct { // NewManager builds a manager from the node config over backends the caller selected from it. func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *egress.SecretStore, b Backends) (*Manager, error) { + if err := cfg.ValidateManagedVolumeStore(); err != nil { + return nil, err + } refill := cmp.Or(cfg.RefillConcurrency, defaultRefill) m := &Manager{ eng: eng, @@ -550,6 +554,26 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg } m.warnVolumeTenants(ctx, cfg.Volumes) m.warnMissingClasses(ctx) + if cfg.ManagedVolumeDir != "" { + node, hostErr := os.Hostname() + if cfg.Mesh != nil { + node = cfg.Mesh.ID() + } + if node == "" { + return nil, fmt.Errorf("managed volume node identity unavailable: %v", hostErr) + } + var err error + m.managed, err = openVolumeRegistry(cfg.ManagedVolumeDir, node) + if err != nil { + return nil, err + } + for name, entry := range m.volumes { + _, collision := m.managed.data.Volumes[name] + if collision || m.managedCatalogAlias(entry.disk.Path) { + return nil, errors.Join(fmt.Errorf("catalog conflicts with existing managed resource"), m.managed.close()) + } + } + } return m, nil } diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index 3bfa3cf1..95564b48 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -36,7 +36,16 @@ func (m *Manager) Reconcile(ctx context.Context) error { snaps, snapsErr := m.eng.SnapshotList(ctx) imageIDs := m.imageIDs(ctx) - owned := map[string]bool{} + if m.managed != nil { + m.store.reset(claims) + } + owned, err := m.recoverManaged(ctx, claims, live) + if err != nil { + return err + } + if err := m.recoverManagedOrphans(ctx, claims, live, owned); err != nil { + return err + } referenced := map[string]bool{} var adopted []*types.Sandbox m.mu.Lock() @@ -66,7 +75,9 @@ func (m *Manager) Reconcile(ctx context.Context) error { continue } m.addClaimLocked(sb) - m.adoptVolumes(heldVolumes(sb)) + if _, managed := m.managedHold(sb.ID); !managed { + m.adoptVolumes(heldVolumes(sb)) + } owned[sb.VMName] = true referenced[sb.HibernateSnap] = true } @@ -108,21 +119,7 @@ func (m *Manager) Reconcile(ctx context.Context) error { m.retryArchiveDeletes(ctx) removed := m.sweepStaleVMs(ctx, live, owned) - // an unreferenced hibernate snapshot is an orphan; fork and golden snapshots never span a restart - if snapsErr != nil { - logger.Warnf(ctx, "snapshot sweep skipped: %v", snapsErr) - } else { - orphans := slices.DeleteFunc(snaps, func(snap string) bool { - if strings.HasPrefix(snap, hibernatePrefix) { - return referenced[snap] - } - return !strings.HasPrefix(snap, forkPrefix) && !strings.HasPrefix(snap, goldenPrefix) - }) - m.runBounded(ctx, len(orphans), func(ctx context.Context, i int) { - m.dropSnap(ctx, orphans[i]) - logger.Infof(ctx, "removed orphan snapshot %s", orphans[i]) - }).Wait() - } + m.sweepOrphanSnapshots(ctx, snaps, snapsErr, referenced) m.resyncEgress(ctx, live, removed) m.warnDeniedClaims(ctx) logger.Infof(ctx, "adopted %d claims, %d VMs live", len(m.claimed), len(live)) @@ -257,8 +254,21 @@ func (m *Manager) resyncEgress(ctx context.Context, live map[string]types.VMReco } } +const egressQuarantineFailure = "sandbox quarantined after egress recovery failure" + // quarantineClaim parks a claim whose teardown failed: out of service and queued until a remove succeeds. func (m *Manager) quarantineClaim(ctx context.Context, sb *types.Sandbox) bool { + if m.retireManaged(sb.ID) != nil { + m.mu.Lock() + sb.Failed = egressQuarantineFailure + m.cancelVolumeRun(sb) + js := m.store.set(sb) + m.mu.Unlock() + m.store.recommit(ctx, js) + td := m.quiesceVolumes(ctx, sb) + m.removeOrRetry(ctx, sb.VMName, sb.ID, td) + return false + } td := m.quiesceVolumes(ctx, sb) gone := m.removeOrRetry(ctx, sb.VMName, sb.ID, td) m.forgetClaim(ctx, sb) @@ -320,3 +330,22 @@ func (m *Manager) reclaimOrphanArchiveCks(ctx context.Context, claims map[string } } } + +func (m *Manager) sweepOrphanSnapshots(ctx context.Context, snaps []string, snapsErr error, referenced map[string]bool) { + logger := log.WithFunc("pool.Reconcile") + // an unreferenced hibernate snapshot is an orphan; fork and golden snapshots never span a restart + if snapsErr != nil { + logger.Warnf(ctx, "snapshot sweep skipped: %v", snapsErr) + } else { + orphans := slices.DeleteFunc(snaps, func(snap string) bool { + if strings.HasPrefix(snap, hibernatePrefix) { + return referenced[snap] + } + return !strings.HasPrefix(snap, forkPrefix) && !strings.HasPrefix(snap, goldenPrefix) + }) + m.runBounded(ctx, len(orphans), func(ctx context.Context, i int) { + m.dropSnap(ctx, orphans[i]) + logger.Infof(ctx, "removed orphan snapshot %s", orphans[i]) + }).Wait() + } +} diff --git a/sandboxd/pool/remove.go b/sandboxd/pool/remove.go index 75efec26..46ec818e 100644 --- a/sandboxd/pool/remove.go +++ b/sandboxd/pool/remove.go @@ -17,7 +17,17 @@ func (m *Manager) removeVM(ctx context.Context, name string) bool { m.out.ClosePrebound(name) // cancellation-immune: on a canceled ctx `cocoon vm rm` no-ops and orphans. ctx = context.WithoutCancel(ctx) + hold, managed := m.managedHoldByVM(name) + if managed { + if err := m.retireManaged(hold.SandboxID); err != nil { + _ = m.eng.Remove(ctx, name) + return false + } + } err := m.eng.Remove(ctx, name) + if managed { + return m.confirmGone(ctx, name) && m.managed.processGone(hold.Process) + } if err == nil { return true } @@ -44,8 +54,17 @@ func (m *Manager) confirmGone(ctx context.Context, name string) bool { // removeOrRetry reports whether the VM is confirmed gone; a survivor is queued for the reap tick. func (m *Manager) removeOrRetry(ctx context.Context, name, sandboxID string, td volumeTeardown) bool { + if h, ok := m.managedHoldByVM(name); ok { + td.managedID = h.SandboxID + } if m.removeVM(ctx, name) { m.finishVolumeTeardown(ctx, td) + if td.managedID != "" { + if _, held := m.managedHold(td.managedID); held { + m.queueRemoval(name, sandboxID, "", td, time.Time{}) + return false + } + } return true } m.queueRemoval(name, sandboxID, "", td, time.Time{}) @@ -102,6 +121,11 @@ func (m *Manager) retryRemoval(ctx context.Context, name string, pending pending return } m.finishRemoval(ctx, pending) + if pending.volumes.managedID != "" { + if _, held := m.managedHold(pending.volumes.managedID); held { + m.queueRemoval(name, pending.sandboxID, pending.tap, pending.volumes, time.Time{}) + } + } } func (m *Manager) finishRemoval(ctx context.Context, pending pendingRemoval) { diff --git a/sandboxd/pool/volume.go b/sandboxd/pool/volume.go index 5a1e8eed..de82875e 100644 --- a/sandboxd/pool/volume.go +++ b/sandboxd/pool/volume.go @@ -31,20 +31,24 @@ type volumeHolders struct { } type resolvedVolume struct { - disk engine.VolumeSpec - applied types.Volume + disk engine.VolumeSpec + applied types.Volume + managedID volumeID + holdID string } // volumeTeardown captures marker paths at quiesce time; a later config can re-point the entry. type volumeTeardown struct { - holds []types.Volume - clears []string + managedID string + holds []types.Volume + clears []string } type catalogVolume struct { - disk engine.VolumeSpec - tenants []string - writable bool + managedID volumeID + disk engine.VolumeSpec + tenants []string + writable bool } func (v catalogVolume) allowed(tenant string) bool { @@ -71,10 +75,12 @@ func (m *Manager) Volumes(tenant string, holders map[string]int) []types.VolumeI } infos = append(infos, info) } + infos = append(infos, m.managedInfos(tenant)...) // root can see peer-only names; a tenant needs local catalog metadata to apply the access list if tenant == "" { for name, nodes := range holders { - if _, ok := m.volumes[name]; !ok { + _, managed, _ := m.lookupManaged(name) + if _, ok := m.volumes[name]; !ok && !managed { infos = append(infos, types.VolumeInfo{ Name: name, DefaultMount: types.DefaultVolumeMount(name), @@ -109,17 +115,29 @@ func (m *Manager) VolumePlacement(key types.PoolKey, tenant string, names []stri } local := true for _, name := range names { - volume, ok := m.volumes[name] + volume, ok, err := m.lookupVolume(name) + if !ok && err != nil { + return false, err + } if !ok { if tenant == "" { local = false continue } + if m.managed != nil { + return false, managedNotFound() + } return false, ErrVolumeUnavailable } if !volume.allowed(tenant) { + if m.managed != nil { + return false, managedNotFound() + } return false, ErrVolumeUnavailable } + if err != nil { + return false, err + } if _, err := os.Stat(volume.disk.Path); err != nil { local = false } @@ -138,10 +156,19 @@ func (m *Manager) resolveVolumes(ctx context.Context, tenant string, requested [ } resolved := make([]resolvedVolume, 0, len(applied)) for _, volume := range applied { - entry, ok := m.volumes[volume.Name] + entry, ok, lookupErr := m.lookupVolume(volume.Name) + if !ok && lookupErr != nil { + return nil, lookupErr + } if !ok || !entry.allowed(tenant) { + if m.managed != nil { + return nil, managedNotFound() + } return nil, ErrVolumeUnavailable } + if lookupErr != nil { + return nil, lookupErr + } resolvedEntry, resolveErr := m.resolveCatalogVolume(ctx, volume) if resolveErr != nil { return nil, resolveErr @@ -152,7 +179,10 @@ func (m *Manager) resolveVolumes(ctx context.Context, tenant string, requested [ } func (m *Manager) resolveCatalogVolume(ctx context.Context, volume types.Volume) (resolvedVolume, error) { - entry, ok := m.volumes[volume.Name] + entry, ok, lookupErr := m.lookupVolume(volume.Name) + if lookupErr != nil { + return resolvedVolume{}, lookupErr + } if !ok { return resolvedVolume{}, ErrVolumeUnavailable } @@ -167,7 +197,17 @@ func (m *Manager) resolveCatalogVolume(ctx context.Context, volume types.Volume) disk := entry.disk disk.RW = volume.RW() volume.AttachOnly = false // past validation the empty mount carries it - return resolvedVolume{disk: disk, applied: volume}, nil + return resolvedVolume{disk: disk, applied: volume, managedID: entry.managedID}, nil +} + +func (m *Manager) lookupVolume(name string) (catalogVolume, bool, error) { + if entry, ok := m.volumes[name]; ok { + if m.managedCatalogAlias(entry.disk.Path) { + return catalogVolume{}, false, managedStorageError() + } + return entry, true, nil + } + return m.lookupManaged(name) } // confirmVolumesClean refuses a read-only claim of a dirty image; run after holds are taken. @@ -185,6 +225,9 @@ func (m *Manager) applyVolumes(ctx context.Context, sb *types.Sandbox, volumes [ if len(volumes) == 0 { return nil } + if err := m.bindManaged(ctx, sb, volumes); err != nil { + return err + } group, groupCtx := errgroup.WithContext(ctx) for _, volume := range volumes { group.Go(func() error { return m.applyVolume(groupCtx, sb, volume, false) }) @@ -198,6 +241,20 @@ func (m *Manager) applyVolumes(ctx context.Context, sb *types.Sandbox, volumes [ // applyVolume keeps one volume's steps strictly ordered; siblings overlap freely. func (m *Manager) applyVolume(ctx context.Context, sb *types.Sandbox, volume resolvedVolume, live bool) error { + if volume.managedID == "" { + if err := m.verifyCatalogBeforeAttach(sb.ID, volume); err != nil { + return err + } + } + if volume.managedID != "" { + m.managed.mu.Lock() + v, ok := m.managed.data.Volumes[volume.applied.Name] + valid := ok && v.ID == volume.managedID && m.managed.fault == nil && m.managed.verify(v) == nil + m.managed.mu.Unlock() + if !valid { + return managedStorageError() + } + } // attach-only mounts nothing, so there is no umount to verify and no marker. attachOnly := volume.applied.Mount == "" // write-ahead: the marker must be durable before any guest write can be. @@ -225,9 +282,33 @@ func (m *Manager) applyVolume(ctx context.Context, sb *types.Sandbox, volume res return nil } +func (m *Manager) verifyCatalogBeforeAttach(id string, volume resolvedVolume) error { + if m.managedCatalogAlias(volume.disk.Path) { + return managedStorageError() + } + h, managed := m.managedHold(id) + if !managed { + return nil + } + for _, ref := range h.Refs { + if ref.Volume.Name != volume.applied.Name || ref.Path != volume.disk.Path { + continue + } + identity, err := catalogBacking(volume.disk.Path) + if err != nil || identity != ref.Backing { + return managedStorageError() + } + return nil + } + return managedStorageError() +} + // quiesceVolumes unmounts a claim's writable mounts in reverse; a failed unmount keeps the marker. func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeTeardown { td := volumeTeardown{holds: heldVolumes(sb)} + if h, managed := m.managedHold(sb.ID); managed { + td.managedID = h.SandboxID + } mounts := writableMounts(sb.Volumes) if mounts == 0 { return td @@ -246,7 +327,9 @@ func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeT stuck = true continue } - if entry, ok := m.volumes[volume.Name]; ok { + if path, ok := m.managedRefPath(sb.ID, volume.Name); ok { + td.clears = append(td.clears, path) + } else if entry, ok := m.volumes[volume.Name]; ok { td.clears = append(td.clears, entry.disk.Path) } } @@ -263,6 +346,10 @@ func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeT // finishVolumeTeardown runs only once the VM is gone, when its image lock is finally released. func (m *Manager) finishVolumeTeardown(ctx context.Context, td volumeTeardown) { + if td.managedID != "" { + m.finishManaged(td.managedID, td.clears) + return + } if len(td.holds) == 0 { return } diff --git a/sandboxd/pool/volume_device_test.go b/sandboxd/pool/volume_device_test.go new file mode 100644 index 00000000..998f68a8 --- /dev/null +++ b/sandboxd/pool/volume_device_test.go @@ -0,0 +1,31 @@ +package pool + +import ( + "encoding/json/v2" + "testing" +) + +func TestBackingDeviceIDPreservesPersistedNumbers(t *testing.T) { + for _, tc := range []struct { + name string + device uint64 + want string + }{ + {"linux-zero", backingDeviceID(uint64(0)), "0"}, + {"linux-high-bit", backingDeviceID(uint64(1) << 63), "9223372036854775808"}, + {"linux-max", backingDeviceID(^uint64(0)), "18446744073709551615"}, + {"darwin-positive", backingDeviceID(int32(16777234)), "16777234"}, + {"darwin-negative", backingDeviceID(int32(-1)), "18446744073709551615"}, + {"darwin-min", backingDeviceID(int32(-2147483648)), "18446744071562067968"}, + } { + t.Run(tc.name, func(t *testing.T) { + raw, err := json.Marshal(backingIdentity{Device: tc.device}) + if err != nil { + t.Fatal(err) + } + if want := `{"device":` + tc.want + `,"inode":0,"size":0}`; string(raw) != want { + t.Fatalf("persisted backing identity = %s, want %s", raw, want) + } + }) + } +} diff --git a/sandboxd/pool/volume_mutation.go b/sandboxd/pool/volume_mutation.go index 33dc50e2..ff309432 100644 --- a/sandboxd/pool/volume_mutation.go +++ b/sandboxd/pool/volume_mutation.go @@ -85,7 +85,7 @@ func (m *Manager) changeVolumes(ctx context.Context, id, tenant string, change v view = m.finishVolumeRun(sb, run) }() m.mu.Lock() - current, held := slices.Clone(sb.VolumesWant), heldVolumes(sb) + current, held := slices.Clone(sb.VolumesWant), slices.Clone(run.holds) m.mu.Unlock() target, err := change(current) if err != nil { @@ -98,6 +98,9 @@ func (m *Manager) changeVolumes(ctx context.Context, id, tenant string, change v if err = validateLiveTarget(held, target); err != nil { return view, err } + if err = m.validateManagedTarget(sb.ID, target); err != nil { + return view, err + } additions := volumeDifference(target, held) if len(additions) != 0 { if err = m.tenantRemoved(ctx, sb.Tenant); err != nil { @@ -108,12 +111,17 @@ func (m *Manager) changeVolumes(ctx context.Context, id, tenant string, change v if err != nil { return view, err } + for _, volume := range resolved { + if volume.managedID != "" { + return view, fmt.Errorf("%w: managed volumes support claim-time attachment only", ErrBadVolume) + } + } if err = m.checkVolumeRun(ctx, sb, run); err != nil { return view, err } sb.Touch() m.mu.Lock() - uncertain := volumeDifference(volumeDifference(sb.VolumesWant, sb.Volumes), target) + uncertain := volumeDifference(volumeDifference(run.holds, sb.Volumes), target) m.mu.Unlock() for _, v := range uncertain { failedName = v.Name @@ -177,6 +185,16 @@ func (m *Manager) beginVolumeRun(ctx context.Context, sb *types.Sandbox) (contex m.mu.Unlock() return ctx, nil, ErrUnknownSandbox } + if h, managed := m.managedHold(sb.ID); managed { + if m.ManagedVolumeStatus() != nil { + m.mu.Unlock() + return ctx, nil, managedStorageError() + } + if h.State != holdClaimed { + m.mu.Unlock() + return ctx, nil, ErrFailed + } + } if active := m.volumeRuns[sb]; active != nil { done := active.done m.mu.Unlock() @@ -189,6 +207,9 @@ func (m *Manager) beginVolumeRun(ctx context.Context, sb *types.Sandbox) (contex } runCtx, cancel := context.WithCancel(ctx) run := &volumeRun{cancel: cancel, done: make(chan struct{}), holds: heldVolumes(sb)} + if held, managed := m.managedVolumesHeld(sb.ID); managed { + run.holds = held + } m.volumeRuns[sb] = run m.mu.Unlock() return runCtx, run, nil @@ -197,7 +218,7 @@ func (m *Manager) beginVolumeRun(ctx context.Context, sb *types.Sandbox) (contex func (m *Manager) finishVolumeRun(sb *types.Sandbox, run *volumeRun) types.SandboxVolumesResponse { m.mu.Lock() - if m.claimed[sb.ID] == sb { + if _, managed := m.managedHold(sb.ID); m.claimed[sb.ID] == sb && !managed { m.releaseVolumes(volumeDifference(run.holds, heldVolumes(sb))) } delete(m.volumeRuns, sb) @@ -209,6 +230,9 @@ func (m *Manager) finishVolumeRun(sb *types.Sandbox, run *volumeRun) types.Sandb } func (m *Manager) restoreVolumeHolds(sb *types.Sandbox, holds []types.Volume) { + if _, managed := m.managedHold(sb.ID); managed { + return + } if m.volumeRuns[sb] == nil { m.releaseVolumes(volumeDifference(holds, heldVolumes(sb))) } @@ -227,6 +251,9 @@ func (m *Manager) checkVolumeRunLocked(ctx context.Context, sb *types.Sandbox, r if !m.ownsRunLocked(sb, run) { return ErrUnknownSandbox } + if h, managed := m.managedHold(sb.ID); managed && h.State != holdClaimed { + return ErrFailed + } switch { case sb.Failed != "": return ErrFailed @@ -244,6 +271,9 @@ func (m *Manager) cancelVolumeRun(sb *types.Sandbox) []types.Volume { run.cancel() return slices.Clone(run.holds) } + if held, managed := m.managedVolumesHeld(sb.ID); managed { + return held + } return heldVolumes(sb) } @@ -272,6 +302,14 @@ func (m *Manager) stageVolumeTarget(ctx context.Context, sb *types.Sandbox, run return claimSnapshot{}, false, err } run.holds = append(run.holds, additions...) + if err := m.managedLiveAdd(sb.ID, resolved); err != nil { + if m.ManagedVolumeStatus() == nil { + m.releaseVolumes(additions) + run.holds = volumeDifference(run.holds, additions) + } + m.mu.Unlock() + return claimSnapshot{}, false, err + } m.mu.Unlock() cleanErr := m.confirmVolumesClean(resolved) m.mu.Lock() @@ -280,6 +318,9 @@ func (m *Manager) stageVolumeTarget(ctx context.Context, sb *types.Sandbox, run return claimSnapshot{}, false, err } if cleanErr != nil { + if err := m.managedLiveRemove(sb.ID, additions); err != nil { + return claimSnapshot{}, false, err + } m.releaseVolumes(additions) run.holds = volumeDifference(run.holds, additions) return claimSnapshot{}, false, cleanErr @@ -287,7 +328,7 @@ func (m *Manager) stageVolumeTarget(ctx context.Context, sb *types.Sandbox, run replaced := func(v types.Volume) bool { return volumeIndex(sb.VolumesWant, v.Name) < 0 && volumeIndex(target, v.Name) >= 0 } - if len(additions) == 0 && slices.Equal(target, sb.VolumesWant) && !slices.ContainsFunc(sb.Volumes, replaced) && m.store.synced() { + if len(additions) == 0 && slices.Equal(target, sb.VolumesWant) && !slices.ContainsFunc(sb.Volumes, replaced) && m.store.synced() && len(volumeDifference(run.holds, heldVolumes(sb))) == 0 { return claimSnapshot{}, false, nil } sb.Volumes = slices.DeleteFunc(slices.Clone(sb.Volumes), replaced) @@ -334,6 +375,9 @@ func (m *Manager) commitVolumeRun(ctx context.Context, sb *types.Sandbox, run *v return err } dropped := volumeDifference(run.holds, heldVolumes(sb)) + if err := m.managedLiveRemove(sb.ID, dropped); err != nil { + return err + } m.releaseVolumes(dropped) run.holds = volumeDifference(run.holds, dropped) return nil diff --git a/sandboxd/pool/volume_registry.go b/sandboxd/pool/volume_registry.go new file mode 100644 index 00000000..5fc25a32 --- /dev/null +++ b/sandboxd/pool/volume_registry.go @@ -0,0 +1,431 @@ +package pool + +import ( + "encoding/json/v2" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "strings" + "sync" + "syscall" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type ( + volumeID string + volumePhase string + holdPhase string +) + +const ( + volumeCreating volumePhase = "creating" + volumeReady volumePhase = "ready" + volumeDeleting volumePhase = "deleting" + volumeRecovery volumePhase = "recovery_required" + holdAttaching holdPhase = "attaching" + holdClaimed holdPhase = "claimed" + holdRetiring holdPhase = "retiring" +) + +type backingIdentity struct { + Device uint64 `json:"device"` + Inode uint64 `json:"inode"` + Size int64 `json:"size"` +} + +type managedVolume struct { + ID volumeID `json:"id"` + Name string `json:"name"` + Tenant string `json:"tenant"` + Node string `json:"node"` + Size int64 `json:"size"` + Backing backingIdentity `json:"backing"` + State volumePhase `json:"state"` + CreatedAt time.Time `json:"created_at"` +} + +type volumeRef struct { + Volume types.Volume `json:"volume"` + ID volumeID `json:"id,omitempty"` + Path string `json:"path"` + Backing backingIdentity `json:"backing"` +} + +type processIdentity struct { + BootID string `json:"boot_id"` + PID int `json:"pid"` + Start uint64 `json:"start"` +} + +type volumeHold struct { + SandboxID string `json:"sandbox_id"` + Tenant string `json:"tenant"` + VMName string `json:"vm_name,omitempty"` + Process processIdentity `json:"process"` + State holdPhase `json:"state"` + Refs []volumeRef `json:"refs"` +} + +type volumeTombstone struct { + ID volumeID `json:"id"` + Name string `json:"name"` + Tenant string `json:"tenant"` +} + +type volumeRegistryData struct { + Orphans map[string]processIdentity `json:"orphans,omitempty"` + Version int `json:"version"` + Volumes map[string]managedVolume `json:"volumes"` + Holds map[string]volumeHold `json:"holds"` + Deleted map[volumeID]volumeTombstone `json:"deleted"` +} + +type volumeRegistry struct { + rootDir, disksDir os.FileInfo + mu sync.Mutex + dir, node string + lock *os.File + data volumeRegistryData + fault error + commitFile func(string, []byte) error + process func(int) (processIdentity, error) +} + +var managedIDPattern = regexp.MustCompile(`^vol_[0-9a-f]{32}$`) + +func openVolumeRegistry(dir, node string) (*volumeRegistry, error) { + if !filepath.IsAbs(dir) || filepath.Clean(dir) != dir || dir == "/" { + return nil, fmt.Errorf("invalid managed volume directory") + } + if err := privateVolumeDir(dir); err != nil { + return nil, err + } + if err := privateVolumeDir(filepath.Join(dir, "disks")); err != nil { + return nil, err + } + lock, err := openPrivateFile(filepath.Join(dir, "lock"), syscall.O_CREAT|syscall.O_RDWR) + if err != nil { + return nil, fmt.Errorf("open managed volume lock: %w", err) + } + if err = syscall.Flock(int(lock.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil { + return nil, errors.Join(fmt.Errorf("managed volume directory is already in use: %w", err), lock.Close()) + } + r := &volumeRegistry{dir: dir, node: node, lock: lock, commitFile: writeVolumeSnapshot, process: readProcessIdentity} + r.rootDir, err = os.Lstat(dir) + if err != nil { + return nil, errors.Join(err, lock.Close()) + } + r.disksDir, err = os.Lstat(filepath.Join(dir, "disks")) + if err != nil { + return nil, errors.Join(err, lock.Close()) + } + r.data = volumeRegistryData{Version: 1, Volumes: map[string]managedVolume{}, Holds: map[string]volumeHold{}, Deleted: map[volumeID]volumeTombstone{}} + f, err := openPrivateFile(filepath.Join(dir, "registry.json"), syscall.O_RDONLY) + if errors.Is(err, os.ErrNotExist) { + entries, readErr := os.ReadDir(filepath.Join(dir, "disks")) + if readErr != nil || len(entries) != 0 { + r.fault = errors.New("missing registry with existing backing files") + return r, nil + } + if err = r.commit(); err != nil { + return r, nil + } + return r, nil + } + if err != nil { + r.fault = err + return r, nil + } + defer func() { _ = f.Close() }() + if err := json.UnmarshalRead(f, &r.data, json.RejectUnknownMembers(true)); err != nil { + r.fault = err + return r, nil + } + if err := r.validate(); err != nil { + r.fault = err + } + return r, nil +} + +func privateVolumeDir(path string) error { + for p := path; p != "/"; p = filepath.Dir(p) { + st, err := os.Lstat(p) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return err + } + owner := st.Sys().(*syscall.Stat_t).Uid + if owner != 0 && int64(owner) != int64(os.Geteuid()) { + return fmt.Errorf("managed directory ancestors must belong to root or the daemon user") + } + if st.Mode().Perm()&0o022 != 0 && st.Mode()&os.ModeSticky == 0 { + return fmt.Errorf("managed directory ancestors must not be writable by others") + } + if !st.IsDir() || st.Mode()&os.ModeSymlink != 0 { + return fmt.Errorf("managed directory component is not a directory") + } + } + if err := os.MkdirAll(path, 0o700); err != nil { + return err + } + st, err := os.Lstat(path) + if err != nil { + return err + } + if !st.IsDir() || st.Mode().Perm()&0o077 != 0 { + return fmt.Errorf("managed directory must be private (0700)") + } + if int64(st.Sys().(*syscall.Stat_t).Uid) != int64(os.Geteuid()) { + return fmt.Errorf("managed directory must belong to the daemon user") + } + return nil +} + +func openPrivateFile(path string, flags int) (*os.File, error) { + fd, err := syscall.Open(path, flags|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0o600) + if err != nil { + return nil, err + } + f := os.NewFile(uintptr(fd), path) + st, err := f.Stat() + if err != nil || !st.Mode().IsRegular() || st.Mode().Perm()&0o077 != 0 || st.Sys().(*syscall.Stat_t).Nlink != 1 || int64(st.Sys().(*syscall.Stat_t).Uid) != int64(os.Geteuid()) { + _ = f.Close() + return nil, fmt.Errorf("managed file must be private, owned, regular, and unlinked elsewhere") + } + return f, nil +} + +func (r *volumeRegistry) path(id volumeID) string { + return filepath.Join(r.dir, "disks", string(id)+".raw") +} + +func inspectBacking(path string) (backingIdentity, error) { + f, err := openPrivateFile(path, syscall.O_RDONLY) + if err != nil { + return backingIdentity{}, err + } + defer func() { _ = f.Close() }() + st, err := f.Stat() + if err != nil { + return backingIdentity{}, err + } + raw := st.Sys().(*syscall.Stat_t) + return backingIdentity{Device: backingDeviceID(raw.Dev), Inode: raw.Ino, Size: st.Size()}, nil +} + +// Darwin uses signed device numbers; retain their existing uint64 representation. +func backingDeviceID[T ~uint64 | ~int32](device T) uint64 { + if device < 0 { + return ^uint64(^device) + } + return uint64(device) +} + +func (r *volumeRegistry) verify(v managedVolume) error { + if err := r.verifyDirectories(); err != nil { + return err + } + actual, err := inspectBacking(r.path(v.ID)) + if err != nil { + return err + } + if v.Backing.Inode == 0 || actual != v.Backing || actual.Size != v.Size { + return errors.New("managed backing identity changed") + } + return nil +} + +func (r *volumeRegistry) validate() error { + d := r.data + if d.Version != 1 || d.Volumes == nil || d.Holds == nil || d.Deleted == nil { + return errors.New("invalid volume registry format") + } + ids := map[volumeID]bool{} + for name, v := range d.Volumes { + if err := validateVolumeRecord(name, v, ids); err != nil { + return err + } + } + vms := map[string]bool{} + holders := map[string]volumeHolders{} + for id, h := range d.Holds { + if err := r.validateHoldRecord(id, h, vms, holders); err != nil { + return err + } + } + for name, p := range d.Orphans { + if !strings.HasPrefix(name, vmPrefix) || p.PID <= 0 || p.Start == 0 || p.BootID == "" { + return errors.New("invalid orphan process record") + } + } + for id, t := range d.Deleted { + if id != t.ID || !managedIDPattern.MatchString(string(id)) || !types.ValidVolumeName(t.Name) || !types.NameRe.MatchString(t.Tenant) || ids[id] { + return errors.New("invalid volume tombstone") + } + } + entries, err := os.ReadDir(filepath.Join(r.dir, "disks")) + if err != nil { + return err + } + for _, entry := range entries { + name := strings.TrimSuffix(entry.Name(), volumeDirtySuffix) + id := volumeID(strings.TrimSuffix(name, ".raw")) + if name != string(id)+".raw" || !ids[id] { + return errors.New("unregistered managed backing requires quarantine") + } + } + return nil +} + +func validateVolumeRecord(name string, v managedVolume, ids map[volumeID]bool) error { + if name != v.Name || !types.ValidVolumeName(name) || !managedIDPattern.MatchString(string(v.ID)) || !types.NameRe.MatchString(v.Tenant) || v.Size < 1<<30 || v.Size > 1024<<30 || v.Size%(1<<30) != 0 || v.CreatedAt.IsZero() || v.Node == "" || ids[v.ID] { + return errors.New("invalid managed volume record") + } + switch v.State { + case volumeCreating, volumeReady, volumeDeleting, volumeRecovery: + default: + return errors.New("invalid managed volume state") + } + if (v.State == volumeReady || v.Backing.Inode != 0) && (v.Backing.Inode == 0 || v.Backing.Size != v.Size) { + return errors.New("invalid managed backing record") + } + ids[v.ID] = true + return nil +} + +func (r *volumeRegistry) validateHoldRecord(id string, h volumeHold, vms map[string]bool, holders map[string]volumeHolders) error { + if id != h.SandboxID || !strings.HasPrefix(id, "sb_") || len(h.Refs) == 0 { + return errors.New("invalid volume holder") + } + switch h.State { + case holdAttaching, holdClaimed, holdRetiring: + default: + return errors.New("invalid volume holder state") + } + if h.VMName != "" { + if !strings.HasPrefix(h.VMName, vmPrefix) || vms[h.VMName] || h.Process.PID <= 0 || h.Process.Start == 0 || h.Process.BootID == "" { + return errors.New("invalid volume process identity") + } + vms[h.VMName] = true + } else if h.State == holdClaimed { + return errors.New("claimed volume lacks VM") + } + names := map[string]bool{} + managed := false + for _, ref := range h.Refs { + isManaged, err := r.validateHoldRef(h, ref, names, holders) + if err != nil { + return err + } + managed = managed || isManaged + } + if !managed { + return errors.New("managed holder has no managed resource") + } + return nil +} + +func (r *volumeRegistry) validateHoldRef(h volumeHold, ref volumeRef, names map[string]bool, holders map[string]volumeHolders) (bool, error) { + if !types.ValidVolumeName(ref.Volume.Name) || names[ref.Volume.Name] || !filepath.IsAbs(ref.Path) || (ref.Volume.Mode != "" && ref.Volume.Mode != types.VolumeModeRW) { + return false, errors.New("invalid volume reference") + } + names[ref.Volume.Name] = true + counts := holders[ref.Volume.Name] + if counts.writers > 0 || ref.Volume.RW() && counts.readers > 0 { + return false, errors.New("conflicting volume holder records") + } + if ref.Volume.RW() { + counts.writers++ + } else { + counts.readers++ + } + holders[ref.Volume.Name] = counts + if ref.ID == "" { + return false, nil + } + v, ok := r.data.Volumes[ref.Volume.Name] + if !ok || (v.State != volumeReady && v.State != volumeRecovery) || ref.ID != v.ID || ref.Path != r.path(v.ID) || ref.Backing != v.Backing || (h.Tenant != "" && h.Tenant != v.Tenant) { + return false, errors.New("inconsistent managed volume reference") + } + return true, nil +} + +// Any failed snapshot poisons mutations until restart. The attempted state remains +// owned in memory because rename may already have published it. +func (r *volumeRegistry) commit() error { + if r.fault != nil { + return managedStorageError() + } + if err := r.verifyDirectories(); err != nil { + r.fault = err + return managedStorageError() + } + raw, err := json.Marshal(r.data) + if err == nil { + err = r.commitFile(filepath.Join(r.dir, "registry.json"), raw) + } + if err != nil { + r.fault = err + return managedStorageError() + } + return nil +} + +func writeVolumeSnapshot(path string, raw []byte) error { + tmp, err := os.CreateTemp(filepath.Dir(path), ".registry-") + if err != nil { + return err + } + defer func() { _ = os.Remove(tmp.Name()) }() + if _, err := tmp.Write(raw); err != nil { + return errors.Join(err, tmp.Close()) + } + if err := errors.Join(tmp.Sync(), tmp.Close()); err != nil { + return err + } + if err := os.Rename(tmp.Name(), path); err != nil { + return err + } + return syncDir(filepath.Dir(path)) +} + +func (r *volumeRegistry) close() error { + r.mu.Lock() + defer r.mu.Unlock() + if r.lock == nil { + return nil + } + err := r.lock.Close() + r.lock = nil + return err +} + +// Close releases the managed directory lock after requests and workers have stopped. +func (m *Manager) Close() error { + if m.managed == nil { + return nil + } + return m.managed.close() +} + +func (r *volumeRegistry) verifyDirectories() error { + for _, pair := range []struct { + path string + original os.FileInfo + }{{r.dir, r.rootDir}, {filepath.Join(r.dir, "disks"), r.disksDir}} { + st, err := os.Lstat(pair.path) + if err != nil { + return err + } + if !st.IsDir() || st.Mode().Perm()&0o077 != 0 || !os.SameFile(st, pair.original) { + return errors.New("managed directory identity or permissions changed") + } + } + return nil +} diff --git a/sandboxd/server/api.go b/sandboxd/server/api.go index 6e26e0c9..f612d71b 100644 --- a/sandboxd/server/api.go +++ b/sandboxd/server/api.go @@ -6,6 +6,10 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) +type VolumeResponse struct { + Volume types.ManagedVolumeInfo `json:"volume"` +} + // TTLField is the shared requested-lease field; zero means the server default. type TTLField struct { TTLSeconds int `json:"ttl_seconds,omitzero"` diff --git a/sandboxd/server/managed_volumes.go b/sandboxd/server/managed_volumes.go new file mode 100644 index 00000000..f45b0d9e --- /dev/null +++ b/sandboxd/server/managed_volumes.go @@ -0,0 +1,129 @@ +package server + +import ( + "context" + "errors" + "io" + "net/http" + "strings" + + "github.com/cocoonstack/sandbox/sandboxd/pool" + "github.com/cocoonstack/sandbox/sandboxd/types" + "github.com/cocoonstack/sandbox/sandboxd/utils" +) + +type managedVolumeManager interface { + CreateVolume(context.Context, string, types.VolumeCreate, string) (types.ManagedVolumeInfo, bool, error) + GetVolume(string, string) (types.ManagedVolumeInfo, error) + DeleteVolume(context.Context, string, string, string) error + ManagedVolumeStatus() error + ManagedVolumesEnabled() bool +} + +func volumeHTTPError(w http.ResponseWriter, status int, code, message string) { + writeJSON(w, status, &pool.VolumeError{Status: status, Code: code, Message: message}) +} + +func (s *Server) requireVolumeListToken(next http.HandlerFunc) http.HandlerFunc { + if mgr, ok := s.mgr.(managedVolumeManager); ok && mgr.ManagedVolumesEnabled() { + return s.requireVolumeManager(next) + } + return s.requireToken(next) +} + +func (s *Server) requireVolumeManager(next http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + token, ok := bearerToken(r) + if !ok { + volumeHTTPError(w, 401, "volume_unauthorized", "explicit root or tenant bearer token required") + return + } + if s.isRootToken(token) { + next(w, r.WithContext(withTenant(r.Context(), nil))) + return + } + tenant, err := s.mgr.TenantByToken(r.Context(), token) + if errors.Is(err, pool.ErrTenantStoreDown) { + w.Header().Set("Retry-After", retryAfterSeconds) + volumeHTTPError(w, 503, "tenant_store_unavailable", "tenant store unavailable; retry") + return + } + if err != nil || tenant == nil { + volumeHTTPError(w, 401, "volume_unauthorized", "explicit root or tenant bearer token required") + return + } + next(w, r.WithContext(withTenant(r.Context(), tenant))) + } +} + +func (s *Server) volumeManager(w http.ResponseWriter) (managedVolumeManager, bool) { + mgr, ok := s.mgr.(managedVolumeManager) + if !ok || !mgr.ManagedVolumesEnabled() { + volumeHTTPError(w, 503, "volume_storage_unavailable", "managed volume storage is not enabled") + return nil, false + } + return mgr, true +} + +func (s *Server) handleCreateVolume(w http.ResponseWriter, r *http.Request) { + mgr, ok := s.volumeManager(w) + if !ok { + return + } + raw, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxBodyBytes)) + var req types.VolumeCreate + if err != nil || utils.DecodeStrictJSON(raw, &req) != nil { + volumeHTTPError(w, 400, "invalid_volume", "invalid request body") + return + } + v, created, err := mgr.CreateVolume(r.Context(), r.PathValue("name"), req, tenantFrom(r.Context())) + if err != nil { + if !writePoolErr(w, err) { + volumeHTTPError(w, 503, "volume_storage_unavailable", "volume storage unavailable; retry") + } + return + } + status := http.StatusOK + if created { + status = http.StatusCreated + } + w.Header().Set("ETag", `"`+v.ID+`"`) + writeJSON(w, status, VolumeResponse{Volume: v}) +} + +func (s *Server) handleGetVolume(w http.ResponseWriter, r *http.Request) { + mgr, ok := s.volumeManager(w) + if !ok { + return + } + v, err := mgr.GetVolume(r.PathValue("name"), tenantFrom(r.Context())) + if err != nil { + writePoolErr(w, err) + return + } + w.Header().Set("ETag", `"`+v.ID+`"`) + writeJSON(w, http.StatusOK, VolumeResponse{Volume: v}) +} + +func (s *Server) handleDeleteVolume(w http.ResponseWriter, r *http.Request) { + mgr, ok := s.volumeManager(w) + if !ok { + return + } + expected := r.Header.Get("If-Match") + if expected == "" { + volumeHTTPError(w, 428, "volume_precondition_required", "If-Match must identify the volume") + return + } + if len(expected) < 3 || expected[0] != '"' || expected[len(expected)-1] != '"' || strings.ContainsAny(expected[1:len(expected)-1], "\",\\") { + volumeHTTPError(w, 412, "volume_precondition_failed", "If-Match must contain one quoted volume identity") + return + } + if err := mgr.DeleteVolume(r.Context(), r.PathValue("name"), expected[1:len(expected)-1], tenantFrom(r.Context())); err != nil { + if !writePoolErr(w, err) { + volumeHTTPError(w, 503, "volume_storage_unavailable", "volume storage unavailable; retry") + } + return + } + w.WriteHeader(http.StatusNoContent) +} diff --git a/sandboxd/server/server.go b/sandboxd/server/server.go index 78b44fd6..9a4f0669 100644 --- a/sandboxd/server/server.go +++ b/sandboxd/server/server.go @@ -268,7 +268,10 @@ func (s *Server) SetConfigReload(f ConfigReloader) { func (s *Server) Handler() http.Handler { mux := http.NewServeMux() mux.HandleFunc("POST /v1/claim", s.requireToken(s.handleClaim)) - mux.HandleFunc("GET /v1/volumes", s.requireToken(s.handleVolumes)) + mux.HandleFunc("GET /v1/volumes", s.requireVolumeListToken(s.handleVolumes)) + mux.HandleFunc("PUT /v1/volumes/{name}", s.requireVolumeManager(s.handleCreateVolume)) + mux.HandleFunc("GET /v1/volumes/{name}", s.requireVolumeManager(s.handleGetVolume)) + mux.HandleFunc("DELETE /v1/volumes/{name}", s.requireVolumeManager(s.handleDeleteVolume)) mux.HandleFunc("POST /v1/sandboxes/{id}/release", s.handleSandboxVerb("release", s.mgr.Release)) mux.HandleFunc("POST /v1/sandboxes/{id}/hibernate", s.handleSandboxVerb("hibernate", s.mgr.Hibernate)) mux.HandleFunc("POST /v1/sandboxes/{id}/wake", s.handleSandboxVerb("wake", s.mgr.Wake)) @@ -383,6 +386,12 @@ func (s *Server) handleVolumeClaim(w http.ResponseWriter, r *http.Request, req C } func (s *Server) handleVolumes(w http.ResponseWriter, r *http.Request) { + if mgr, ok := s.mgr.(managedVolumeManager); ok { + if err := mgr.ManagedVolumeStatus(); err != nil { + writePoolErr(w, err) + return + } + } var holders map[string]int if s.placer != nil { holders = s.placer.VolumeHolders() diff --git a/sandboxd/server/server_http.go b/sandboxd/server/server_http.go index 48bf2a94..59a90d98 100644 --- a/sandboxd/server/server_http.go +++ b/sandboxd/server/server_http.go @@ -15,6 +15,7 @@ import ( "github.com/projecteru2/core/log" + "github.com/cocoonstack/sandbox/sandboxd/pool" "github.com/cocoonstack/sandbox/sandboxd/tenants" "github.com/cocoonstack/sandbox/sandboxd/types" "github.com/cocoonstack/sandbox/sandboxd/utils" @@ -170,6 +171,13 @@ func metadataFilter(pairs []string) (types.Metadata, error) { } func writePoolErr(w http.ResponseWriter, err error) bool { + if volumeErr, ok := errors.AsType[*pool.VolumeError](err); ok { + if volumeErr.Status == http.StatusServiceUnavailable { + w.Header().Set("Retry-After", retryAfterSeconds) + } + writeJSON(w, volumeErr.Status, volumeErr) + return true + } code, msg, ok := mapPoolErr(w, err) if ok { writeErr(w, code, msg) @@ -186,6 +194,9 @@ func mapPoolErr(w http.ResponseWriter, err error) (code int, msg string, ok bool } func lookupPoolErr(err error) (code int, msg string, ok bool) { + if volumeErr, ok := errors.AsType[*pool.VolumeError](err); ok { + return volumeErr.Status, volumeErr.Message, true + } for _, m := range poolErrHTTP { if errors.Is(err, m.err) { return m.code, cmp.Or(m.msg, err.Error()), true diff --git a/sandboxd/types/api.go b/sandboxd/types/api.go index ac0cd6b4..0b0e8b11 100644 --- a/sandboxd/types/api.go +++ b/sandboxd/types/api.go @@ -13,15 +13,37 @@ const ( // VolumeInfo is the caller-visible, host-path-free catalog projection. type VolumeInfo struct { - Name string `json:"name"` - DefaultMount string `json:"default_mount"` - SizeBytes int64 `json:"size_bytes"` - Available bool `json:"available"` - Nodes int `json:"nodes"` + Kind string `json:"kind,omitempty"` + ID string `json:"id,omitempty"` + Tenant string `json:"tenant,omitempty"` + Node string `json:"node,omitempty"` + State string `json:"state,omitempty"` + CreatedAt time.Time `json:"created_at,omitzero"` + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` // Writable reports whether the operator allows rw claims of this name. Writable bool `json:"writable,omitzero"` } +type ManagedVolumeInfo struct { + VolumeInfo + Holders []VolumeHolder `json:"holders,omitempty"` +} + +type VolumeHolder struct { + SandboxID string `json:"sandbox_id"` + Mode string `json:"mode"` + State string `json:"state"` +} + +type VolumeCreate struct { + SizeGiB int64 `json:"size_gib"` + Tenant string `json:"tenant,omitempty"` +} + // VolumePatch is the wire body of PATCH /v1/sandboxes/{id}/volumes. type VolumePatch struct { Add []Volume `json:"add,omitzero"` diff --git a/sdk/go/client.go b/sdk/go/client.go index ccb932fd..082c7727 100644 --- a/sdk/go/client.go +++ b/sdk/go/client.go @@ -71,7 +71,10 @@ type renewResponse struct { } type errorResponse struct { - Error string `json:"error"` + Error string `json:"error"` + Code string `json:"code"` + Volume string `json:"volume"` + Holders []VolumeHolder `json:"holders"` } // APIError is a non-2xx control-plane reply. @@ -443,7 +446,11 @@ func encodeBody(verb string, v any) ([]byte, error) { func apiError(verb string, resp *http.Response) error { var er errorResponse - _ = json.UnmarshalRead(io.LimitReader(resp.Body, 4096), &er) + _ = json.UnmarshalRead(resp.Body, &er) _, _ = io.Copy(io.Discard, resp.Body) - return &APIError{Verb: verb, Status: resp.StatusCode, Message: er.Error} + base := &APIError{Verb: verb, Status: resp.StatusCode, Message: er.Error} + if er.Code != "" || len(er.Holders) != 0 { + return &ManagedVolumeError{APIError: base, Code: er.Code, Volume: er.Volume, Holders: er.Holders} + } + return base } diff --git a/sdk/go/managed_volume.go b/sdk/go/managed_volume.go new file mode 100644 index 00000000..376667f4 --- /dev/null +++ b/sdk/go/managed_volume.go @@ -0,0 +1,129 @@ +package sandbox + +import ( + "bytes" + "context" + "encoding/json/v2" + "fmt" + "io" + "net/http" + "net/url" + "regexp" + "strings" +) + +type ManagedVolumeInfo struct { + VolumeInfo + Holders []VolumeHolder `json:"holders,omitempty"` +} + +type ManagedVolumeError struct { + *APIError + Code string + Volume string + Holders []VolumeHolder +} + +func (e *ManagedVolumeError) Unwrap() error { return e.APIError } + +var managedVolumeName = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,19}$`) + +// VolumeCreate requests a raw managed disk owned by Tenant. A tenant token may leave Tenant empty. +type VolumeCreate struct { + SizeGiB int64 `json:"size_gib"` + Tenant string `json:"tenant,omitempty"` +} + +type volumeResponse struct { + Volume *ManagedVolumeInfo `json:"volume"` +} + +// CreateVolume creates a managed disk or returns the same disk for an identical retry. +func (c *Client) CreateVolume(ctx context.Context, name string, create VolumeCreate) (*ManagedVolumeInfo, error) { + if err := validManagedVolumeName(name); err != nil { + return nil, err + } + if create.SizeGiB < 1 || create.SizeGiB > 1024 { + return nil, fmt.Errorf("volume size_gib must be between 1 and 1024") + } + body, err := encodeBody("create volume", create) + if err != nil { + return nil, err + } + resp, err := c.roundTrip(ctx, http.MethodPut, c.addr, volumePath(name), bytes.NewReader(body), c.apiToken) + if err != nil { + return nil, err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK { + return nil, apiError("create volume", resp) + } + return decodeVolumeResponse("create volume", resp.Body) +} + +// GetVolume inspects one managed disk visible to this token. +func (c *Client) GetVolume(ctx context.Context, name string) (*ManagedVolumeInfo, error) { + if err := validManagedVolumeName(name); err != nil { + return nil, err + } + resp, err := c.roundTrip(ctx, http.MethodGet, c.addr, volumePath(name), nil, c.apiToken) + if err != nil { + return nil, err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return nil, apiError("get volume", resp) + } + return decodeVolumeResponse("get volume", resp.Body) +} + +// DeleteVolume deletes exactly id, including when an earlier response was lost. +func (c *Client) DeleteVolume(ctx context.Context, name, id string) error { + if err := validManagedVolumeName(name); err != nil { + return err + } + if id == "" || strings.ContainsAny(id, "\"\r\n") { + return fmt.Errorf("volume id must be a nonempty unquoted resource id") + } + req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.requestURL(c.addr, volumePath(name)), nil) + if err != nil { + return err + } + req.Header.Set("If-Match", `"`+id+`"`) + if c.apiToken != "" { + req.Header.Set("Authorization", "Bearer "+c.apiToken) + } + resp, err := c.hc.Do(req) + if err != nil { + return err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusNoContent { + return apiError("delete volume", resp) + } + return nil +} + +func validManagedVolumeName(name string) error { + if !managedVolumeName.MatchString(name) || strings.HasPrefix(name, "cocoon-") { + return fmt.Errorf("volume name %q must match %s and not start with cocoon-", name, managedVolumeName) + } + return nil +} + +func volumePath(name string) string { + return "/v1/volumes/" + url.PathEscape(name) +} + +func decodeVolumeResponse(verb string, body io.Reader) (*ManagedVolumeInfo, error) { + var reply volumeResponse + if err := json.UnmarshalRead(body, &reply); err != nil { + return nil, fmt.Errorf("decode %s response: %w", verb, err) + } + if reply.Volume == nil || reply.Volume.Kind != "managed" || reply.Volume.ID == "" || + reply.Volume.Name == "" || reply.Volume.Tenant == "" || reply.Volume.Node == "" || + reply.Volume.State == "" || reply.Volume.CreatedAt.IsZero() || reply.Volume.SizeBytes < 1 { + return nil, fmt.Errorf("decode %s response: missing managed volume fields", verb) + } + return reply.Volume, nil +} diff --git a/sdk/go/managed_volume_test.go b/sdk/go/managed_volume_test.go new file mode 100644 index 00000000..cbb428d7 --- /dev/null +++ b/sdk/go/managed_volume_test.go @@ -0,0 +1,144 @@ +package sandbox + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" + "time" +) + +func TestManagedVolumeRequests(t *testing.T) { + want := ManagedVolumeInfo{Name: "data", Kind: "managed", ID: "vol_123", Tenant: "t1", Node: "node-1", State: "ready", CreatedAt: time.Date(2026, 10, 19, 0, 0, 0, 0, time.UTC), SizeBytes: 20 << 30, Available: true, Writable: true, Holders: []VolumeHolder{{SandboxID: "sb_1", Mode: "rw", State: "attached"}}} + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.URL.Path != "/v1/volumes/data" || r.Header.Get("Authorization") != "Bearer token" { + t.Errorf("request = %s %s, authorization %q", r.Method, r.URL.Path, r.Header.Get("Authorization")) + } + switch r.Method { + case http.MethodPut: + var body map[string]any + if err := json.UnmarshalRead(r.Body, &body); err != nil { + t.Errorf("decode create: %v", err) + } + if body["size_gib"] != float64(20) || body["tenant"] != "t1" { + t.Errorf("create body = %v", body) + } + if requests == 1 { + w.WriteHeader(http.StatusCreated) + } + _ = json.MarshalWrite(w, volumeResponse{Volume: &want}) + case http.MethodGet: + w.Header().Set("ETag", `"vol_123"`) + _ = json.MarshalWrite(w, volumeResponse{Volume: &want}) + case http.MethodDelete: + if got := r.Header.Get("If-Match"); got != `"vol_123"` { + t.Errorf("If-Match = %q", got) + } + w.WriteHeader(http.StatusNoContent) + default: + t.Errorf("unexpected method %s", r.Method) + } + })) + t.Cleanup(ts.Close) + c := testClient(t, ts, WithAPIToken("token")) + for range 2 { + got, err := c.CreateVolume(t.Context(), "data", VolumeCreate{SizeGiB: 20, Tenant: "t1"}) + if err != nil || !reflect.DeepEqual(got, &want) { + t.Fatalf("create = %+v, %v", got, err) + } + } + got, err := c.GetVolume(t.Context(), "data") + if err != nil || !reflect.DeepEqual(got, &want) { + t.Fatalf("get = %+v, %v", got, err) + } + if err := c.DeleteVolume(t.Context(), "data", want.ID); err != nil { + t.Fatal(err) + } + if err := c.DeleteVolume(t.Context(), "data", want.ID); err != nil { + t.Fatal(err) + } + if requests != 5 { + t.Fatalf("requests = %d, want 5", requests) + } +} + +func TestManagedVolumeFailures(t *testing.T) { + statuses := []int{400, 401, 403, 404, 409, 412, 428, 503} + for _, status := range statuses { + t.Run(http.StatusText(status), func(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(status) + _ = json.MarshalWrite(w, map[string]any{"error": "volume unavailable", "code": "volume_busy", "volume": "data", "holders": []map[string]string{{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring", "host_path": "/secret"}}}) + })) + t.Cleanup(ts.Close) + _, err := testClient(t, ts).GetVolume(t.Context(), "data") + var apiErr *ManagedVolumeError + if !errors.As(err, &apiErr) || apiErr.Status != status || apiErr.Code != "volume_busy" || apiErr.Volume != "data" || !reflect.DeepEqual(apiErr.Holders, []VolumeHolder{{SandboxID: "sb_1", Mode: "rw", State: "retiring"}}) { + t.Fatalf("error = %#v", err) + } + if strings.Contains(apiErr.Error(), "/secret") { + t.Fatal("error exposed a host path") + } + }) + } +} + +func TestManagedVolumeRejectsInvalidRepliesAndInputs(t *testing.T) { + requests := 0 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + requests++ + _, _ = w.Write([]byte(`{}`)) + })) + t.Cleanup(ts.Close) + c := testClient(t, ts) + if _, err := c.CreateVolume(t.Context(), "bad/name", VolumeCreate{SizeGiB: 20}); err == nil { + t.Fatal("accepted invalid name") + } + if _, err := c.CreateVolume(t.Context(), "data", VolumeCreate{SizeGiB: 0}); err == nil { + t.Fatal("accepted zero size") + } + if err := c.DeleteVolume(t.Context(), "data", `"weak"`); err == nil { + t.Fatal("accepted quoted id") + } + if requests != 0 { + t.Fatalf("invalid input sent %d requests", requests) + } + if _, err := c.GetVolume(t.Context(), "data"); err == nil { + t.Fatal("accepted missing response volume") + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + if _, err := c.GetVolume(ctx, "data"); !errors.Is(err, context.Canceled) { + t.Fatalf("cancellation = %v", err) + } + if requests != 1 { + t.Fatalf("canceled request reached server, count %d", requests) + } +} + +func TestManagedVolumeConflictPreservesAllHolders(t *testing.T) { + holders := make([]VolumeHolder, 100) + for i := range holders { + holders[i] = VolumeHolder{SandboxID: fmt.Sprintf("sb_%032d", i), Mode: "ro", State: "claimed"} + } + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusConflict) + _ = json.MarshalWrite(w, map[string]any{"error": "volume held", "code": "volume_busy", "volume": "data", "holders": holders}) + })) + t.Cleanup(ts.Close) + _, err := testClient(t, ts).GetVolume(t.Context(), "data") + var conflict *ManagedVolumeError + if !errors.As(err, &conflict) || conflict.Status != http.StatusConflict || conflict.Code != "volume_busy" { + t.Fatalf("expected typed volume conflict, got %#v", err) + } + if !reflect.DeepEqual(conflict.Holders, holders) { + t.Fatalf("conflict returned %d holders, want all %d with intact fields", len(conflict.Holders), len(holders)) + } +} diff --git a/sdk/go/options.go b/sdk/go/options.go index 20b75401..0d04873a 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -39,12 +39,24 @@ type Volume struct { // VolumeInfo describes one visible fleet catalog entry. type VolumeInfo struct { - Name string `json:"name"` - DefaultMount string `json:"default_mount"` - SizeBytes int64 `json:"size_bytes"` - Available bool `json:"available"` - Nodes int `json:"nodes"` - Writable bool `json:"writable,omitzero"` + Kind string `json:"kind,omitempty"` + ID string `json:"id,omitempty"` + Tenant string `json:"tenant,omitempty"` + Node string `json:"node,omitempty"` + State string `json:"state,omitempty"` + CreatedAt time.Time `json:"created_at,omitzero"` + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` + Writable bool `json:"writable,omitzero"` +} + +type VolumeHolder struct { + SandboxID string `json:"sandbox_id"` + Mode string `json:"mode"` + State string `json:"state"` } // EnvInject makes the node's egress proxy send a host-only entry's value to Hosts, exact or "*." suffix, as Header or as the Query parameter, where the guest sent a set Placeholder. diff --git a/sdk/python/cocoonsandbox/client.py b/sdk/python/cocoonsandbox/client.py index aa3c8fd1..80904c67 100644 --- a/sdk/python/cocoonsandbox/client.py +++ b/sdk/python/cocoonsandbox/client.py @@ -5,11 +5,13 @@ import http.client import json import queue +import re import ssl import threading import time import urllib.parse from collections.abc import Callable, Iterable, Mapping, Sequence +from datetime import datetime from typing import Any, TypeVar, cast from .checkpoint import Checkpoint @@ -21,6 +23,8 @@ T = TypeVar("T") +_VOLUME_NAME = re.compile(r"^[a-z][a-z0-9_-]{0,19}$") + _PEERS_TIMEOUT = 5.0 _IDLE_CONNS = 8 @@ -133,6 +137,37 @@ def volumes(self) -> list[dict[str, Any]]: reply = self._request(self.addr, "GET", "/v1/volumes", None, "list volumes") return [dict(volume) for volume in reply.get("volumes") or []] + def create_volume(self, name: str, size_gib: int, tenant: str = "") -> dict[str, Any]: + """Creates a raw managed disk or returns it for an identical retry.""" + path = _volume_path(name) + if isinstance(size_gib, bool) or not isinstance(size_gib, int) or not 1 <= size_gib <= 1024: + raise ValueError("volume size_gib must be an integer between 1 and 1024") + body: dict[str, Any] = {"size_gib": size_gib} + if tenant: + body["tenant"] = tenant + reply = self._request(self.addr, "PUT", path, body, "create volume", expected_status=(200, 201)) + return _managed_volume(reply, "create volume") + + def get_volume(self, name: str) -> dict[str, Any]: + """Inspects one managed disk visible to this token.""" + reply = self._request(self.addr, "GET", _volume_path(name), None, "get volume", expected_status=(200,)) + return _managed_volume(reply, "get volume") + + def delete_volume(self, name: str, resource_id: str) -> None: + """Deletes exactly resource_id, including when an earlier response was lost.""" + path = _volume_path(name) + if not resource_id or '"' in resource_id or "\r" in resource_id or "\n" in resource_id: + raise ValueError("volume resource_id must be a nonempty unquoted id") + self._request( + self.addr, + "DELETE", + path, + None, + "delete volume", + headers={"If-Match": f'"{resource_id}"'}, + expected_status=(204,), + ) + def info(self) -> dict[str, Any]: """The node's pool/claim counters, as served by GET /v1/info.""" return self._request(self.addr, "GET", "/v1/info", None, "info") @@ -198,11 +233,13 @@ def _request( bearer: str = "", timeout: float = 0.0, deadline: float | None = None, + headers: dict[str, str] | None = None, + expected_status: tuple[int, ...] | None = None, ) -> dict[str, Any]: timeout = remaining_timeout(timeout or self.timeout, deadline, verb) data = json.dumps(body).encode() if body is not None else None url = urllib.parse.urlsplit(addr + path if "://" in addr else f"{self._scheme}://{addr}{path}") - headers = {"Content-Type": "application/json"} if data is not None else {} + headers = {**(headers or {}), **({"Content-Type": "application/json"} if data is not None else {})} token = bearer or self.api_token if token: headers["Authorization"] = f"Bearer {token}" @@ -218,6 +255,8 @@ def _request( except (ValueError, TypeError): reply = None raise APIError(verb, status, _error_message(raw), reply=reply if isinstance(reply, dict) else None) + if expected_status is not None and status not in expected_status: + raise APIError(verb, status, "unexpected response status") if not raw: return {} try: @@ -347,3 +386,39 @@ def run(addr: str) -> None: return cast(T, value) last_error = error raise cast(Exception, last_error) + + +def _volume_path(name: str) -> str: + if not _VOLUME_NAME.fullmatch(name) or name.startswith("cocoon-"): + raise ValueError(f"invalid volume name {name!r}") + return "/v1/volumes/" + urllib.parse.quote(name, safe="") + + +def _managed_volume(reply: dict[str, Any], verb: str) -> dict[str, Any]: + if not isinstance(reply, dict): + raise APIError(verb, 0, "reply without managed volume fields") + volume = require_field(reply, "volume", verb) + required = ("id", "name", "tenant", "node", "state", "created_at") + if ( + not isinstance(volume, dict) + or volume.get("kind") != "managed" + or any(not isinstance(volume.get(key), str) or not volume[key] for key in required) + ): + raise APIError(verb, 0, "reply without managed volume fields") + size = volume.get("size_bytes") + if isinstance(size, bool) or not isinstance(size, int) or size < 1: + raise APIError(verb, 0, "reply with invalid managed volume capacity") + timestamp = re.fullmatch( + r"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\.\d{1,9})?(Z|[+-]\d{2}:\d{2})", + volume["created_at"], + ) + if timestamp is None: + raise APIError(verb, 0, "reply with invalid managed volume timestamp") + offset = timestamp[2] + if offset != "Z" and (int(offset[1:3]) > 23 or int(offset[4:6]) > 59): + raise APIError(verb, 0, "reply with invalid managed volume timestamp") + try: + datetime.fromisoformat(timestamp[1] + timestamp[2].replace("Z", "+00:00")) + except ValueError as exc: + raise APIError(verb, 0, "reply with invalid managed volume timestamp") from exc + return dict(volume) diff --git a/sdk/python/cocoonsandbox/errors.py b/sdk/python/cocoonsandbox/errors.py index 4e1ed4dc..314b2fe5 100644 --- a/sdk/python/cocoonsandbox/errors.py +++ b/sdk/python/cocoonsandbox/errors.py @@ -25,6 +25,14 @@ def __init__(self, verb: str, status: int, message: str, *, reply: dict[str, Any self.status = status self.message = message self.reply = reply + self.code = (reply or {}).get("code", "") + self.volume = (reply or {}).get("volume", "") + holders = (reply or {}).get("holders") + self.holders = [ + {key: entry[key] for key in ("sandbox_id", "mode", "state") if isinstance(entry.get(key), str)} + for entry in (holders if isinstance(holders, list) else []) + if isinstance(entry, dict) + ] class VolumeError(APIError): diff --git a/sdk/python/tests/test_managed_volumes.py b/sdk/python/tests/test_managed_volumes.py new file mode 100644 index 00000000..198d367e --- /dev/null +++ b/sdk/python/tests/test_managed_volumes.py @@ -0,0 +1,185 @@ +import threading +from http.server import HTTPServer + +import pytest +from conftest import FakeNode + +from cocoonsandbox import APIError, Client + + +class VolumeNode(FakeNode): + def do_PUT(self): + self._dispatch("PUT") + + +@pytest.fixture +def node(): + server = HTTPServer(("127.0.0.1", 0), VolumeNode) + FakeNode.routes = {} + threading.Thread(target=server.serve_forever, daemon=True).start() + yield f"127.0.0.1:{server.server_port}" + server.shutdown() + + +def test_create_inspect_and_delete_use_immutable_identity(node): + volume = { + "name": "data", + "kind": "managed", + "id": "vol_123", + "tenant": "t1", + "node": "node-1", + "state": "ready", + "created_at": "2026-10-19T00:00:00Z", + "size_bytes": 20 << 30, + "available": True, + "holders": [{"sandbox_id": "sb_1", "mode": "rw", "state": "attached"}], + } + seen = [] + + def create(body, path): + seen.append(("PUT", body, dict(FakeNode.last_headers))) + return (201 if len(seen) == 1 else 200), {"volume": volume} + + def inspect(body, path): + seen.append(("GET", body, dict(FakeNode.last_headers))) + return 200, {"volume": volume} + + def delete(body, path): + seen.append(("DELETE", body, dict(FakeNode.last_headers))) + return 204, {} + + path = "/v1/volumes/data" + FakeNode.routes[("PUT", path)] = create + FakeNode.routes[("GET", path)] = inspect + FakeNode.routes[("DELETE", path)] = delete + client = Client(node, api_token="token") + assert client.create_volume("data", 20, tenant="t1") == volume + assert client.create_volume("data", 20, tenant="t1") == volume + assert client.get_volume("data") == volume + client.delete_volume("data", volume["id"]) + client.delete_volume("data", volume["id"]) + assert [request[0] for request in seen] == ["PUT", "PUT", "GET", "DELETE", "DELETE"] + assert all(request[2]["Authorization"] == "Bearer token" for request in seen) + assert seen[0][1] == {"size_gib": 20, "tenant": "t1"} + assert seen[3][2]["If-Match"] == seen[4][2]["If-Match"] == '"vol_123"' + + +@pytest.mark.parametrize("status", [400, 401, 403, 404, 409, 412, 428, 503]) +def test_management_error_keeps_safe_details(node, status): + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: ( + status, + { + "error": "volume unavailable", + "code": "volume_busy", + "volume": "data", + "holders": [{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring", "host_path": "/secret"}], + }, + ) + with pytest.raises(APIError) as raised: + Client(node).get_volume("data") + error = raised.value + assert error.status == status and error.message == "volume unavailable" + assert error.code == "volume_busy" and error.volume == "data" + assert error.holders == [{"sandbox_id": "sb_1", "mode": "rw", "state": "retiring"}] + assert "/secret" not in str(error) and "/secret" not in repr(error.holders) + + +def test_invalid_inputs_and_incomplete_reply(node): + calls = [] + + def incomplete(body, path): + calls.append(body) + return 200, {} + + FakeNode.routes[("GET", "/v1/volumes/data")] = incomplete + client = Client(node) + with pytest.raises(ValueError): + client.create_volume("bad/name", 20) + with pytest.raises(ValueError): + client.create_volume("data", 0) + with pytest.raises(ValueError): + client.create_volume("data", True) + with pytest.raises(ValueError): + client.delete_volume("data", '"weak"') + assert calls == [] + with pytest.raises(APIError, match="reply without volume"): + client.get_volume("data") + assert calls == [{}] + + +def test_management_timeout(black_hole): + with pytest.raises(APIError) as raised: + Client(black_hole, timeout=0.05).get_volume("data") + assert raised.value.status == 0 + + +@pytest.mark.parametrize("status", [200, 202, 302]) +def test_delete_requires_confirmed_no_content(node, status): + FakeNode.routes[("DELETE", "/v1/volumes/data")] = lambda body, path: (status, {}) + with pytest.raises(APIError) as raised: + Client(node).delete_volume("data", "vol_123") + assert raised.value.status == status + + +@pytest.mark.parametrize( + "field,value", + [ + ("id", 123), + ("created_at", "not-a-time"), + ("created_at", "2026-10-19T00:00:00+00:99"), + ("created_at", "2026-10-19T00:00:00+01:60"), + ("size_bytes", -1), + ("size_bytes", True), + ("size_bytes", 1.5), + ], +) +def test_managed_response_rejects_invalid_field_types(node, field, value): + volume = { + "name": "data", + "kind": "managed", + "id": "vol_123", + "tenant": "t1", + "node": "node-1", + "state": "ready", + "created_at": "2026-10-19T00:00:00Z", + "size_bytes": 20 << 30, + "available": True, + } + volume[field] = value + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: (200, {"volume": volume}) + with pytest.raises(APIError, match="managed volume"): + Client(node).get_volume("data") + + +@pytest.mark.parametrize( + "timestamp", + [ + "2026-10-19T00:00:00.1Z", + "2026-10-19T00:00:00.123456789Z", + "2026-10-19T00:00:00.123456789+05:30", + ], +) +def test_managed_response_accepts_server_fractional_timestamp(node, timestamp): + volume = { + "name": "data", + "kind": "managed", + "id": "vol_123", + "tenant": "t1", + "node": "node-1", + "state": "ready", + "created_at": timestamp, + "size_bytes": 20 << 30, + "available": True, + } + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: (200, {"volume": volume}) + assert Client(node).get_volume("data")["created_at"] == timestamp + + +@pytest.mark.parametrize("holders", [1, "invalid", {}, None]) +def test_malformed_holder_container_preserves_api_error(node, holders): + FakeNode.routes[("GET", "/v1/volumes/data")] = lambda body, path: (409, {"error": "busy", "holders": holders}) + with pytest.raises(APIError) as raised: + Client(node).get_volume("data") + assert raised.value.status == 409 and raised.value.message == "busy" + assert raised.value.holders == [] + assert raised.value.reply == {"error": "busy", "holders": holders}