From 9059b83712c1dd62869ff7c29515a2b61709ed8b Mon Sep 17 00:00:00 2001 From: doge Date: Thu, 30 Jul 2026 14:49:12 +0800 Subject: [PATCH] Shard egress-lane VMs over several raw bridges MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The networks list shards over CNI conflists, and the CNI attachment is what it prices in: a per-VM network namespace, a CNI plugin execution, a veth pair and a TC redirect on every clone. Measured on a 384-core host filling 1000 egress VMs (quiet console, udev exec queue stopped), the conflist shards ran at 25.8/s where a single raw bridge ran 48/s — the sharding win was smaller than the machinery cost. bridges is the same shard list over host bridge devices with the raw TAP-on-bridge attachment kept: one tap in the root netns, no netns, no plugin, no veth. The same fill runs at 83.3/s — 3.2x the conflist shards, level with four separate daemons sharding one bridge each without the fourfold goldens — and the ceiling story is unchanged: each entry is one Linux bridge, so N entries give N x BR_MAX_PORTS. Because the taps stay in the root netns, guarded egress works over bridges, which the conflist form cannot offer. The shard pick is the same VM-name hash networks uses, so a record's attachment stays reproducible without process state. The scalar bridge key is retired like network before it — a one-entry bridges list is the same argv, and strict config decoding fails the old key loudly. --- docs/cluster.md | 2 +- docs/deploy.md | 11 ++++-- docs/egress.md | 10 ++--- e2e/fakeengine_test.go | 2 +- sandboxd/config/config.go | 36 ++++++++++------- sandboxd/config/config_test.go | 25 ++++++++---- sandboxd/engine/cloneargs_test.go | 66 +++++++++++++++++-------------- sandboxd/engine/engine.go | 40 +++++++++++-------- sandboxd/engine/engine_test.go | 16 ++++---- sandboxd/engine/installca_test.go | 6 +-- sandboxd/main.go | 2 +- sandboxd/pool/egress_test.go | 2 +- sandboxd/pool/pool.go | 2 +- scripts/egress-e2e.sh | 2 +- scripts/sandboxd-e2e.sh | 2 +- 15 files changed, 130 insertions(+), 94 deletions(-) diff --git a/docs/cluster.md b/docs/cluster.md index eab2dfa..97dae10 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -179,7 +179,7 @@ fully from it: | state | source of truth | survives restart | |---|---|---| -| operator config (`tenants`, `secrets`, egress policies, `bridge`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot | +| operator config (`tenants`, `secrets`, egress policies, `bridges`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot | | API-applied pool targets (`PUT /v1/pools`) | `/pools.json` (machine owned) | yes | | claims | the claims journal + `Reconcile` | yes | | placement hints (warm counts, template sets) | gossip | rebuilt | diff --git a/docs/deploy.md b/docs/deploy.md index 7e3b0b7..cff6ddd 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -32,6 +32,11 @@ This CH-only release does not convert existing VM or snapshot state. Drain old claims and use fresh `data_dir` and `checkpoint_dir` locations when upgrading; older checkpoints and promoted templates must not be reused. +The scalar egress-attachment keys are retired: rename `"bridge": "br0"` to +`"bridges": ["br0"]` and `"network": "cni"` to `"networks": ["cni"]` before +starting the new binary — config loading rejects the old spellings loudly +rather than silently dropping the egress lane. + ## Configuration sandboxd reads one JSON file (`-config`, default @@ -45,7 +50,7 @@ sandboxd reads one JSON file (`-config`, default "restore_mode": "mmap", "no_direct_io": true, "advertise_addr": "10.0.0.5:7777", - "bridge": "br0", + "bridges": ["br0"], "api_token": "…", "mesh": { "node_id": "node-a", @@ -68,7 +73,7 @@ sandboxd reads one JSON file (`-config`, default | `restore_mode` | unset | clone and wake-restore memory mode: `copy`, `ondemand`, or `mmap`; use `mmap` for dense pools | | `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention | | `advertise_addr` | = `listen` | the host:port clients reach this node at; returned as a claim's owner address and gossiped to peers. Must be routable when `listen` is a wildcard | -| `bridge` / `networks` | unset | egress-lane attachment: a host bridge device, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. Each conflist attaches its own Linux bridge and a bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so N conflists raise the node's egress ceiling to N×1024 — VMs spread over them by a stable hash of the VM name, so size the list with headroom (the spread is statistical, not exact). [Guarded egress](egress.md) needs the bridge form and rejects a CNI network at load | +| `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) needs `bridges` and rejects a CNI network at load | | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` | | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls | | `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview) and everything it creates is stamped with the tenant name; operator surfaces (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set (operator surfaces need it). Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays a tenant token across a redirect; a peer missing that tenant answers 401), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior | @@ -112,7 +117,7 @@ here validates on load: "listen": ":7777", "data_dir": "/var/lib/sandboxd", "advertise_addr": "10.0.0.5:7777", - "bridge": "br0", + "bridges": ["br0"], "restore_mode": "mmap", "no_direct_io": true, diff --git a/docs/egress.md b/docs/egress.md index 9aca988..dee6ab2 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -83,10 +83,10 @@ domain policy first; the allow-list widens the IP gate only. reach the local L2 segment (never routed off-link). Give egress-lane VMs a bridge they do not share with an untrusted listener. - **Bridge lane only (egress lane).** A CNI network's tap lives in the VM netns, - out of reach of the root-netns lock, so a guarded egress *lane* needs a bridge - and is rejected on CNI `networks`. None-lane policies ride the proxy and work - on either. A bridge egress lane locks every NIC default-deny, even with no - policy configured. + out of reach of the root-netns lock, so a guarded egress *lane* needs the + `bridges` form (those taps stay in the root netns) and is rejected on CNI + `networks`. None-lane policies ride the proxy and work on either. A bridge + egress lane locks every NIC default-deny, even with no policy configured. - **No custom NAT64/DNS64 prefix routed to the host.** The SSRF guard folds the standard NAT64 forms (RFC 6052 well-known `64:ff9b::/96`, RFC 8215 local-use `64:ff9b:1::/48`), but an operator-specific network-specific prefix (RFC 6052 @@ -103,7 +103,7 @@ the environment, never the config file. ```jsonc { - "bridge": "sbxbr0", // egress-lane pools only; none-lane needs no attachment + "bridges": ["sbxbr0"], // egress-lane pools only; none-lane needs no attachment "secrets": [ { "name": "gh", "header": "Authorization", "value_env": "GH_TOKEN" } ], diff --git a/e2e/fakeengine_test.go b/e2e/fakeengine_test.go index 1103ab7..d33124e 100644 --- a/e2e/fakeengine_test.go +++ b/e2e/fakeengine_test.go @@ -31,7 +31,7 @@ type fakeEngine struct { func newFakeEngine(dir string) *fakeEngine { return &fakeEngine{ - real: engine.New("cocoon", "", nil, false, ""), + real: engine.New("cocoon", nil, nil, false, ""), dir: dir, listeners: map[string]io.Closer{}, socks: map[string]string{}, diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index c51f3ee..2abc2c2 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -170,10 +170,11 @@ type Config struct { // to Listen, which is correct when Listen is a routable host:port. AdvertiseAddr string `json:"advertise_addr,omitempty"` - // Bridge and Networks pick the egress-lane attachment (TAP-on-bridge vs - // CNI conflists); mutually exclusive. With neither set the node serves - // only the no-network lane. - Bridge string `json:"bridge,omitempty"` + // Bridges shards egress-lane VMs over several host bridge devices with + // the raw TAP-on-bridge attachment: taps stay in the root netns, so + // guarded egress can lock them. Mutually exclusive with Networks; with + // neither set the node serves only the no-network lane. + Bridges []string `json:"bridges,omitempty"` // Networks shards egress-lane VMs over several CNI conflists, one Linux // bridge each: a bridge holds at most 1024 ports (BR_MAX_PORTS, no @@ -276,7 +277,7 @@ type Config struct { // HasEgress reports whether the node can attach egress-lane VMs. func (c *Config) HasEgress() bool { - return c.Bridge != "" || len(c.Networks) > 0 + return len(c.Bridges) > 0 || len(c.Networks) > 0 } // ClusterDigest fingerprints the must-match config so a divergent node is @@ -336,7 +337,7 @@ func autoRefillConcurrency(cpus int) int { } func (c *Config) validate() error { - if err := c.validateNetworks(); err != nil { + if err := c.validateAttachment(); err != nil { return err } // A CNI network's tap lives in the VM netns, unreachable from the root-netns @@ -459,19 +460,26 @@ func (c *Config) validateSecrets() (map[string]struct{}, error) { return names, nil } -// validateNetworks checks the egress-lane attachment; a repeated conflist +// validateAttachment checks the egress-lane attachment; a repeated shard // would report N shards while filling one bridge. -func (c *Config) validateNetworks() error { - if c.Bridge != "" && len(c.Networks) > 0 { - return fmt.Errorf("bridge and networks are mutually exclusive") +func (c *Config) validateAttachment() error { + if len(c.Bridges) > 0 && len(c.Networks) > 0 { + return fmt.Errorf("bridges and networks are mutually exclusive") } - seen := make(map[string]struct{}, len(c.Networks)) - for _, n := range c.Networks { + if err := validateShards(c.Bridges, "bridges", "bridge device"); err != nil { + return err + } + return validateShards(c.Networks, "networks", "conflist") +} + +func validateShards(names []string, field, kind string) error { + seen := make(map[string]struct{}, len(names)) + for _, n := range names { if n == "" { - return fmt.Errorf("networks must not contain an empty conflist name") + return fmt.Errorf("%s must not contain an empty %s name", field, kind) } if _, ok := seen[n]; ok { - return fmt.Errorf("duplicate conflist %q", n) + return fmt.Errorf("duplicate %s %q", kind, n) } seen[n] = struct{}{} } diff --git a/sandboxd/config/config_test.go b/sandboxd/config/config_test.go index e757e83..f9ea8c2 100644 --- a/sandboxd/config/config_test.go +++ b/sandboxd/config/config_test.go @@ -85,7 +85,6 @@ func TestLoadRejectsInvalid(t *testing.T) { name, body, want string }{ {"bad json", `{`, "config"}, - {"bridge and networks", `{"bridge":"br0","networks":["cni"],"pools":[]}`, "mutually exclusive"}, {"bad fork count", `{"max_fork_count":-1,"pools":[]}`, "max_fork_count"}, {"negative refill concurrency", `{"refill_concurrency":-1,"pools":[]}`, "refill_concurrency"}, {"bad restore mode", `{"restore_mode":"Mmap","pools":[]}`, "restore_mode"}, @@ -218,14 +217,14 @@ func TestHasEgress(t *testing.T) { if (&Config{}).HasEgress() { t.Error("no attachment must mean no egress") } - if !(&Config{Bridge: "br0"}).HasEgress() || !(&Config{Networks: []string{"cni"}}).HasEgress() { - t.Error("bridge or network must enable egress") + if !(&Config{Bridges: []string{"br0", "br1"}}).HasEgress() || !(&Config{Networks: []string{"cni"}}).HasEgress() { + t.Error("bridges or networks must enable egress") } } func TestLoadKeepsExplicitValues(t *testing.T) { path := writeConfig(t, `{"listen":"0.0.0.0:9999","advertise_addr":"10.0.0.5:9999","max_fork_count":4, - "refill_concurrency":8,"no_direct_io":true,"bridge":"br0","pools":[{"template":"rt:24.04","net":"egress","size":"small","warm":3}]}`) + "refill_concurrency":8,"no_direct_io":true,"bridges":["br0"],"pools":[{"template":"rt:24.04","net":"egress","size":"small","warm":3}]}`) cfg, err := Load(path) if err != nil { t.Fatalf("Load: %v", err) @@ -240,10 +239,13 @@ func TestLoadKeepsExplicitValues(t *testing.T) { func TestLoadRejectsUnusableNetworkLists(t *testing.T) { for name, body := range map[string]string{ - "bridge and networks together": `{"bridge":"br0","networks":["a"],"pools":[]}`, - "retired scalar network key": `{"network":"a","pools":[]}`, - "empty conflist name": `{"networks":["a",""],"pools":[]}`, - "repeated conflist": `{"networks":["a","b","a"],"pools":[]}`, + "bridges and networks together": `{"bridges":["br0"],"networks":["a"],"pools":[]}`, + "retired scalar network key": `{"network":"a","pools":[]}`, + "retired scalar bridge key": `{"bridge":"br0","pools":[]}`, + "empty conflist name": `{"networks":["a",""],"pools":[]}`, + "empty bridge name": `{"bridges":["br0",""],"pools":[]}`, + "repeated conflist": `{"networks":["a","b","a"],"pools":[]}`, + "repeated bridge": `{"bridges":["br0","br1","br0"],"pools":[]}`, } { t.Run(name, func(t *testing.T) { if _, err := Load(writeConfig(t, body)); err == nil { @@ -253,6 +255,13 @@ func TestLoadRejectsUnusableNetworkLists(t *testing.T) { } } +func TestLoadAcceptsGuardedEgressOnABridgesList(t *testing.T) { + path := writeConfig(t, `{"bridges":["sbx0","sbx1"],"pools":[{"template":"rt:24.04","net":"egress","size":"small","egress":{"allow":[{"host":"x"}]}}]}`) + if _, err := Load(path); err != nil { + t.Fatalf("guarded egress on a bridges list must load (taps stay in the root netns): %v", err) + } +} + func TestLoadAcceptsAShardedNetworkList(t *testing.T) { path := writeConfig(t, `{"networks":["cocoon-sbx0","cocoon-sbx1","cocoon-sbx2","cocoon-sbx3"],"pools":[]}`) cfg, err := Load(path) diff --git a/sandboxd/engine/cloneargs_test.go b/sandboxd/engine/cloneargs_test.go index f828a2d..ffa2540 100644 --- a/sandboxd/engine/cloneargs_test.go +++ b/sandboxd/engine/cloneargs_test.go @@ -23,7 +23,7 @@ func TestCloneArgsRestoreMode(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - e := New("cocoon", "br0", nil, false, tc.mode) + e := New("cocoon", []string{"br0"}, nil, false, tc.mode) for _, args := range [][]string{ e.cloneArgs("/goldens/g1", "sbx-1", tc.key), e.cloneSnapArgs("ck_1", "sbx-1", tc.key), @@ -45,7 +45,7 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) { key := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeMedium} for _, noDirectIO := range []bool{false, true} { t.Run(strconv.FormatBool(noDirectIO), func(t *testing.T) { - e := New("cocoon", "", nil, noDirectIO, "") + e := New("cocoon", nil, nil, noDirectIO, "") want := "--no-direct-io=" + strconv.FormatBool(noDirectIO) cold := e.runColdArgs("sbx-1", key) for _, args := range [][]string{ @@ -64,38 +64,46 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) { } } -// TestEgressVMsSpreadOverEveryConfiguredNetwork pins the point of the list: -// a shard the hash never picks is bridge capacity that does not exist. -func TestEgressVMsSpreadOverEveryConfiguredNetwork(t *testing.T) { - nets := []string{"cocoon-sbx0", "cocoon-sbx1", "cocoon-sbx2", "cocoon-sbx3"} - e := New("cocoon", "", nets, false, "") +// TestEgressVMsSpreadOverEveryConfiguredShard pins the point of both shard +// lists: a shard the hash never picks is bridge capacity that does not exist. +func TestEgressVMsSpreadOverEveryConfiguredShard(t *testing.T) { + shards := []string{"sbx0", "sbx1", "sbx2", "sbx3"} key := types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeMedium} - - counts := map[string]int{} - for i := range 4000 { - args := e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key) - j := slices.Index(args, "--network") - if j < 0 { - t.Fatalf("args %v carry no --network", args) - } - counts[args[j+1]]++ - } - for _, n := range nets { - // The bound catches a starved shard, not non-uniformity. - if counts[n] < 4000/len(nets)/2 { - t.Errorf("shard %s got %d of 4000, want a fair share: %v", n, counts[n], counts) - } + for name, tc := range map[string]struct { + e *Engine + flag string + }{ + "networks": {New("cocoon", nil, shards, false, ""), "--network"}, + "bridges": {New("cocoon", shards, nil, false, ""), "--bridge"}, + } { + t.Run(name, func(t *testing.T) { + counts := map[string]int{} + for i := range 4000 { + args := tc.e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key) + j := slices.Index(args, tc.flag) + if j < 0 { + t.Fatalf("args %v carry no %s", args, tc.flag) + } + counts[args[j+1]]++ + } + for _, s := range shards { + // The bound catches a starved shard, not non-uniformity. + if counts[s] < 4000/len(shards)/2 { + t.Errorf("shard %s got %d of 4000, want a fair share: %v", s, counts[s], counts) + } + } + }) } } // TestNetworkChoiceIsStableForAName guards restore and teardown: the record // persists the network a VM was built on, so the choice must not move. func TestNetworkChoiceIsStableForAName(t *testing.T) { - e := New("cocoon", "", []string{"a", "b", "c"}, false, "") - first := e.networkFor("sbx-pool1-42") + shards := []string{"a", "b", "c"} + first := shardOf(shards, "sbx-pool1-42") for range 100 { - if got := e.networkFor("sbx-pool1-42"); got != first { - t.Fatalf("networkFor returned %q then %q for one name", first, got) + if got := shardOf(shards, "sbx-pool1-42"); got != first { + t.Fatalf("shardOf returned %q then %q for one name", first, got) } } } @@ -107,13 +115,13 @@ func TestNetArgsHonorsTheLaneAndTheAttachment(t *testing.T) { none := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeMedium} egress := types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeMedium} - if args := New("cocoon", "", []string{"cni"}, false, "").netArgs("sbx-1", none, false); len(args) != 0 { + if args := New("cocoon", nil, []string{"cni"}, false, "").netArgs("sbx-1", none, false); len(args) != 0 { t.Errorf("none lane took an attachment: %v", args) } - if args := New("cocoon", "br0", nil, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--bridge", "br0"}) { + if args := New("cocoon", []string{"br0"}, nil, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--bridge", "br0"}) { t.Errorf("bridge lane args = %v", args) } - if args := New("cocoon", "", []string{"cocoon-dhcp"}, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--network", "cocoon-dhcp"}) { + if args := New("cocoon", nil, []string{"cocoon-dhcp"}, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--network", "cocoon-dhcp"}) { t.Errorf("single-network args = %v", args) } } diff --git a/sandboxd/engine/engine.go b/sandboxd/engine/engine.go index d86fe1d..2313b64 100644 --- a/sandboxd/engine/engine.go +++ b/sandboxd/engine/engine.go @@ -34,6 +34,8 @@ const ( argName = "--name" argOutput = "--output" + argNetwork = "--network" + argBridge = "--bridge" formatJSON = "json" silkdPort = 2048 // silkd's fixed guest vsock port, the claim-ready anchor egressPort = 2049 // guest→host egress port; VMM maps it to _2049 @@ -60,16 +62,17 @@ var capacitySignatures = []string{ // Engine runs cocoon commands on the local node. type Engine struct { bin string - bridge string + bridges []string networks []string noDirectIO bool restoreMode types.RestoreMode } -// New returns a cocoon engine with node-wide network and disk policy. networks -// are CNI conflists to spread egress-lane VMs over; see networkFor. -func New(bin, bridge string, networks []string, noDirectIO bool, restoreMode types.RestoreMode) *Engine { - return &Engine{bin: bin, bridge: bridge, networks: networks, noDirectIO: noDirectIO, restoreMode: restoreMode} +// New returns a cocoon engine with node-wide network and disk policy. bridges +// and networks are the egress-lane shard lists (host bridge devices vs CNI +// conflists) egress VMs spread over; see shardOf. +func New(bin string, bridges, networks []string, noDirectIO bool, restoreMode types.RestoreMode) *Engine { + return &Engine{bin: bin, bridges: bridges, networks: networks, noDirectIO: noDirectIO, restoreMode: restoreMode} } // Version reports cocoon's version string — a "vX.Y.Z" release or a @@ -351,19 +354,9 @@ func (e *Engine) netArgs(name string, key types.PoolKey, cold bool) []string { return nil } if len(e.networks) > 0 { - return []string{"--network", e.networkFor(name)} + return []string{argNetwork, shardOf(e.networks, name)} } - return []string{"--bridge", e.bridge} -} - -// networkFor picks a conflist by hashing the VM name, not by counter: the -// record persists the network a VM was built on, so the choice must be -// reproducible without process state. -func (e *Engine) networkFor(name string) string { - h := fnv.New32a() - _, _ = h.Write([]byte(name)) - // >>1 keeps the index conversion positive even where int is 32-bit. - return e.networks[int(h.Sum32()>>1)%len(e.networks)] + return []string{argBridge, shardOf(e.bridges, name)} } func (e *Engine) run(ctx context.Context, args ...string) ([]byte, error) { @@ -416,6 +409,19 @@ func EgressSocketPath(vsockSocket string) string { return fmt.Sprintf("%s_%d", vsockSocket, egressPort) } +// shardOf picks a shard by hashing the VM name, not by counter: the record +// persists the attachment a VM was built on, so the choice must be +// reproducible without process state. +func shardOf(shards []string, name string) string { + if len(shards) == 0 { + return "" + } + h := fnv.New32a() + _, _ = h.Write([]byte(name)) + // >>1 keeps the index conversion positive even where int is 32-bit. + return shards[int(h.Sum32()>>1)%len(shards)] +} + // respFail renders a non-success reply: the error frame's own text, or the // unexpected frame's type. func respFail(resp wire.Response) string { diff --git a/sandboxd/engine/engine_test.go b/sandboxd/engine/engine_test.go index 5981167..d16967a 100644 --- a/sandboxd/engine/engine_test.go +++ b/sandboxd/engine/engine_test.go @@ -24,7 +24,7 @@ func TestDialSilkdConsumesOnlyHandshake(t *testing.T) { // over-reads past the newline, the first Read below loses it. listenMuxer(t, path, "OK 2048\nX") - conn, err := New("cocoon", "", nil, false, "").DialSilkd(t.Context(), path) + conn, err := New("cocoon", nil, nil, false, "").DialSilkd(t.Context(), path) if err != nil { t.Fatalf("DialSilkd: %v", err) } @@ -42,7 +42,7 @@ func TestDialSilkdRejectedHandshake(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "ERR no guest listener\n") - _, err := New("cocoon", "", nil, false, "").DialSilkd(t.Context(), path) + _, err := New("cocoon", nil, nil, false, "").DialSilkd(t.Context(), path) if err == nil || !strings.Contains(err.Error(), "no guest listener") { t.Errorf("got %v, want handshake rejection", err) } @@ -52,7 +52,7 @@ func TestProbeSucceeds(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", infoFrame) - if err := New("cocoon", "", nil, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { + if err := New("cocoon", nil, nil, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { t.Errorf("Probe: %v", err) } } @@ -61,7 +61,7 @@ func TestInfoRoundTripRejectsErrorFrame(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", errFrame) - err := New("cocoon", "", nil, false, "").infoRoundTrip(t.Context(), path) + err := New("cocoon", nil, nil, false, "").infoRoundTrip(t.Context(), path) if err == nil || !strings.Contains(err.Error(), `info reply type "error"`) { t.Errorf("got %v, want error-frame rejection", err) } @@ -71,7 +71,7 @@ func TestProbeRetriesPastFailures(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", errFrame, errFrame, infoFrame) - if err := New("cocoon", "", nil, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { + if err := New("cocoon", nil, nil, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { t.Errorf("Probe: %v", err) } } @@ -80,7 +80,7 @@ func TestProbeRetriesUntilListenerAppears(t *testing.T) { path := sockPath(t) done := make(chan error, 1) go func() { - done <- New("cocoon", "", nil, false, "").Probe(t.Context(), path, 2*time.Second) + done <- New("cocoon", nil, nil, false, "").Probe(t.Context(), path, 2*time.Second) }() time.Sleep(60 * time.Millisecond) @@ -118,7 +118,7 @@ func TestDialGuestPortCtxCancel(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), 150*time.Millisecond) defer cancel() - if _, err := New("cocoon", "", nil, false, "").DialGuestPort(ctx, path, 8080); !errors.Is(err, context.DeadlineExceeded) { + if _, err := New("cocoon", nil, nil, false, "").DialGuestPort(ctx, path, 8080); !errors.Is(err, context.DeadlineExceeded) { t.Errorf("got %v, want context.DeadlineExceeded", err) } } @@ -126,7 +126,7 @@ func TestDialGuestPortCtxCancel(t *testing.T) { func TestProbeTimeout(t *testing.T) { path := sockPath(t) - err := New("cocoon", "", nil, false, "").Probe(t.Context(), path, 150*time.Millisecond) + err := New("cocoon", nil, nil, false, "").Probe(t.Context(), path, 150*time.Millisecond) if err == nil || !strings.Contains(err.Error(), "silkd probe") { t.Errorf("got %v, want probe timeout", err) } diff --git a/sandboxd/engine/installca_test.go b/sandboxd/engine/installca_test.go index 5f7c125..7ab237f 100644 --- a/sandboxd/engine/installca_test.go +++ b/sandboxd/engine/installca_test.go @@ -14,7 +14,7 @@ import ( func TestInstallCACertWritesCertAndUpdates(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - if err := New("cocoon", "", nil, false, "").InstallCACert(t.Context(), path, []byte("CERT-PEM")); err != nil { + if err := New("cocoon", nil, nil, false, "").InstallCACert(t.Context(), path, []byte("CERT-PEM")); err != nil { t.Fatalf("InstallCACert: %v", err) } fake.mu.Lock() @@ -44,7 +44,7 @@ func TestInstallCACertNonzeroExitFails(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) fake.execCode = 3 - err := New("cocoon", "", nil, false, "").InstallCACert(t.Context(), path, []byte("x")) + err := New("cocoon", nil, nil, false, "").InstallCACert(t.Context(), path, []byte("x")) if err == nil || !strings.Contains(err.Error(), "exit code 3") { t.Errorf("got %v, want exit code 3 failure", err) } @@ -54,7 +54,7 @@ func TestInstallCACertWriteErrorFrameFails(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) fake.writeErr = "disk full" - err := New("cocoon", "", nil, false, "").InstallCACert(t.Context(), path, []byte("x")) + err := New("cocoon", nil, nil, false, "").InstallCACert(t.Context(), path, []byte("x")) if err == nil || !strings.Contains(err.Error(), "disk full") { t.Errorf("got %v, want fs_write error-frame failure", err) } diff --git a/sandboxd/main.go b/sandboxd/main.go index 4356e27..071c13d 100644 --- a/sandboxd/main.go +++ b/sandboxd/main.go @@ -70,7 +70,7 @@ func main() { if err != nil { logger.Fatalf(ctx, err, "load config") } - eng := engine.New(cfg.CocoonBin, cfg.Bridge, cfg.Networks, cfg.NoDirectIO, cfg.RestoreMode) + eng := engine.New(cfg.CocoonBin, cfg.Bridges, cfg.Networks, cfg.NoDirectIO, cfg.RestoreMode) if v, warn := eng.VersionWarning(ctx); warn != "" { logger.Warn(ctx, warn) } else { diff --git a/sandboxd/pool/egress_test.go b/sandboxd/pool/egress_test.go index 957cff9..5af85ca 100644 --- a/sandboxd/pool/egress_test.go +++ b/sandboxd/pool/egress_test.go @@ -529,7 +529,7 @@ func egressManager(t *testing.T, eng *fakeEngine, pools ...config.PoolSpec) *Man t.Helper() t.Setenv("GH_TOKEN", "s3cr3t") secrets := testSecrets(t, egress.SecretSpec{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}) - cfg := &config.Config{DataDir: t.TempDir(), Bridge: "sbxbr0", EgressCA: writeTestEgressCA(t), Pools: pools} + cfg := &config.Config{DataDir: t.TempDir(), Bridges: []string{"sbxbr0"}, EgressCA: writeTestEgressCA(t), Pools: pools} m, err := NewManager(t.Context(), cfg, eng, secrets) if err != nil { t.Fatalf("manager: %v", err) diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 3d5d592..f7f213d 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -367,7 +367,7 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg eng: eng, dataDir: cfg.DataDir, egress: cfg.HasEgress(), - lockEgress: cfg.Bridge != "", + lockEgress: len(cfg.Bridges) > 0, maxFork: maxFork, store: newClaimStore(cfg.DataDir), poolStore: newPoolStore(cfg.DataDir), diff --git a/scripts/egress-e2e.sh b/scripts/egress-e2e.sh index b7678a0..672e729 100755 --- a/scripts/egress-e2e.sh +++ b/scripts/egress-e2e.sh @@ -21,7 +21,7 @@ NICADDR="" if [[ $NET == egress ]]; then REACH=$(ip -4 addr show "$BRIDGE" 2>/dev/null | awk '/inet /{print $2}' | cut -d/ -f1 | head -1) [[ -z $REACH ]] && { echo "bridge $BRIDGE has no IPv4 — egress lane needs a gateway"; exit 1; } - BRIDGE_CFG="\"bridge\": \"$BRIDGE\"," + BRIDGE_CFG="\"bridges\": [\"$BRIDGE\"]," NICADDR="${REACH%.*}.222/24" # static guest NIC (image does not DHCP), so a blocked direct egress is the nft lock fi diff --git a/scripts/sandboxd-e2e.sh b/scripts/sandboxd-e2e.sh index 89d4e86..fcfc032 100755 --- a/scripts/sandboxd-e2e.sh +++ b/scripts/sandboxd-e2e.sh @@ -63,7 +63,7 @@ fi BRIDGE_LINE="" EGRESS_POOL="" if [[ -n ${BRIDGE:-} ]]; then - BRIDGE_LINE="\"bridge\": \"$BRIDGE\"," + BRIDGE_LINE="\"bridges\": [\"$BRIDGE\"]," EGRESS_POOL=", {\"template\": \"$TEMPLATE\", \"net\": \"egress\", \"size\": \"small\", \"warm\": 1}" fi