sandboxd: per-claim credential injection (#301), streamed egress responses, TLS resumption, egress in the cluster digest

docs/cluster.md

@@ -331,13 +331,15 @@ Some config must match on every node or the cluster fails in confusing ways:
 | `preview_secret` | a preview URL signed on one node fails verification on another |
 | `mesh.cluster_key` | nodes cannot join / decrypt gossip at all |
 | `egress_ca` cluster root | a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception |
+| egress: `egress_classes`, each pool's `egress` and `egress_upstream_env`, `secrets` names, headers and `value_env`, `egress_internal_allow`, `egress_upstream` | a claim or tenant redirected to a divergent node reaches different hosts, and its [claim credentials](egress.md#claim-credentials) are refused or not injected there |
 | the engine root path (cocoon's `root_dir`) | a checkpoint or template export pins the base image blobs' absolute path under the root that captured it; a node whose root sits at another path refuses to clone a record it healed or pulled (`untrusted storage path in snapshot metadata`) — the heal succeeds, the branch fails. Not part of the gossiped digest below |
 
 Each node gossips a digest of these over its live tenant set and quotas (HMAC-keyed by
 `cluster_key` when set, covering each tenant token's SHA-256; otherwise a
 token-free digest of tenant names and quotas + the CA root + `checkpoint_ttl_hours`, so
-nothing brute-forceable rides cleartext gossip), and republishes it after every
-tenant change. A mismatch logs a warning at the moment
+nothing brute-forceable rides cleartext gossip) plus the egress config above in
+an order-free form, never a secret value, and republishes it after every tenant
+change and every config reload. A mismatch logs a warning at the moment
 the divergent node appears — not at the first unlucky redirect — and raises the
 `sandboxd_config_digest_mismatch` gauge. It is warn-only: a rolling credential
 rotation is a legitimate transient mismatch, so a divergence never partitions

docs/egress.md

@@ -285,8 +285,46 @@ gateway key per user, while the guest holds a placeholder.
 - **Audit.** An injection is recorded as the secret's name; the value is
   never logged, listed or returned.
 
-The proxy reads the claim's env under the node's manager lock, once per
-request whose matched rule names a secret; other requests pay nothing.
+The proxy reads a snapshot of the claim's env that each env write publishes,
+without a lock, once per request whose matched rule names a secret or whose
+host has a claim credential; other requests pay nothing.
+
+### Claim credentials
+
+A `guest: false` env entry can also name where its value goes, so a control
+plane binds each sandbox's own credential to a host without editing the node
+config:
+
+```jsonc
+{ "env": {
+    "API_TOKEN": { "value": "Bearer …", "guest": false,
+                   "inject": { "hosts": ["api.example.com", "*.git.example.com"], "header": "Authorization" } }
+} }
+```
+
+On an intercepted request to one of `hosts`, the proxy sets `header` to the
+entry's value, overwriting whatever the guest sent, so the guest can hold a
+placeholder. The entry rides the claim request or any `/env` write, and
+rotation is the same `PATCH`, accepted on a hibernated or archived claim
+without waking it.
+
+- **Envelope.** The operator still decides which hosts can carry a
+  credential: every host must be covered by an `intercept: true` rule of the
+  claim's pool and allowed by its tenant class, or the write answers 400 and
+  nothing is stored. A credential never widens egress and never rides a
+  plaintext forward request.
+- **Precedence.** The pool rule's own `secret` is set first and wins: a
+  credential naming that header for a host the rule covers is a 400. Among
+  credentials for the same header, the first by entry name wins. An empty
+  value injects nothing.
+- **Shape.** `hosts` holds 1 to 8 lowercase names or `*.suffix` patterns
+  (no bare `*`, no port); `header` is any header name except `Host`,
+  `Content-Length` and the hop-by-hop ones. `GET /env` returns `inject` with
+  the value blanked.
+- **Reload.** A reload that drops the covering intercept rule stops that
+  injection on the next request and logs which claims lost which hosts.
+- **Audit.** An injection is recorded as `claim:<entry name>` next to the
+  pool secret's name, comma-separated.
 
 ## Upstream proxies
 
@@ -341,7 +379,11 @@ A rule with `intercept: true` makes the proxy terminate that host's TLS with a
 leaf it signs, so it sees the request's method and path and can inject the
 secret into HTTPS — the same guarantees plaintext already has. Upstream is
 re-originated and verified against the host's real root store; the proxy never
-trusts an unverified origin.
+trusts an unverified origin. A guest that caches TLS sessions resumes them on
+its next CONNECT through the same sandbox, which skips the leaf signature and
+chain check. A response that is an event stream or has no length
+reaches the guest as each chunk arrives, on intercepted and forward requests
+alike.
 
 Trust is a two-tier PKI. One **cluster root CA** is the trust anchor: its
 certificate (public) is baked into every interception guest's store, so a leaf

docs/sandboxd-api.md

@@ -68,12 +68,16 @@ Auth: `Authorization: Bearer <api_token>` (when configured).
   cold-booting an image named like the template
 - `env` is the claim's own environment, at most 64 entries: each name a shell
   identifier of at most 128 bytes, each value at most 8 KiB with no control
-  characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value` and `guest`; any other
+  characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other
   member, or `"guest": null`, answers 400, so a mistyped flag never lands a
   host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a
   `guest: false` entry never enters it and only feeds the node's egress
-  [secrets](egress.md#claim-env-and-secrets). A bad entry answers 400 before
-  any VM is touched; see [claim env](#getputpatch-v1sandboxesidenv)
+  [secrets](egress.md#claim-env-and-secrets), or, with `inject`
+  (`{"hosts": [...], "header": "..."}`), becomes a
+  [claim credential](egress.md#claim-credentials) the proxy sends to those
+  hosts. A bad entry answers 400, and an `inject` host the claim's egress does
+  not intercept answers 400 with the claim's VM released; see
+  [claim env](#getputpatch-v1sandboxesidenv)
 - `egress` `false` claims with no [egress policy](egress.md), whatever the
   pool's (or, for a promoted template, its source pool's): the guest's doors are
   never bound. Omitted or `true` keeps the policy the claim would get
@@ -725,6 +729,11 @@ values:
 {"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}}
 ```
 
+An entry with `inject` comes back with its hosts and header, value blanked.
+An `inject` host the claim's pool does not intercept, or its tenant class does
+not allow, and a header the pool rule's own secret already sets on that host,
+answer 400 and store nothing.
+
 The env is persisted with the claim. A `guest: false` change takes effect on
 the egress proxy's next request at any time, a hibernated or archived claim
 included, and never touches the guest. A change to the guest entries is
@@ -738,8 +747,8 @@ result does not rewrite the claims journal. A 500
 after the store step means the env is stored and its `guest: false` values
 are already live, but the guest write failed; resend the same request to
 deliver it. The guest side is described in [silkd](silkd.md#claim-env).
-`PUT`/`PATCH`/`DELETE` answer 204, `GET` 200; 400 a bad entry or an unknown body
-field; 404 unknown id or another tenant's claim; 409 a guest change on a
+`PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body
+field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a
 paused or failed sandbox.
 
 ## PUT /v1/sandboxes/{id}/instance-metadata

e2e/cmd/credsmoke/main.go

@@ -0,0 +1,212 @@
+// credsmoke is the claim-credential acceptance: claims on one intercept pool each
+// send their own env-injected header to an HTTPS echo through the proxy, the guest
+// never holds the value, and rotation, refusal, clearing and forks behave as the API says.
+package main
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"net/http"
+	"os"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/cocoonstack/sandbox/e2e/internal/harness"
+	sandbox "github.com/cocoonstack/sandbox/sdk/go"
+)
+
+const (
+	proxy  = "http://127.0.0.1:3128"
+	header = "X-Api-Key"
+	entry  = "API_KEY"
+)
+
+func main() {
+	addr := flag.String("addr", "127.0.0.1:7882", "sandboxd address")
+	token := flag.String("token", "", "node api token")
+	template := flag.String("template", "rt:24.04", "template of the intercept pool")
+	echo := flag.String("echo", "postman-echo.com", "HTTPS host that echoes request headers at /headers")
+	secret := flag.String("secret", "", "the pool secret's value the origin should also observe")
+	reattach := flag.String("reattach", "", "id:token:value of a claim to recheck after a restart")
+	sse := flag.String("sse", "", "plain HTTP event stream that sends one event, then holds")
+	flag.Parse()
+	var err error
+	if *reattach != "" {
+		err = recheck(*addr, *token, *echo, *reattach)
+	} else {
+		err = run(*addr, *token, *template, *echo, *secret, *sse)
+	}
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "credsmoke:", err)
+		os.Exit(1)
+	}
+}
+
+func run(addr, token, template, echo, secret, sse string) error {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+	stamp := strconv.FormatInt(time.Now().UnixNano(), 10)
+	valueA, valueB := "key-a-"+stamp, "key-b-"+stamp
+
+	claim := func(env map[string]sandbox.EnvVar) (*sandbox.Sandbox, error) {
+		opts := []sandbox.Option{sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.Small)}
+		if env != nil {
+			opts = append(opts, sandbox.WithEnv(env))
+		}
+		_, sb, err := harness.Claim(ctx, addr, token, template, opts...)
+		return sb, err
+	}
+	a, err := claim(credential(echo, valueA))
+	if err != nil {
+		return err
+	}
+	b, err := claim(credential(echo, valueB))
+	if err != nil {
+		return err
+	}
+	defer func() { _ = b.Close() }()
+	plain, err := claim(nil)
+	if err != nil {
+		return err
+	}
+	defer func() { _ = plain.Close() }()
+	fmt.Printf("  claimed %s (A), %s (B), %s (plain) on one intercept pool\n", a.ID, b.ID, plain.ID)
+
+	for _, tc := range []struct {
+		name      string
+		sb        *sandbox.Sandbox
+		want, not string
+	}{
+		{"A", a, valueA, valueB},
+		{"B", b, valueB, valueA},
+		{"plain", plain, "placeholder", "key-"},
+	} {
+		if err = expectHeader(ctx, tc.sb, echo, tc.want, tc.not, secret); err != nil {
+			return fmt.Errorf("%s: %w", tc.name, err)
+		}
+	}
+	fmt.Println("  each claim's origin saw only its own credential, the plain claim the guest's placeholder, all the pool secret")
+	if sse != "" {
+		out, _ := plain.Exec(ctx, "sh", "-c", fmt.Sprintf("timeout 4 curl -sSN --noproxy '' -x %s %s 2>&1; true", proxy, sse))
+		if !strings.Contains(out, "data: one") {
+			return fmt.Errorf("the first event did not reach the guest while the stream stayed open: %q", out)
+		}
+		fmt.Println("  an open event stream reached the guest event by event through the proxy")
+	}
+
+	leak, err := a.Exec(ctx, "sh", "-c", fmt.Sprintf("grep -rls %q /proc/[0-9]*/environ /run /etc 2>/dev/null; env | grep -c %q; true", valueA, valueA))
+	if err != nil {
+		return fmt.Errorf("leak probe: %w", err)
+	}
+	if strings.TrimSpace(leak) != "0" {
+		return fmt.Errorf("the credential reached the guest: %q", leak)
+	}
+	env, err := a.Env(ctx)
+	if err != nil {
+		return fmt.Errorf("read env: %w", err)
+	}
+	if v := env[entry]; v.Value != "" || v.Inject == nil || v.Inject.Header != header {
+		return fmt.Errorf("GET env %+v, want the inject named with the value blanked", v)
+	}
+	fmt.Println("  value absent from the guest (env, /proc/*/environ, /run, /etc) and blanked in GET env")
+
+	rotated := valueA + "-r1"
+	if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil {
+		return fmt.Errorf("rotate: %w", err)
+	}
+	if err = expectHeader(ctx, a, echo, rotated, valueA+"\"", secret); err != nil {
+		return fmt.Errorf("after a rotation: %w", err)
+	}
+	if err = a.Hibernate(ctx); err != nil {
+		return fmt.Errorf("hibernate: %w", err)
+	}
+	rotated += "-r2"
+	if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil {
+		return fmt.Errorf("rotate while hibernated: %w", err)
+	}
+	if err = expectHeader(ctx, a, echo, rotated, "-r1\"", secret); err != nil {
+		return fmt.Errorf("after a rotation while hibernated: %w", err)
+	}
+	fmt.Println("  rotation applied on the next request, a hibernated claim rotated without waking")
+
+	for name, patch := range map[string]*sandbox.EnvVar{
+		"an uncovered host":        {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{"example.org"}, Header: header}},
+		"the pool secret's header": {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Egress-Token"}},
+		"a guest entry":            {Value: "x", Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Other"}},
+	} {
+		err = b.PatchEnv(ctx, map[string]*sandbox.EnvVar{"BAD": patch})
+		if apiErr, ok := errors.AsType[*sandbox.APIError](err); !ok || apiErr.Status != http.StatusBadRequest {
+			return fmt.Errorf("inject on %s: %v, want 400", name, err)
+		}
+	}
+	if env, err = b.Env(ctx); err != nil || len(env) != 1 {
+		return fmt.Errorf("env after refused patches %v %v, want only the original entry", env, err)
+	}
+	if err = b.SetEnv(ctx, map[string]sandbox.EnvVar{}); err != nil {
+		return fmt.Errorf("clear: %w", err)
+	}
+	if err = expectHeader(ctx, b, echo, "placeholder", valueB, secret); err != nil {
+		return fmt.Errorf("after a clear: %w", err)
+	}
+	fmt.Println("  uncovered host, pool secret header and guest entry refused with 400; a cleared env stops injecting")
+
+	children, err := a.Fork(ctx, 1, time.Minute)
+	if err != nil {
+		return fmt.Errorf("fork: %w", err)
+	}
+	defer func() { _ = children[0].Close() }()
+	if err = expectHeader(ctx, children[0], echo, "placeholder", rotated, secret); err != nil {
+		return fmt.Errorf("fork child: %w", err)
+	}
+	fmt.Println("  a fork child carries no credential")
+	fmt.Printf("REATTACH %s:%s:%s\n", a.ID, a.Token(), rotated)
+	fmt.Println("CREDSMOKE PASS")
+	return nil
+}
+
+func recheck(addr, token, echo, reattach string) error {
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+	defer cancel()
+	parts := strings.SplitN(reattach, ":", 3)
+	if len(parts) != 3 {
+		return fmt.Errorf("-reattach %q, want id:token:value", reattach)
+	}
+	client, err := sandbox.Connect(addr, sandbox.WithAPIToken(token))
+	if err != nil {
+		return err
+	}
+	sb, err := client.Lookup(ctx, parts[0], parts[1])
+	if err != nil {
+		return fmt.Errorf("lookup after restart: %w", err)
+	}
+	defer func() { _ = sb.Close() }()
+	if err = expectHeader(ctx, sb, echo, parts[2], "placeholder", ""); err != nil {
+		return fmt.Errorf("after a restart: %w", err)
+	}
+	fmt.Println("  the credential survived a sandboxd restart")
+	fmt.Println("CREDSMOKE REATTACH PASS")
+	return nil
+}
+
+func credential(echo, value string) map[string]sandbox.EnvVar {
+	return map[string]sandbox.EnvVar{entry: {Value: value, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: header}}}
+}
+
+func expectHeader(ctx context.Context, sb *sandbox.Sandbox, echo, want, not, secret string) error {
+	out, err := sb.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sS -x %s -H '%s: placeholder' https://%s/headers", proxy, header, echo))
+	if err != nil {
+		return fmt.Errorf("proxied HTTPS exec: %w (%s)", err, out)
+	}
+	switch {
+	case !strings.Contains(out, want):
+		return fmt.Errorf("origin did not see %q:\n%s", want, out)
+	case not != "" && strings.Contains(out, not):
+		return fmt.Errorf("origin saw %q:\n%s", not, out)
+	case secret != "" && !strings.Contains(out, secret):
+		return fmt.Errorf("origin did not see the pool secret:\n%s", out)
+	}
+	return nil
+}

sandboxd/config/config.go

@@ -272,13 +272,14 @@ func (c *Config) HasEgress() bool {
 	return len(c.Bridges) > 0 || len(c.Networks) > 0
 }
 
-// ClusterDigest fingerprints the must-match config and live tenant records; without cluster_key it omits token hashes.
+// ClusterDigest fingerprints the must-match config, its egress layer and the live tenant records; without cluster_key it omits token hashes.
 func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) string {
 	tenants = slices.Clone(tenants)
 	slices.SortFunc(tenants, func(a, b TenantRecord) int { return strings.Compare(a.Name, b.Name) })
+	eg := c.egressFingerprint()
 	if c.Mesh != nil {
 		if key, _ := c.Mesh.DecodedKey(); key != nil {
-			raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours})
+			raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours, eg})
 			mac := hmac.New(sha256.New, key)
 			mac.Write(raw)
 			return hex.EncodeToString(mac.Sum(nil))
@@ -287,7 +288,37 @@ func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) str
 	for i := range tenants {
 		tenants[i].TokenSHA256 = ""
 	}
-	return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours})
+	return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours, eg})
+}
+
+type egressFingerprint struct {
+	Classes  []EgressClass         `json:"classes,omitempty"`
+	Secrets  []egress.SecretSpec   `json:"secrets,omitempty"`
+	Internal []string              `json:"internal,omitempty"`
+	Upstream *EgressUpstreamConfig `json:"upstream,omitempty"`
+	Pools    map[string]PoolSpec   `json:"pools,omitempty"`
+}
+
+// egressFingerprint is the egress config every node must share, in an order-free form; secret values stay out.
+func (c *Config) egressFingerprint() egressFingerprint {
+	fp := egressFingerprint{
+		Classes:  slices.SortedFunc(slices.Values(c.EgressClasses), func(a, b EgressClass) int { return strings.Compare(a.Name, b.Name) }),
+		Internal: slices.Sorted(slices.Values(c.EgressInternalAllow)),
+		Pools:    map[string]PoolSpec{},
+	}
+	for _, s := range c.Secrets {
+		fp.Secrets = append(fp.Secrets, egress.SecretSpec{Name: s.Name, Header: s.Header, ValueEnv: s.ValueEnv})
+	}
+	slices.SortFunc(fp.Secrets, func(a, b egress.SecretSpec) int { return strings.Compare(a.Name, b.Name) })
+	if u := c.EgressUpstream; u != nil {
+		fp.Upstream = &EgressUpstreamConfig{ClaimEnv: u.ClaimEnv, Allow: slices.Sorted(slices.Values(u.Allow))}
+	}
+	for _, p := range c.Pools {
+		if p.Egress != nil || p.EgressUpstreamEnv != "" {
+			fp.Pools[poolLabel(p.PoolKey)] = PoolSpec{Egress: p.Egress, EgressUpstreamEnv: p.EgressUpstreamEnv}
+		}
+	}
+	return fp
 }
 
 func (c *Config) guardsEgressLane() bool {