From 32cb967b95eee9f05ac4920284084c0e99195c11 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:45:04 +0800 Subject: [PATCH 1/6] sandboxd: per-claim credential injection through env inject (#301) A guest: false env entry may name hosts and a header; the egress proxy sets its value there on intercepted requests only. Every host must be intercepted by the claim's pool and allowed by its tenant class, and may not take a header the pool rule's own secret sets, or the claim or env write answers 400. The proxy reads a per-sandbox env snapshot published on each env write instead of taking the manager lock per request. --- docs/egress.md | 42 +++++++- docs/sandboxd-api.md | 19 +++- sandboxd/egress/intercept_test.go | 53 +++++++++- sandboxd/egress/policy.go | 21 ++++ sandboxd/egress/policy_test.go | 28 +++++ sandboxd/egress/proxy.go | 42 ++++++-- sandboxd/egress/proxy_test.go | 43 ++++++++ sandboxd/egress/secrets.go | 18 +++- sandboxd/pool/checkpoint.go | 2 +- sandboxd/pool/claim.go | 10 +- sandboxd/pool/egress.go | 8 +- sandboxd/pool/egress_test.go | 7 +- sandboxd/pool/env.go | 56 ++++++++-- sandboxd/pool/env_test.go | 170 +++++++++++++++++++++++++++++- sandboxd/pool/reconcile.go | 1 + sandboxd/pool/reload.go | 20 ++++ sandboxd/pool/upstream.go | 5 +- sandboxd/pool/upstream_test.go | 9 +- sandboxd/pool/view.go | 42 ++++++++ sandboxd/types/env_test.go | 75 +++++++++++++ sandboxd/types/types.go | 146 +++++++++++++++++++++++-- sdk/go/options.go | 11 +- 22 files changed, 773 insertions(+), 55 deletions(-) diff --git a/docs/egress.md b/docs/egress.md index f5a36965..d371f1ee 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -285,8 +285,46 @@ gateway key per user, while the guest holds a placeholder. - **Audit.** An injection is recorded as the secret's name; the value is never logged, listed or returned. -The proxy reads the claim's env under the node's manager lock, once per -request whose matched rule names a secret; other requests pay nothing. +The proxy reads a snapshot of the claim's env that each env write publishes, +without a lock, once per request whose matched rule names a secret or whose +host has a claim credential; other requests pay nothing. + +### Claim credentials + +A `guest: false` env entry can also name where its value goes, so a control +plane binds each sandbox's own credential to a host without editing the node +config: + +```jsonc +{ "env": { + "API_TOKEN": { "value": "Bearer …", "guest": false, + "inject": { "hosts": ["api.example.com", "*.git.example.com"], "header": "Authorization" } } +} } +``` + +On an intercepted request to one of `hosts`, the proxy sets `header` to the +entry's value, overwriting whatever the guest sent, so the guest can hold a +placeholder. The entry rides the claim request or any `/env` write, and +rotation is the same `PATCH`, accepted on a hibernated or archived claim +without waking it. + +- **Envelope.** The operator still decides which hosts can carry a + credential: every host must be covered by an `intercept: true` rule of the + claim's pool and allowed by its tenant class, or the write answers 400 and + nothing is stored. A credential never widens egress and never rides a + plaintext forward request. +- **Precedence.** The pool rule's own `secret` is set first and wins: a + credential naming that header for a host the rule covers is a 400. Among + credentials for the same header, the first by entry name wins. An empty + value injects nothing. +- **Shape.** `hosts` holds 1 to 8 lowercase names or `*.suffix` patterns + (no bare `*`, no port); `header` is any header name except `Host`, + `Content-Length` and the hop-by-hop ones. `GET /env` returns `inject` with + the value blanked. +- **Reload.** A reload that drops the covering intercept rule stops that + injection on the next request and logs which claims lost which hosts. +- **Audit.** An injection is recorded as `claim:` next to the + pool secret's name, comma-separated. ## Upstream proxies diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 49105414..c185c488 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -68,12 +68,16 @@ Auth: `Authorization: Bearer ` (when configured). cold-booting an image named like the template - `env` is the claim's own environment, at most 64 entries: each name a shell identifier of at most 128 bytes, each value at most 8 KiB with no control - characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value` and `guest`; any other + characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other member, or `"guest": null`, answers 400, so a mistyped flag never lands a host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a `guest: false` entry never enters it and only feeds the node's egress - [secrets](egress.md#claim-env-and-secrets). A bad entry answers 400 before - any VM is touched; see [claim env](#getputpatch-v1sandboxesidenv) + [secrets](egress.md#claim-env-and-secrets), or, with `inject` + (`{"hosts": [...], "header": "..."}`), becomes a + [claim credential](egress.md#claim-credentials) the proxy sends to those + hosts. A bad entry answers 400, and an `inject` host the claim's egress does + not intercept answers 400 with the claim's VM released; see + [claim env](#getputpatch-v1sandboxesidenv) - `egress` `false` claims with no [egress policy](egress.md), whatever the pool's (or, for a promoted template, its source pool's): the guest's doors are never bound. Omitted or `true` keeps the policy the claim would get @@ -725,6 +729,11 @@ values: {"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}} ``` +An entry with `inject` comes back with its hosts and header, value blanked. +An `inject` host the claim's pool does not intercept, or its tenant class does +not allow, and a header the pool rule's own secret already sets on that host, +answer 400 and store nothing. + The env is persisted with the claim. A `guest: false` change takes effect on the egress proxy's next request at any time, a hibernated or archived claim included, and never touches the guest. A change to the guest entries is @@ -738,8 +747,8 @@ result does not rewrite the claims journal. A 500 after the store step means the env is stored and its `guest: false` values are already live, but the guest write failed; resend the same request to deliver it. The guest side is described in [silkd](silkd.md#claim-env). -`PUT`/`PATCH`/`DELETE` answer 204, `GET` 200; 400 a bad entry or an unknown body -field; 404 unknown id or another tenant's claim; 409 a guest change on a +`PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body +field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a paused or failed sandbox. ## PUT /v1/sandboxes/{id}/instance-metadata diff --git a/sandboxd/egress/intercept_test.go b/sandboxd/egress/intercept_test.go index f4449c61..791235b3 100644 --- a/sandboxd/egress/intercept_test.go +++ b/sandboxd/egress/intercept_test.go @@ -246,21 +246,72 @@ func TestInterceptRejectsUntrustedUpstream(t *testing.T) { } } +func TestInterceptInjectsClaimCredentialsBelowThePoolSecret(t *testing.T) { + events := make(chan Event, 4) + secrets := credSecrets{fakeSecrets: fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}}, creds: map[string][]Credential{"example.com": { + {Name: "API", Header: "X-Api-Key", Value: "k1"}, + {Name: "AUTH", Header: "authorization", Value: "claim"}, + {Name: "EMPTY", Header: "X-Empty", Value: ""}, + {Name: "LATER", Header: "X-Api-Key", Value: "k2"}, + }}} + p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Secret: "gh", Intercept: true}}}, secrets, events) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + front := httptest.NewServer(p) + defer front.Close() + + tc := connectTLS(t, front.Listener.Addr().String(), "Example.com:443", "example.com", guestRoots) + defer func() { _ = tc.Close() }() + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "https://example.com/x", nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + req.Header.Set("X-Api-Key", "placeholder") + req.Header.Set("X-Empty", "guest") + if err = req.Write(tc); err != nil { + t.Fatalf("write request: %v", err) + } + resp, err := http.ReadResponse(bufio.NewReader(tc), req) + if err != nil { + t.Fatalf("read response: %v", err) + } + defer func() { _ = resp.Body.Close() }() + + if got := resp.Header.Get("X-Key-Seen"); got != "k1" { + t.Errorf("upstream saw X-Api-Key %q, want the first claim credential in order", got) + } + if got := resp.Header.Get("X-Auth-Seen"); got != "Bearer SECRET" { + t.Errorf("upstream saw Authorization %q, want the pool secret kept over the claim's", got) + } + if got := resp.Header.Get("X-Empty-Seen"); got != "guest" { + t.Errorf("upstream saw X-Empty %q, want the guest's value kept for an empty credential", got) + } + _ = recvEvent(t, events) + if ev := recvEvent(t, events); ev.Injected != "gh,claim:API" { + t.Errorf("audit Injected %q, want gh,claim:API", ev.Injected) + } +} + func interceptProxy(t *testing.T, rule Rule, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { return interceptProxyPolicy(t, Policy{Allow: []Rule{rule}}, events) } func interceptProxyPolicy(t *testing.T, policy Policy, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { + t.Helper() + return interceptProxySecrets(t, policy, fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}}, events) +} + +func interceptProxySecrets(t *testing.T, policy Policy, secrets Secrets, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { t.Helper() upstream := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("X-Auth-Seen", r.Header.Get("Authorization")) + w.Header().Set("X-Key-Seen", r.Header.Get("X-Api-Key")) + w.Header().Set("X-Empty-Seen", r.Header.Get("X-Empty")) w.Header().Set("X-Host-Seen", r.Host) _, _ = io.WriteString(w, "hello") })) t.Cleanup(upstream.Close) ca, _ := testCA(t) - secrets := fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}} audit := func(ev Event) { if events != nil { events <- ev diff --git a/sandboxd/egress/policy.go b/sandboxd/egress/policy.go index ff7dd257..980302a8 100644 --- a/sandboxd/egress/policy.go +++ b/sandboxd/egress/policy.go @@ -25,6 +25,16 @@ type Rule struct { Intercept bool `json:"intercept,omitzero"` } +// Covers reports whether the rule matches every host the pattern, exact or "*." suffix, matches. +func (r Rule) Covers(pattern string) bool { + suffix, wild := strings.CutPrefix(pattern, "*") + if !wild { + return r.matchHost(pattern) + } + rule, ruleWild := strings.CutPrefix(strings.ToLower(r.Host), "*") + return ruleWild && strings.HasSuffix(suffix, rule) +} + // matches expects host already lowercased by Eval. func (r Rule) matches(host, method string, port uint16) bool { return r.matchHost(host) && r.matchMethod(method) && r.matchPort(port) @@ -133,6 +143,11 @@ func (p Policy) ServesSocks() bool { return p.Socks5 } +// InterceptsHost reports whether an intercept rule covers the host pattern, so a claim credential for it can be injected. +func (p Policy) InterceptsHost(pattern string) bool { + return slices.ContainsFunc(p.Allow, func(r Rule) bool { return r.Intercept && r.Covers(pattern) }) +} + func (p Policy) admitsTunnel() bool { return slices.ContainsFunc(p.Allow, func(r Rule) bool { return !r.Intercept && r.matchMethod(http.MethodConnect) }) } @@ -143,6 +158,7 @@ type Evaluator interface { EvalHost(host string, port uint16) (Rule, Decision) EvalInner(host, method string, port uint16) (Rule, Decision) ServesSocks() bool + InterceptsHost(pattern string) bool } type composite struct { @@ -191,3 +207,8 @@ func (c composite) EvalInner(host, method string, port uint16) (Rule, Decision) func (c composite) ServesSocks() bool { return c.pool.ServesSocks() } + +// InterceptsHost needs the pool to intercept the pattern and the tenant to allow it. +func (c composite) InterceptsHost(pattern string) bool { + return c.pool.InterceptsHost(pattern) && slices.ContainsFunc(c.tenant.Allow, func(r Rule) bool { return r.Covers(pattern) }) +} diff --git a/sandboxd/egress/policy_test.go b/sandboxd/egress/policy_test.go index 9d6835db..2d6af474 100644 --- a/sandboxd/egress/policy_test.go +++ b/sandboxd/egress/policy_test.go @@ -95,6 +95,34 @@ func TestCompositeEvalInnerIntersectsTenant(t *testing.T) { } } +func TestInterceptsHostCoversPatterns(t *testing.T) { + pool := Policy{Allow: []Rule{ + {Host: "api.example.com", Intercept: true}, + {Host: "*.Corp.Test", Intercept: true}, + {Host: "plain.example.com"}, + }} + for pattern, want := range map[string]bool{ + "api.example.com": true, + "*.example.com": false, + "git.corp.test": true, + "*.corp.test": true, + "*.eu.corp.test": true, + "corp.test": false, + "plain.example.com": false, + } { + if got := pool.InterceptsHost(pattern); got != want { + t.Errorf("pool InterceptsHost(%q) = %v, want %v", pattern, got, want) + } + } + if !(Policy{Allow: []Rule{{Host: "*", Intercept: true}}}).InterceptsHost("*.anything.test") { + t.Error("a bare * intercept rule does not cover a wildcard pattern") + } + composed := Compose(pool, Policy{Allow: []Rule{{Host: "*.corp.test"}}}) + if composed.InterceptsHost("api.example.com") || !composed.InterceptsHost("git.corp.test") { + t.Error("a composed policy must need the pool to intercept and the tenant to allow the pattern") + } +} + func TestWildcardApexIsNotMatched(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "*.example.com"}}} if _, d := policy.Eval("example.com", "GET", 443); d != DecisionDeny { diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index 9ad1ba70..7202d9ae 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -41,9 +41,17 @@ type TransferFunc func(ev Event, sent, received int64) // AuditFunc receives every egress decision the proxy takes. type AuditFunc func(Event) -// Secrets resolves a rule's Secret name to the header it injects. +// Credential is a claim-supplied header the proxy sets on an intercepted request. +type Credential struct { + Name string + Header string + Value string +} + +// Secrets resolves a rule's Secret name to the header it injects, and a host to the claim's own credentials for it. type Secrets interface { Header(name string) (header, value string, ok bool) + Credentials(host string) []Credential } // Holder is held for the life of every request and tunnel the proxy serves. @@ -52,7 +60,7 @@ type Holder interface { Unhold() } -// Event is one audited egress attempt; Injected names the credential and Upstream the proxy host, never their secrets. +// Event is one audited egress attempt; Injected lists the credentials set, comma-separated, and Upstream the proxy host, never their secrets. type Event struct { Method string Host string @@ -307,7 +315,7 @@ func (p *Proxy) relay(w http.ResponseWriter, r *http.Request, ev Event, rule Rul out.RequestURI = "" out.Close = false // the guest's Connection: close is its own; the upstream pool keeps the conn stripHop(out.Header) - ev.Injected = p.inject(rule, out.Header) + ev.Injected = p.inject(rule, out.Header, ev.Host, mitm) var body *countingBody if p.transfer != nil && out.Body != nil && out.Body != http.NoBody { body = &countingBody{ReadCloser: out.Body} @@ -375,17 +383,29 @@ func (p *Proxy) reportTransfer(ev Event, sent, received int64) { } } -// inject sets the rule's secret header, overwriting any guest-supplied value. -func (p *Proxy) inject(rule Rule, h http.Header) string { - if rule.Secret == "" || p.secrets == nil { +// inject sets the rule's secret header, then on an intercepted request each claim credential for host on a header still unset by this hop; guest values are overwritten. +func (p *Proxy) inject(rule Rule, h http.Header, host string, mitm bool) string { + if p.secrets == nil { return "" } - header, value, ok := p.secrets.Header(rule.Secret) - if !ok || value == "" { - return "" + var names, set []string + if rule.Secret != "" { + if header, value, ok := p.secrets.Header(rule.Secret); ok && value != "" { + h.Set(header, value) + names, set = append(names, rule.Secret), append(set, header) + } + } + if !mitm { + return strings.Join(names, ",") + } + for _, c := range p.secrets.Credentials(strings.ToLower(host)) { + if c.Value == "" || slices.ContainsFunc(set, func(s string) bool { return strings.EqualFold(s, c.Header) }) { + continue + } + h.Set(c.Header, c.Value) + names, set = append(names, "claim:"+c.Name), append(set, c.Header) } - h.Set(header, value) - return rule.Secret + return strings.Join(names, ",") } func (p *Proxy) record(ev Event) { diff --git a/sandboxd/egress/proxy_test.go b/sandboxd/egress/proxy_test.go index fc1b9d2c..94b3ddc3 100644 --- a/sandboxd/egress/proxy_test.go +++ b/sandboxd/egress/proxy_test.go @@ -174,6 +174,36 @@ func TestForwardNeverInjectsInterceptSecret(t *testing.T) { } } +func TestForwardNeverInjectsClaimCredentials(t *testing.T) { + var seen string + upstream := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + seen = r.Header.Get("X-Api-Key") + })) + defer upstream.Close() + events := make(chan Event, 1) + secrets := credSecrets{creds: map[string][]Credential{"api.internal": {{Name: "API", Header: "X-Api-Key", Value: "k1"}}}} + p := New(Policy{Allow: []Rule{{Host: "api.internal"}}}, secrets, nil, fixedDial(upstream.Listener.Addr().String()), func(ev Event) { events <- ev }, nil) + front := httptest.NewServer(p) + defer front.Close() + + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + req.Header.Set("X-Api-Key", "placeholder") + resp, err := proxyClient(t, front.URL).Do(req) + if err != nil { + t.Fatalf("proxied GET: %v", err) + } + _ = resp.Body.Close() + if seen != "placeholder" { + t.Errorf("upstream saw X-Api-Key %q, want the guest's value: a claim credential never rides a cleartext forward", seen) + } + if ev := recvEvent(t, events); ev.Injected != "" { + t.Errorf("audit Injected %q, want none", ev.Injected) + } +} + func TestForwardStripsHopHeaders(t *testing.T) { var gotHop, gotConn string upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { @@ -525,6 +555,19 @@ func (f fakeSecrets) Header(name string) (string, string, bool) { return hv[0], hv[1], ok } +func (f fakeSecrets) Credentials(string) []Credential { + return nil +} + +type credSecrets struct { + fakeSecrets + creds map[string][]Credential +} + +func (c credSecrets) Credentials(host string) []Credential { + return c.creds[host] +} + type holdCounter struct{ atomic.Int32 } func (h *holdCounter) Hold() { h.Add(1) } diff --git a/sandboxd/egress/secrets.go b/sandboxd/egress/secrets.go index 07825936..a85ac958 100644 --- a/sandboxd/egress/secrets.go +++ b/sandboxd/egress/secrets.go @@ -28,12 +28,11 @@ func (s SecretSpec) Validate() error { return fmt.Errorf("secret name must not be empty") case s.Value != nil: return fmt.Errorf("secret %q: value is not supported, use value_env", s.Name) - case !httpguts.ValidHeaderFieldName(s.Header): - return fmt.Errorf("secret %q: header %q is not a valid header name", s.Name, s.Header) case s.ValueEnv == "" && !types.EnvNameRe.MatchString(s.Name): return fmt.Errorf("secret %q: without value_env the name is the claim env to read, so it must match %s", s.Name, types.EnvNameRe) - case slices.ContainsFunc(nonInjectable, func(h string) bool { return strings.EqualFold(h, s.Header) }): - return fmt.Errorf("secret %q: header %s is not injectable", s.Name, s.Header) + } + if err := CheckInjectHeader(s.Header); err != nil { + return fmt.Errorf("secret %q: %w", s.Name, err) } return nil } @@ -79,3 +78,14 @@ func (s *SecretStore) Header(name string) (header, value string, ok bool) { func (s *SecretStore) EnvName(name string) string { return s.byName[name].env } + +// CheckInjectHeader rejects a header name the proxy cannot set: an invalid one, or one the transport owns or strips. +func CheckInjectHeader(header string) error { + switch { + case !httpguts.ValidHeaderFieldName(header): + return fmt.Errorf("header %q is not a valid header name", header) + case slices.ContainsFunc(nonInjectable, func(h string) bool { return strings.EqualFold(h, header) }): + return fmt.Errorf("header %s is not injectable", header) + } + return nil +} diff --git a/sandboxd/pool/checkpoint.go b/sandboxd/pool/checkpoint.go index 9f0352f1..ff730394 100644 --- a/sandboxd/pool/checkpoint.go +++ b/sandboxd/pool/checkpoint.go @@ -201,7 +201,7 @@ func (m *Manager) publishCheckpoint(ctx context.Context, sb *types.Sandbox, ckID // claimLoaded re-fetches under the record lock, so a delete racing the pre-check cannot slip in. func (m *Manager) claimLoaded(ctx context.Context, ckpt types.Checkpoint, o ClaimOptions) (*types.Sandbox, error) { - if err := m.checkUpstreamEnv(o.Env); err != nil { + if err := m.checkEnv(o.Env); err != nil { return nil, err } l := m.recLock(ckpt.ID) diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 4bff597d..a1b1302b 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -42,7 +42,8 @@ type ClaimOptions struct { // apply stamps the options a claim carries for its life. func (o ClaimOptions) apply(sb *types.Sandbox) { sb.Tenant, sb.ClaimRef, sb.Metadata, sb.OnExpire, sb.NoEgress = o.Tenant, o.ClaimRef, o.Metadata, o.OnExpire.Or(""), o.NoEgress - sb.EgressClass, sb.Env = o.EgressClass, o.Env + sb.EgressClass = o.EgressClass + sb.SetEnv(o.Env) } // ClaimWarm transfers ownership of a warm sandbox without provisioning; ErrNoWarm means empty. @@ -419,7 +420,10 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t } sb.TouchAt(now) } - err := m.deliverEnv(ctx, sbs, inherited) + err := m.checkInjects(m.view.Load(), sbs) + if err == nil { + err = m.deliverEnv(ctx, sbs, inherited) + } m.mu.Lock() if err == nil { err = m.quotaErr(len(sbs), sbs[0].Tenant) @@ -717,7 +721,7 @@ func (m *Manager) admitOptions(ctx context.Context, key types.PoolKey, o ClaimOp if err := archivable(key, len(o.Volumes) > 0, o.OnExpire); err != nil { return nil, nil, err } - if err := m.checkUpstreamEnv(o.Env); err != nil { + if err := m.checkEnv(o.Env); err != nil { return nil, nil, err } volumeSpecs, err := m.resolveVolumes(ctx, o.Tenant, o.Volumes) diff --git a/sandboxd/pool/egress.go b/sandboxd/pool/egress.go index ea816d73..4b20c991 100644 --- a/sandboxd/pool/egress.go +++ b/sandboxd/pool/egress.go @@ -322,13 +322,17 @@ func (m *Manager) disarmEgress(id string, removed bool) { // effectivePolicy resolves pool ∩ tenant; root has no tenant layer, an unpooled key no pool one, a NoEgress claim none at all. func (m *Manager) effectivePolicy(v *configView, sb *types.Sandbox) (egress.Evaluator, bool) { + m.mu.Lock() + defer m.mu.Unlock() + return m.effectivePolicyLocked(v, sb) +} + +func (m *Manager) effectivePolicyLocked(v *configView, sb *types.Sandbox) (egress.Evaluator, bool) { if sb.NoEgress { return nil, false } poolPol := v.poolEgress[sb.PolicyKey()] tenantPol := v.classEgress[sb.EgressClass] - m.mu.Lock() - defer m.mu.Unlock() if poolPol == nil { pooled := sb.Layer == types.LayerPooled if sb.Layer == "" { diff --git a/sandboxd/pool/egress_test.go b/sandboxd/pool/egress_test.go index 050b0865..d1cde37f 100644 --- a/sandboxd/pool/egress_test.go +++ b/sandboxd/pool/egress_test.go @@ -1074,10 +1074,15 @@ func egressClient(path string) *http.Client { } func egressManager(t *testing.T, eng *fakeEngine, pools ...config.PoolSpec) *Manager { + t.Helper() + return egressManagerAt(t, eng, t.TempDir(), pools...) +} + +func egressManagerAt(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { t.Helper() t.Setenv("GH_TOKEN", "s3cr3t") secrets := testSecrets(t, egress.SecretSpec{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}) - cfg := &config.Config{DataDir: t.TempDir(), Bridges: []string{"sbxbr0"}, EgressCA: writeTestEgressCA(t), Pools: pools} + cfg := &config.Config{DataDir: dataDir, Bridges: []string{"sbxbr0"}, EgressCA: writeTestEgressCA(t), Pools: pools} m, err := NewManager(t.Context(), cfg, eng, secrets) if err != nil { t.Fatalf("manager: %v", err) diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index d5ca6148..ffd1e879 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "slices" + "strings" "golang.org/x/sync/errgroup" @@ -59,10 +60,18 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu m.mu.Unlock() return fmt.Errorf("%w: %w", ErrBadEnv, err) } - if err := m.checkUpstreamEnv(env); err != nil { + if err := m.checkEnv(env); err != nil { m.mu.Unlock() return err } + if fresh := env.InjectsChanged(prev); fresh != nil { + v := m.view.Load() + eval, ok := m.effectivePolicyLocked(v, sb) + if gap := v.injectGap(eval, ok, sb, fresh); len(gap) > 0 { + m.mu.Unlock() + return fmt.Errorf("%w: inject %s", ErrBadEnv, strings.Join(gap, ", ")) + } + } changed := !env.SameGuest(prev) paused := !locked || sb.HibernateSnap != "" || sb.PendingSnap != "" || sb.ArchiveCk != "" || failed if changed && paused { @@ -76,14 +85,14 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu m.mu.Unlock() return m.deliverGuestEnv(ctx, verb, id, sock, env, !paused) } - sb.Env = env + sb.SetEnv(env) js := m.store.set(sb) m.mu.Unlock() if err := m.store.commit(js); err != nil { m.mu.Lock() var rb claimSnapshot if m.claimed[id] == sb && sb.Env.Equal(env) { - sb.Env = prev + sb.SetEnv(prev) rb = m.store.set(sb) } m.mu.Unlock() @@ -124,6 +133,33 @@ func (m *Manager) deliverEnv(ctx context.Context, sbs []*types.Sandbox, inherite return group.Wait() } +// checkEnv admits an env this node can serve: its upstream entry, and inject headers the proxy may set. +func (m *Manager) checkEnv(env types.Env) error { + for name, v := range env { + if v.Inject == nil { + continue + } + if err := egress.CheckInjectHeader(v.Inject.Header); err != nil { + return fmt.Errorf("%w: env %s: inject %w", ErrBadEnv, name, err) + } + } + return m.checkUpstreamEnv(env) +} + +// checkInjects refuses a claim batch whose inject hosts a claim's egress does not intercept. +func (m *Manager) checkInjects(v *configView, sbs []*types.Sandbox) error { + for _, sb := range sbs { + if !sb.Env.HasInject() { + continue + } + eval, ok := m.effectivePolicy(v, sb) + if gap := v.injectGap(eval, ok, sb, sb.Env); len(gap) > 0 { + return fmt.Errorf("%w: inject %s", ErrBadEnv, strings.Join(gap, ", ")) + } + } + return nil +} + // heldGuestEnv reports whether sb's guest holds an env file; the caller holds sb.Transition, so no guest write races its capture. func (m *Manager) heldGuestEnv(sb *types.Sandbox) bool { m.mu.Lock() @@ -144,12 +180,16 @@ func (c claimSecrets) Header(name string) (header, value string, ok bool) { if header, value, ok = secrets.Header(name); !ok { return "", "", false } - env := secrets.EnvName(name) - c.m.mu.Lock() - own, set := c.sb.Env.Hidden(env) - c.m.mu.Unlock() - if set { + if own, set := c.sb.HiddenEnv(secrets.EnvName(name)); set { value = own } return header, value, true } + +func (c claimSecrets) Credentials(host string) []egress.Credential { + var out []egress.Credential + for name, v := range c.sb.Injections(host) { + out = append(out, egress.Credential{Name: name, Header: v.Inject.Header, Value: v.Value}) + } + return out +} diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index 8fd0b4a6..9222e4c0 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -450,6 +450,150 @@ func TestPatchEnvMergesAndRepairsTheGuest(t *testing.T) { } } +func TestAnInjectNeedsAHostThePoolIntercepts(t *testing.T) { + pol := &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Secret: "gh", Intercept: true}, {Host: "*.corp.test", Intercept: true}, {Host: "plain.test"}}} + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: pol}) + sb := mustClaim(t, m, testKey) + for name, patch := range map[string]types.EnvPatch{ + "a plain host": injectPatch("X-Api-Key", "plain.test"), + "an unlisted host": injectPatch("X-Api-Key", "other.test"), + "the pool secret's header": injectPatch("authorization", "api.github.com"), + "a transport header": injectPatch("Content-Length", "git.corp.test"), + } { + if err := m.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, ErrBadEnv) { + t.Errorf("%s: %v, want ErrBadEnv", name, err) + } + } + if env, _ := m.Env(sb.ID, ""); env != nil { + t.Errorf("a refused inject left env %v", env) + } + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com", "*.corp.test"), ""); err != nil { + t.Fatalf("an intercepted inject: %v", err) + } + if got := credentialsFor(m, sb, "git.corp.test"); got != "KEY:X-Api-Key=k" { + t.Errorf("credentials for git.corp.test %q, want KEY:X-Api-Key=k", got) + } +} + +func TestAnInjectNeedsTheTenantClassToAllowTheHost(t *testing.T) { + pol := &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: true}, {Host: "*.corp.test", Intercept: true}}} + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: pol}) + editView(m, func(v *configView) { + v.classEgress = map[string]*egress.Policy{"desk": {Allow: []egress.Rule{{Host: "*.corp.test"}}}} + }) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "acme", EgressClass: "desk"}) + if err != nil { + t.Fatalf("claim: %v", err) + } + if err = m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com"), "acme"); !errors.Is(err, ErrBadEnv) { + t.Errorf("a host the class denies: %v, want ErrBadEnv", err) + } + if err = m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "git.corp.test"), "acme"); err != nil { + t.Errorf("a host the pool intercepts and the class allows: %v", err) + } +} + +func TestAClaimWithAnUninterceptedInjectIsRefused(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"other.test"}, Header: "X-Api-Key"}}} + if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}); !errors.Is(err, ErrBadEnv) { + t.Fatalf("claim: %v, want ErrBadEnv", err) + } + m.mu.Lock() + claimed := len(m.claimed) + m.mu.Unlock() + if claimed != 0 || len(eng.removedNames()) != 1 { + t.Errorf("%d claims left and %v removed, want none kept and the VM destroyed", claimed, eng.removedNames()) + } +} + +func TestEachClaimInjectsItsOwnCredentialAndAHostOnlyChangeNeedsNoGuest(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + claim := func(value string) *types.Sandbox { + t.Helper() + env := types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) + if err != nil { + t.Fatalf("claim: %v", err) + } + return sb + } + a, b, plain := claim("a"), claim("b"), mustClaim(t, m, testKey) + if got := credentialsFor(m, a, "api.github.com") + "|" + credentialsFor(m, b, "api.github.com") + "|" + credentialsFor(m, plain, "api.github.com"); got != "KEY:X-Api-Key=a|KEY:X-Api-Key=b|" { + t.Errorf("credentials %q, want each claim's own and none for the plain claim", got) + } + m.mu.Lock() + a.HibernateSnap = "snap" + m.mu.Unlock() + if err := m.PatchEnv(t.Context(), a.ID, types.EnvPatch{"KEY": {Value: "a2", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}}, ""); err != nil { + t.Fatalf("rotate on a hibernated claim: %v", err) + } + if got, wrote := eng.guestEnvs[a.VsockSocket]; wrote || credentialsFor(m, a, "api.github.com") != "KEY:X-Api-Key=a2" { + t.Errorf("guest file %q (written %v), credentials %q; want no guest write and the rotated value", got, wrote, credentialsFor(m, a, "api.github.com")) + } + if err := m.SetEnv(t.Context(), b.ID, types.Env{}, ""); err != nil || credentialsFor(m, b, "api.github.com") != "" { + t.Errorf("clear: %v, credentials %q; want none", err, credentialsFor(m, b, "api.github.com")) + } +} + +func TestInjectsSurviveARestartAndAReloadThatDropsTheInterceptStopsThem(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + dataDir := t.TempDir() + pool := config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: true}}}} + m := egressManagerAt(t, eng, dataDir, pool) + env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) + if err != nil { + t.Fatalf("claim: %v", err) + } + m2 := egressManagerAt(t, eng, dataDir, pool) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + if got := credentialsFor(m2, restored, "api.github.com"); got != "KEY:X-Api-Key=k" { + t.Errorf("credentials after a restart %q, want the persisted inject", got) + } + editView(m2, func(v *configView) { + v.poolEgress[testKey] = &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com"}}} + }) + v := m2.view.Load() + eval, ok := m2.effectivePolicy(v, restored) + if gap := v.injectGap(eval, ok, restored, env); len(gap) != 1 { + t.Errorf("gap after the intercept rule went %v, want the inject named", gap) + } +} + +func TestALostInterceptRefusesOnlyAChangedInject(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + sb := mustClaim(t, m, testKey) + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com"), ""); err != nil { + t.Fatalf("inject: %v", err) + } + editView(m, func(v *configView) { + v.poolEgress[testKey] = &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com"}}} + }) + if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"OTHER": {Value: "o", Guest: hostOnly}}, ""); err != nil { + t.Errorf("an unrelated patch after the intercept went: %v", err) + } + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Other", "api.github.com"), ""); !errors.Is(err, ErrBadEnv) { + t.Errorf("a changed inject after the intercept went: %v, want ErrBadEnv", err) + } +} + func BenchmarkClaimSecretsHeader(b *testing.B) { b.Setenv("GH_TOKEN", "node-gh") store, err := egress.NewSecretStore([]egress.SecretSpec{{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}}) @@ -466,14 +610,38 @@ func BenchmarkClaimSecretsHeader(b *testing.B) { for _, arm := range []string{"no-claim-env", "claim-env"} { sb := &types.Sandbox{} if arm == "claim-env" { - sb.Env = types.Env{"GH_TOKEN": {Value: "Bearer claim", Guest: hostOnly}, "MODE": {Value: "on"}} + sb.SetEnv(types.Env{"GH_TOKEN": {Value: "Bearer claim", Guest: hostOnly}, "MODE": {Value: "on"}}) } b.Run(arm, func(b *testing.B) { for b.Loop() { _, _, _ = claimSecrets{m: m, sb: sb}.Header("gh") } }) + b.Run(arm+"/credentials", func(b *testing.B) { + for b.Loop() { + _ = claimSecrets{m: m, sb: sb}.Credentials("api.example.com") + } + }) + } + sb := &types.Sandbox{} + sb.SetEnv(types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.example.com"}, Header: "X-Key"}}}) + b.Run("inject/credentials", func(b *testing.B) { + for b.Loop() { + _ = claimSecrets{m: m, sb: sb}.Credentials("api.example.com") + } + }) +} + +func injectPatch(header string, hosts ...string) types.EnvPatch { + return types.EnvPatch{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: header}}} +} + +func credentialsFor(m *Manager, sb *types.Sandbox, host string) string { + var out []string + for _, c := range (claimSecrets{m: m, sb: sb}).Credentials(host) { + out = append(out, c.Name+":"+c.Header+"="+c.Value) } + return strings.Join(out, ",") } func envManager(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index 728894d1..22e826c9 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -39,6 +39,7 @@ func (m *Manager) Reconcile(ctx context.Context) error { var adopted []*types.Sandbox m.mu.Lock() for id, sb := range claims { + sb.SetEnv(sb.Env) rec, ok := live[sb.VMName] switch { case sb.ArchiveCk != "": diff --git a/sandboxd/pool/reload.go b/sandboxd/pool/reload.go index 6dad5d1f..c991fe01 100644 --- a/sandboxd/pool/reload.go +++ b/sandboxd/pool/reload.go @@ -43,9 +43,15 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload } var trim []string var doors []*egressListener + var injecting []*types.Sandbox m.mu.Lock() m.view.Store(view) live := slices.Collect(maps.Values(m.egressListeners)) + for _, sb := range m.claimed { + if sb.Env.HasInject() { + injecting = append(injecting, sb) + } + } if !view.guardedEgress { doors = slices.Collect(maps.Values(m.egressPrebound)) clear(m.egressPrebound) @@ -68,10 +74,24 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload m.recordAudit(ctx, "", auditFrame{Op: "config_reload", Changed: changed}) log.WithFunc("pool.ReloadConfig").Infof(ctx, "config reloaded: %s; %d warm VMs of retired goldens destroyed", strings.Join(changed, ", "), len(trim)) m.warnMissingClasses(ctx) + m.warnLostInjects(ctx, view, injecting) m.kickRefill() return res, nil } +// warnLostInjects names each claim whose inject hosts the reloaded egress no longer intercepts; the proxy stops sending them. +func (m *Manager) warnLostInjects(ctx context.Context, v *configView, sbs []*types.Sandbox) { + for _, sb := range sbs { + m.mu.Lock() + env := sb.Env + m.mu.Unlock() + eval, ok := m.effectivePolicy(v, sb) + if gap := v.injectGap(eval, ok, sb, env); len(gap) > 0 { + log.WithFunc("pool.warnLostInjects").Warnf(ctx, "claim %s: inject no longer intercepted: %s", sb.ID, strings.Join(gap, ", ")) + } + } +} + // refuseInterceptOn turns intercept on only for a key whose guests can trust the CA: one the node has not served, with the CA loaded at boot. func (m *Manager) refuseInterceptOn(old, next *configView) error { for _, key := range slices.SortedFunc(maps.Keys(next.poolEgress), func(a, b types.PoolKey) int { return strings.Compare(a.Hash(), b.Hash()) }) { diff --git a/sandboxd/pool/upstream.go b/sandboxd/pool/upstream.go index 28f31b2b..93576e32 100644 --- a/sandboxd/pool/upstream.go +++ b/sandboxd/pool/upstream.go @@ -28,10 +28,7 @@ func (r claimRouter) Route() string { if view.upstreamEnv == "" { return "" } - r.m.mu.Lock() - v, set := r.sb.Env.Hidden(view.upstreamEnv) - r.m.mu.Unlock() - if set { + if v, set := r.sb.HiddenEnv(view.upstreamEnv); set { if v == upstreamDirect { return "" } diff --git a/sandboxd/pool/upstream_test.go b/sandboxd/pool/upstream_test.go index b07e2c58..4e8ad862 100644 --- a/sandboxd/pool/upstream_test.go +++ b/sandboxd/pool/upstream_test.go @@ -35,7 +35,8 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { {"claim value", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "socks5://127.0.0.1:3003", Guest: hostOnly}}, "socks5://127.0.0.1:3003"}, {"claim direct", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "direct", Guest: hostOnly}}, ""}, } { - sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tc.class, Env: tc.env} + sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tc.class} + sb.SetEnv(tc.env) if got := (claimRouter{m: m, sb: sb}).Route(); got != tc.want { t.Errorf("%s: route %q, want %q", tc.name, got, tc.want) } @@ -51,7 +52,7 @@ func TestClaimsOnOnePoolLeaveThroughTheirOwnUpstreams(t *testing.T) { claim := func(upstream string) *types.Sandbox { sb := &types.Sandbox{ID: "sb_" + upstream, Key: testKey} if upstream != "" { - sb.Env = types.Env{"EGRESS_UPSTREAM": {Value: "http://user:pw@" + upstream, Guest: hostOnly}} + sb.SetEnv(types.Env{"EGRESS_UPSTREAM": {Value: "http://user:pw@" + upstream, Guest: hostOnly}}) } return sb } @@ -82,7 +83,9 @@ func TestATenantClaimUnderAClassDialsItsOwnUpstreamLikeARootClaim(t *testing.T) own, classDefault := tunnelProxy(t), tunnelProxy(t) editView(m, func(v *configView) { v.classUpstream["desk"] = "http://" + classDefault.addr }) tenantClaim := func(upstream string) *types.Sandbox { - return &types.Sandbox{ID: "sb_desk", Key: testKey, Tenant: "acme", EgressClass: "desk", Env: types.Env{"EGRESS_UPSTREAM": {Value: upstream, Guest: hostOnly}}} + sb := &types.Sandbox{ID: "sb_desk", Key: testKey, Tenant: "acme", EgressClass: "desk"} + sb.SetEnv(types.Env{"EGRESS_UPSTREAM": {Value: upstream, Guest: hostOnly}}) + return sb } r := claimRouter{m: m, sb: tenantClaim("http://user:pw@" + own.addr)} conn, err := r.Dial(t.Context(), r.Route(), "tcp", origin) diff --git a/sandboxd/pool/view.go b/sandboxd/pool/view.go index 2fd54cfc..2d18a1b3 100644 --- a/sandboxd/pool/view.go +++ b/sandboxd/pool/view.go @@ -1,7 +1,11 @@ package pool import ( + "fmt" + "maps" "os" + "slices" + "strings" "sync/atomic" "github.com/cocoonstack/sandbox/sandboxd/config" @@ -73,6 +77,40 @@ func newConfigView(cfg *config.Config, secrets *egress.SecretStore) *configView return v } +// injectGap names, in env order, each inject host sb's egress does not intercept and each header a covering pool secret already sets. +func (v *configView) injectGap(eval egress.Evaluator, ok bool, sb *types.Sandbox, env types.Env) []string { + var gap []string + pool := v.poolEgress[sb.PolicyKey()] + for _, name := range slices.Sorted(maps.Keys(env)) { + inj := env[name].Inject + if inj == nil { + continue + } + for _, h := range inj.Hosts { + if !ok || !eval.InterceptsHost(h) { + gap = append(gap, fmt.Sprintf("%s: %s is not intercepted", name, h)) + } else if v.secretSets(pool, h, inj.Header) { + gap = append(gap, fmt.Sprintf("%s: the pool's secret already sets %s on %s", name, inj.Header, h)) + } + } + } + return gap +} + +// secretSets reports whether a pool intercept rule covering host injects a secret into header. +func (v *configView) secretSets(pool *egress.Policy, host, header string) bool { + if pool == nil { + return false + } + return slices.ContainsFunc(pool.Allow, func(r egress.Rule) bool { + if !r.Intercept || r.Secret == "" || !r.Covers(host) { + return false + } + h, _, known := v.secrets.Header(r.Secret) + return known && strings.EqualFold(h, header) + }) +} + type resolvedPolicy struct { view *configView eval egress.Evaluator @@ -109,6 +147,10 @@ func (l *livePolicy) ServesSocks() bool { return l.current().ServesSocks() } +func (l *livePolicy) InterceptsHost(pattern string) bool { + return l.current().InterceptsHost(pattern) +} + // current denies everything once a reload leaves the claim with no policy. func (l *livePolicy) current() egress.Evaluator { v := l.m.view.Load() diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index 9d7ad313..b202f8c4 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -83,6 +83,8 @@ func TestEnvVarDecodesStrictly(t *testing.T) { {`{"value":"s","host_only":true}`, false, false}, {`{"value":"s","guest":null}`, false, false}, {`{"value":"s","guest":"false"}`, false, false}, + {`{"value":"s","guest":false,"inject":{"hosts":["a.test"],"header":"X-Key"}}`, false, true}, + {`{"value":"s","guest":false,"inject":{"hosts":["a.test"],"heder":"X-Key"}}`, false, false}, } { var v EnvVar err := json.Unmarshal([]byte(tt.body), &v) @@ -92,6 +94,79 @@ func TestEnvVarDecodesStrictly(t *testing.T) { } } +func TestEnvInjectValidates(t *testing.T) { + inject := func(header string, hosts ...string) Env { + return Env{"K": {Value: "v", Guest: new(false), Inject: &EnvInject{Hosts: hosts, Header: header}}} + } + tests := []struct { + name string + env Env + want string + }{ + {"host-only exact and wildcard", inject("X-Api-Key", "api.example.com", "*.corp.test", "10.0.0.5"), ""}, + {"guest entry", Env{"K": {Value: "v", Inject: &EnvInject{Hosts: []string{"a.test"}, Header: "X-Key"}}}, "needs guest: false"}, + {"no hosts", inject("X-Key"), "1 to 8 hosts"}, + {"too many hosts", inject("X-Key", "a", "b", "c", "d", "e", "f", "g", "h", "i"), "1 to 8 hosts"}, + {"uppercase host", inject("X-Key", "API.example.com"), "lowercase host name"}, + {"bare wildcard", inject("X-Key", "*"), "lowercase host name"}, + {"host with a port", inject("X-Key", "a.test:443"), "lowercase host name"}, + {"bad header", inject("X Key", "a.test"), "not a valid header name"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := tt.env.Validate() + if (tt.want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), tt.want)) { + t.Errorf("Validate = %v, want %q", err, tt.want) + } + }) + } +} + +func TestEnvInjectMatchesAndCompares(t *testing.T) { + inj := &EnvInject{Hosts: []string{"api.example.com", "*.corp.test"}, Header: "X-Key"} + for host, want := range map[string]bool{"api.example.com": true, "git.corp.test": true, "corp.test": false, "example.com": false} { + if got := inj.Matches(host); got != want { + t.Errorf("Matches(%q) = %v, want %v", host, got, want) + } + } + a := Env{"K": {Value: "v", Guest: new(false), Inject: inj}} + b := Env{"K": {Value: "v", Guest: new(false), Inject: &EnvInject{Hosts: []string{"api.example.com"}, Header: "X-Key"}}} + if a.Equal(b) || !a.Equal(Env{"K": {Value: "v", Guest: new(false), Inject: inj}}) || !a.SameGuest(b) { + t.Error("Equal must see an inject change, and SameGuest must not") + } + if red := a.Redacted(); red["K"].Value != "" || red["K"].Inject != inj { + t.Errorf("Redacted = %+v, want the value dropped and the inject kept", red["K"]) + } +} + +func TestSandboxPublishesItsEnvToTheProxy(t *testing.T) { + sb := &Sandbox{} + if _, ok := sb.HiddenEnv("K"); ok { + t.Fatal("a sandbox without SetEnv served a hidden value") + } + to := func(hosts ...string) *EnvInject { return &EnvInject{Hosts: hosts, Header: "X-Key"} } + sb.SetEnv(Env{ + "B": {Value: "b", Guest: new(false), Inject: to("a.test")}, + "A": {Value: "a", Guest: new(false), Inject: to("*.test")}, + "OTHER": {Value: "o", Guest: new(false), Inject: to("b.example")}, + "PLAIN": {Value: "p", Guest: new(false)}, + }) + var names []string + for name := range sb.Injections("a.test") { + names = append(names, name) + } + if strings.Join(names, ",") != "A,B" { + t.Errorf("Injections(a.test) = %v, want A,B in name order", names) + } + if v, ok := sb.HiddenEnv("PLAIN"); !ok || v != "p" { + t.Errorf("HiddenEnv(PLAIN) = %q %v, want p", v, ok) + } + sb.SetEnv(nil) + if _, ok := sb.HiddenEnv("PLAIN"); ok { + t.Error("a cleared env still serves a hidden value") + } +} + func TestEnvSplitsGuestAndHostOnlyEntries(t *testing.T) { env := Env{"G": {Value: "g"}, "H": {Value: "h", Guest: new(false)}, "T": {Value: "t", Guest: new(true)}} if v, ok := env.Hidden("H"); !ok || v != "h" { diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index fd51cb06..a56b7f70 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -9,6 +9,7 @@ import ( "encoding/json/v2" "errors" "fmt" + "iter" "maps" "path/filepath" "regexp" @@ -69,6 +70,7 @@ const ( maxEnvVars = 64 maxEnvValueBytes = 8 << 10 maxEnvBytes = 64 << 10 + maxInjectHosts = 8 ) var ( @@ -78,6 +80,8 @@ var ( VolumeNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,19}$`) // EnvNameRe is the portable shell variable name. EnvNameRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]{0,127}$`) + // injectHostRe is a lowercase DNS name or IPv4 literal, optionally under a "*." wildcard. + injectHostRe = regexp.MustCompile(`^(\*\.)?([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)*[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$`) // guestFileEscaper covers the characters systemd unescapes inside a double-quoted EnvironmentFile value. guestFileEscaper = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "`", "\\`", `$`, `\$`) @@ -215,10 +219,58 @@ func (md Metadata) encodedSize() int { return size } +// EnvInject sends a host-only entry's value as Header on intercepted requests to Hosts, each exact or "*." suffix. +type EnvInject struct { + Hosts []string `json:"hosts"` + Header string `json:"header"` +} + +// Matches reports whether the lowercased host is one of the inject hosts. +func (i *EnvInject) Matches(host string) bool { + return slices.ContainsFunc(i.Hosts, func(h string) bool { + if suffix, wild := strings.CutPrefix(h, "*"); wild { + return strings.HasSuffix(host, suffix) + } + return host == h + }) +} + +func (i *EnvInject) validate(name string) error { + if len(i.Hosts) == 0 || len(i.Hosts) > maxInjectHosts { + return fmt.Errorf("env %s: inject needs 1 to %d hosts, got %d", name, maxInjectHosts, len(i.Hosts)) + } + if !httpguts.ValidHeaderFieldName(i.Header) { + return fmt.Errorf("env %s: inject header %q is not a valid header name", name, i.Header) + } + for _, h := range i.Hosts { + if len(h) > 253 || !injectHostRe.MatchString(h) { + return fmt.Errorf("env %s: inject host %q must be a lowercase host name or *.suffix", name, h) + } + } + return nil +} + +func (i *EnvInject) size() int { + n := len(i.Header) + for _, h := range i.Hosts { + n += len(h) + } + return n +} + +func (i *EnvInject) equal(other *EnvInject) bool { + if i == nil || other == nil { + return i == other + } + return i.Header == other.Header && slices.Equal(i.Hosts, other.Hosts) +} + // EnvVar is one entry of a claim's environment; a nil Guest means the entry reaches the guest. type EnvVar struct { Value string `json:"value"` Guest *bool `json:"guest,omitempty"` + // Inject, on a host-only entry, makes the egress proxy send Value to the named hosts. + Inject *EnvInject `json:"inject,omitempty"` } // InGuest reports whether the entry is delivered into the guest. @@ -227,13 +279,14 @@ func (v EnvVar) InGuest() bool { return v.Guest == nil || *v.Guest } // UnmarshalJSON rejects unknown members and a null guest, so a mistyped host-only flag never falls back to the guest. func (v *EnvVar) UnmarshalJSON(b []byte) error { var raw struct { - Value string `json:"value"` - Guest jsontext.Value `json:"guest,omitzero"` + Value string `json:"value"` + Guest jsontext.Value `json:"guest,omitzero"` + Inject *EnvInject `json:"inject,omitempty"` } if err := json.Unmarshal(b, &raw, json.RejectUnknownMembers(true)); err != nil { return fmt.Errorf("env entry: %w", err) } - *v = EnvVar{Value: raw.Value} + *v = EnvVar{Value: raw.Value, Inject: raw.Inject} if raw.Guest == nil { return nil } @@ -246,7 +299,7 @@ func (v *EnvVar) UnmarshalJSON(b []byte) error { } func (v EnvVar) same(other EnvVar) bool { - return v.Value == other.Value && v.InGuest() == other.InGuest() + return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Inject.equal(other.Inject) } // Env is a claim's own environment: guest entries reach the guest, the others stay host-side for egress secrets. @@ -269,9 +322,19 @@ func (e Env) Validate() error { if !httpguts.ValidHeaderFieldValue(v.Value) { return fmt.Errorf("env value of %s must not hold control characters other than tab", name) } + if v.Inject == nil { + continue + } + if v.InGuest() { + return fmt.Errorf("env %s: inject needs guest: false, or the guest holds the value", name) + } + if err := v.Inject.validate(name); err != nil { + return err + } + total += v.Inject.size() } if total > maxEnvBytes { - return fmt.Errorf("env names and values must total at most %d bytes, got %d", maxEnvBytes, total) + return fmt.Errorf("env names, values and injects must total at most %d bytes, got %d", maxEnvBytes, total) } return nil } @@ -285,6 +348,30 @@ func (e Env) Hidden(name string) (string, bool) { return v.Value, true } +// HasInject reports whether any entry injects into egress requests. +func (e Env) HasInject() bool { + for _, v := range e { + if v.Inject != nil { + return true + } + } + return false +} + +// InjectsChanged returns the entries of e that inject and that prev lacks or holds differently. +func (e Env) InjectsChanged(prev Env) Env { + var out Env + for name, v := range e { + if old, ok := prev[name]; v.Inject != nil && (!ok || !v.same(old)) { + if out == nil { + out = Env{} + } + out[name] = v + } + } + return out +} + // HasGuest reports whether any entry reaches the guest. func (e Env) HasGuest() bool { for _, v := range e { @@ -310,7 +397,8 @@ func (e Env) Redacted() Env { out := maps.Clone(e) for name, v := range out { if !v.InGuest() { - out[name] = EnvVar{Guest: v.Guest} + v.Value = "" + out[name] = v } } return out @@ -410,6 +498,12 @@ func (k PoolKey) Validate() error { return nil } +// envView is an immutable env with its inject entries named in order. +type envView struct { + env Env + injects []string +} + // Sandbox is the node-local record of one pooled or claimed VM. type Sandbox struct { ID string `json:"id"` @@ -441,7 +535,7 @@ type Sandbox struct { OnExpire ExpireAction `json:"on_expire,omitempty"` // Volumes records the volumes successfully applied to this claim. Volumes []Volume `json:"volumes,omitempty"` - // Env is replaced whole under the manager mutex; host-only values are never served back. + // Env is replaced whole through SetEnv under the manager mutex; host-only values are never served back. Env Env `json:"env,omitempty"` VsockSocket string `json:"vsock_socket,omitempty"` @@ -476,6 +570,8 @@ type Sandbox struct { lastActivity atomic.Int64 // open counts the data-plane connections held right now. open atomic.Int32 + // live is the env as of the last SetEnv, read by the egress proxy without the manager mutex. + live atomic.Pointer[envView] // Transition serializes hibernate/wake; lock it before (never under) the manager mutex. Transition sync.Mutex `json:"-"` @@ -505,6 +601,42 @@ func (s *Sandbox) UnholdIdle() { s.open.Add(-1) } // Busy reports whether a data-plane connection is held. func (s *Sandbox) Busy() bool { return s.open.Load() > 0 } +// SetEnv replaces the env and publishes it to the egress proxy. +func (s *Sandbox) SetEnv(e Env) { + s.Env = e + view := &envView{env: e} + for name, v := range e { + if v.Inject != nil && !v.InGuest() { + view.injects = append(view.injects, name) + } + } + slices.Sort(view.injects) + s.live.Store(view) +} + +// HiddenEnv returns a host-only entry's value as last published. +func (s *Sandbox) HiddenEnv(name string) (string, bool) { + if view := s.live.Load(); view != nil { + return view.env.Hidden(name) + } + return "", false +} + +// Injections yields, in name order, the published host-only entries that inject into host. +func (s *Sandbox) Injections(host string) iter.Seq2[string, EnvVar] { + return func(yield func(string, EnvVar) bool) { + view := s.live.Load() + if view == nil { + return + } + for _, name := range view.injects { + if v := view.env[name]; v.Inject.Matches(host) && !yield(name, v) { + return + } + } + } +} + // PolicyKey is the pool key whose egress policy applies: PolicySource when set, else Key. func (s *Sandbox) PolicyKey() PoolKey { return cmp.Or(s.PolicySource, s.Key) } diff --git a/sdk/go/options.go b/sdk/go/options.go index 4a08b3e1..a2022df9 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -47,10 +47,17 @@ type VolumeInfo struct { Writable bool `json:"writable,omitzero"` } +// EnvInject makes the node's egress proxy send a host-only entry's value as Header on intercepted requests to Hosts, each exact or "*." suffix. +type EnvInject struct { + Hosts []string `json:"hosts"` + Header string `json:"header"` +} + // EnvVar is one entry of a claim's env; a nil Guest delivers it into the guest, false keeps it host-side for the node's egress. type EnvVar struct { - Value string `json:"value"` - Guest *bool `json:"guest,omitempty"` + Value string `json:"value"` + Guest *bool `json:"guest,omitempty"` + Inject *EnvInject `json:"inject,omitempty"` } // Option configures a New claim. From 2433d1ae582126aee257100a85333e2ef6c50665 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:45:41 +0800 Subject: [PATCH 2/6] egress: deliver event streams and unsized bodies as they arrive relay copied the upstream body with io.Copy and never flushed, so an SSE or chunked response, headers included, stayed in the response buffer until it filled or the origin ended the stream. Like httputil.ReverseProxy, a text/event-stream or unsized response is now flushed after each write, headers first, on both the forward and the intercepted path. --- docs/egress.md | 4 ++- sandboxd/egress/intercept_test.go | 23 +++++++++++++++ sandboxd/egress/proxy.go | 47 +++++++++++++++++++++++++++---- sandboxd/egress/proxy_test.go | 30 ++++++++++++++++++++ 4 files changed, 98 insertions(+), 6 deletions(-) diff --git a/docs/egress.md b/docs/egress.md index d371f1ee..249bb4f5 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -379,7 +379,9 @@ A rule with `intercept: true` makes the proxy terminate that host's TLS with a leaf it signs, so it sees the request's method and path and can inject the secret into HTTPS — the same guarantees plaintext already has. Upstream is re-originated and verified against the host's real root store; the proxy never -trusts an unverified origin. +trusts an unverified origin. A response that is an event stream or has no length +reaches the guest as each chunk arrives, on intercepted and forward requests +alike. Trust is a two-tier PKI. One **cluster root CA** is the trust anchor: its certificate (public) is baked into every interception guest's store, so a leaf diff --git a/sandboxd/egress/intercept_test.go b/sandboxd/egress/intercept_test.go index 791235b3..ba44a427 100644 --- a/sandboxd/egress/intercept_test.go +++ b/sandboxd/egress/intercept_test.go @@ -291,6 +291,29 @@ func TestInterceptInjectsClaimCredentialsBelowThePoolSecret(t *testing.T) { } } +func TestInterceptStreamsAnEventStreamAsItArrives(t *testing.T) { + release := make(chan struct{}) + upstream := httptest.NewTLSServer(sseHandler(release)) + defer upstream.Close() + defer close(release) + ca, _ := testCA(t) + p := New(Policy{Allow: []Rule{{Host: "example.com", Intercept: true}}}, nil, ca, fixedDial(upstream.Listener.Addr().String()), nil, nil) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + guestRoots := x509.NewCertPool() + guestRoots.AppendCertsFromPEM(ca.CertPEM()) + front := httptest.NewServer(p) + defer front.Close() + + tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) + defer func() { _ = tc.Close() }() + _ = tc.SetReadDeadline(time.Now().Add(2 * time.Second)) + resp := roundTripTLS(t, tc, http.MethodGet) + defer func() { _ = resp.Body.Close() }() + if line, err := bufio.NewReader(resp.Body).ReadString('\n'); err != nil || line != "data: one\n" { + t.Errorf("first event %q, %v; want it before the origin ends the stream", line, err) + } +} + func interceptProxy(t *testing.T, rule Rule, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { return interceptProxyPolicy(t, Policy{Allow: []Rule{rule}}, events) } diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index 7202d9ae..f9839c67 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -29,11 +29,15 @@ const ( maxIdleConns = 64 ) -// hopHeaders are hop-by-hop and proxy-scoped headers this hop owns, never passed on. -var hopHeaders = []string{ - "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization", - "Proxy-Connection", "TE", "Trailer", "Transfer-Encoding", "Upgrade", -} +var ( + // hopHeaders are hop-by-hop and proxy-scoped headers this hop owns, never passed on. + hopHeaders = []string{ + "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization", + "Proxy-Connection", "TE", "Trailer", "Transfer-Encoding", "Upgrade", + } + + copyBufs = sync.Pool{New: func() any { b := make([]byte, 32<<10); return &b }} +) // TransferFunc receives the payload bytes an allowed tunnel or request moved, once it ends. type TransferFunc func(ev Event, sent, received int64) @@ -343,6 +347,10 @@ func (p *Proxy) relay(w http.ResponseWriter, r *http.Request, ev Event, rule Rul maps.Copy(w.Header(), resp.Header) stripHop(w.Header()) w.WriteHeader(resp.StatusCode) + if streamed(resp) { + received = copyFlushing(w, resp.Body) + return + } received, _ = io.Copy(w, resp.Body) } @@ -491,6 +499,35 @@ func spliceable(c net.Conn) net.Conn { return c } +func streamed(resp *http.Response) bool { + mediaType, _, _ := strings.Cut(resp.Header.Get("Content-Type"), ";") + return resp.ContentLength < 0 || strings.EqualFold(strings.TrimSpace(mediaType), "text/event-stream") +} + +func copyFlushing(w http.ResponseWriter, body io.Reader) int64 { + rc := http.NewResponseController(w) + if rc.Flush() != nil { + n, _ := io.Copy(w, body) + return n + } + buf := copyBufs.Get().(*[]byte) + defer copyBufs.Put(buf) + var total int64 + for { + n, err := body.Read(*buf) + if n > 0 { + written, werr := w.Write((*buf)[:n]) + total += int64(written) + if werr != nil || rc.Flush() != nil { + return total + } + } + if err != nil { + return total + } + } +} + func denied(w http.ResponseWriter, host string) { http.Error(w, fmt.Sprintf("egress denied: %s", host), http.StatusForbidden) } diff --git a/sandboxd/egress/proxy_test.go b/sandboxd/egress/proxy_test.go index 94b3ddc3..8b8b56fe 100644 --- a/sandboxd/egress/proxy_test.go +++ b/sandboxd/egress/proxy_test.go @@ -204,6 +204,27 @@ func TestForwardNeverInjectsClaimCredentials(t *testing.T) { } } +func TestForwardStreamsAnEventStreamAsItArrives(t *testing.T) { + release := make(chan struct{}) + upstream := httptest.NewServer(sseHandler(release)) + defer upstream.Close() + defer close(release) + p := New(Policy{Allow: []Rule{{Host: "sse.internal"}}}, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) + front := httptest.NewServer(p) + defer front.Close() + + client := proxyClient(t, front.URL) + client.Timeout = 2 * time.Second + resp, err := client.Get("http://sse.internal/") + if err != nil { + t.Fatalf("headers did not arrive before the origin ended the stream: %v", err) + } + defer func() { _ = resp.Body.Close() }() + if line, err := bufio.NewReader(resp.Body).ReadString('\n'); err != nil || line != "data: one\n" { + t.Errorf("first event %q, %v; want it before the origin ends the stream", line, err) + } +} + func TestForwardStripsHopHeaders(t *testing.T) { var gotHop, gotConn string upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { @@ -574,6 +595,15 @@ func (h *holdCounter) Hold() { h.Add(1) } func (h *holdCounter) Unhold() { h.Add(-1) } +func sseHandler(release <-chan struct{}) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: one\n\n") + w.(http.Flusher).Flush() + <-release + }) +} + func fixedDial(target string) DialFunc { return func(ctx context.Context, network, _ string) (net.Conn, error) { var d net.Dialer From ad36f63ad40e390f0386f206bedfbb8ec4ed0ee5 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:46:23 +0800 Subject: [PATCH 3/6] egress: let a guest resume intercepted TLS sessions serveIntercept built a fresh tls.Config per CONNECT, each with its own auto ticket keys, so no guest session ever resumed and every new connection paid a full handshake. One ticket key per proxy lets a caching client resume: connect plus one request went from 1.195 ms to 0.962 ms in an interleaved A/B (6 runs each, order swapped); clients without a session cache are unchanged. --- docs/egress.md | 4 +++- sandboxd/egress/bench_test.go | 22 ++++++++++++++++++++++ sandboxd/egress/intercept.go | 1 + sandboxd/egress/intercept_test.go | 25 ++++++++++++++++++++++++- sandboxd/egress/proxy.go | 5 +++++ 5 files changed, 55 insertions(+), 2 deletions(-) diff --git a/docs/egress.md b/docs/egress.md index 249bb4f5..c1f47ebb 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -379,7 +379,9 @@ A rule with `intercept: true` makes the proxy terminate that host's TLS with a leaf it signs, so it sees the request's method and path and can inject the secret into HTTPS — the same guarantees plaintext already has. Upstream is re-originated and verified against the host's real root store; the proxy never -trusts an unverified origin. A response that is an event stream or has no length +trusts an unverified origin. A guest that caches TLS sessions resumes them on +its next CONNECT through the same sandbox, which skips the leaf signature and +chain check. A response that is an event stream or has no length reaches the guest as each chunk arrives, on intercepted and forward requests alike. diff --git a/sandboxd/egress/bench_test.go b/sandboxd/egress/bench_test.go index b43daa11..a1c9d45f 100644 --- a/sandboxd/egress/bench_test.go +++ b/sandboxd/egress/bench_test.go @@ -47,6 +47,28 @@ func BenchmarkInterceptedHandshake(b *testing.B) { } } +func BenchmarkInterceptedConnection(b *testing.B) { + for _, arm := range []string{"fresh", "session-cache"} { + b.Run(arm, func(b *testing.B) { + addr, roots := benchFront(b, true) + var cache tls.ClientSessionCache + if arm == "session-cache" { + cache = tls.NewLRUClientSessionCache(8) + } + resumed := 0 + for b.Loop() { + tc := connectTLSCached(b, addr, "example.com:443", "example.com", roots, cache) + benchRoundTrip(b, tc, bufio.NewReader(tc)) + if tc.ConnectionState().DidResume { + resumed++ + } + _ = tc.Close() + } + b.ReportMetric(float64(resumed)/float64(b.N), "resumed/op") + }) + } +} + func BenchmarkSplicedHandshake(b *testing.B) { addr, roots := benchFront(b, false) for b.Loop() { diff --git a/sandboxd/egress/intercept.go b/sandboxd/egress/intercept.go index b05c53e0..8b09fc3d 100644 --- a/sandboxd/egress/intercept.go +++ b/sandboxd/egress/intercept.go @@ -32,6 +32,7 @@ func (p *Proxy) serveIntercept(w http.ResponseWriter, r *http.Request, host stri NextProtos: []string{"http/1.1"}, Certificates: []tls.Certificate{*leaf}, } + cfg.SetSessionTicketKeys(p.tickets) client, brw, err := http.NewResponseController(w).Hijack() if err != nil { http.Error(w, "egress: connection cannot be hijacked", http.StatusInternalServerError) diff --git a/sandboxd/egress/intercept_test.go b/sandboxd/egress/intercept_test.go index ba44a427..8acd09fd 100644 --- a/sandboxd/egress/intercept_test.go +++ b/sandboxd/egress/intercept_test.go @@ -314,6 +314,24 @@ func TestInterceptStreamsAnEventStreamAsItArrives(t *testing.T) { } } +func TestInterceptResumesTheGuestsTLSSession(t *testing.T) { + p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Intercept: true}, nil) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + front := httptest.NewServer(p) + defer front.Close() + cache := tls.NewLRUClientSessionCache(4) + for i := range 2 { + tc := connectTLSCached(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots, cache) + resp := roundTripTLS(t, tc, http.MethodGet) + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + if resumed := tc.ConnectionState().DidResume; resumed != (i == 1) { + t.Errorf("connection %d resumed=%v, want only the second to resume", i, resumed) + } + _ = tc.Close() + } +} + func interceptProxy(t *testing.T, rule Rule, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { return interceptProxyPolicy(t, Policy{Allow: []Rule{rule}}, events) } @@ -356,6 +374,11 @@ func trustUpstream(upstream *httptest.Server) *x509.CertPool { } func connectTLS(tb testing.TB, proxyAddr, target, serverName string, roots *x509.CertPool) *tls.Conn { + tb.Helper() + return connectTLSCached(tb, proxyAddr, target, serverName, roots, nil) +} + +func connectTLSCached(tb testing.TB, proxyAddr, target, serverName string, roots *x509.CertPool, cache tls.ClientSessionCache) *tls.Conn { tb.Helper() conn, err := net.Dial("tcp", proxyAddr) if err != nil { @@ -367,7 +390,7 @@ func connectTLS(tb testing.TB, proxyAddr, target, serverName string, roots *x509 if status := readPreamble(tb, conn); !strings.Contains(status, "200") { tb.Fatalf("CONNECT status = %q, want 200", status) } - tc := tls.Client(conn, &tls.Config{ServerName: serverName, RootCAs: roots}) + tc := tls.Client(conn, &tls.Config{ServerName: serverName, RootCAs: roots, ClientSessionCache: cache}) if err := tc.Handshake(); err != nil { tb.Fatalf("guest tls handshake: %v", err) } diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index f9839c67..fc566c50 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -4,6 +4,7 @@ import ( "bufio" "cmp" "context" + "crypto/rand" "crypto/tls" "fmt" "io" @@ -111,6 +112,8 @@ type Proxy struct { // leaves caches this sandbox's interception leaves; per-proxy, never shared. leafMu sync.Mutex leaves map[string]*tls.Certificate + // tickets lets a guest resume an intercepted TLS session on its next CONNECT. + tickets [][32]byte // conns tracks both halves of every tunnel and SOCKS5 connection, which http.Server.Close does not reach. connMu sync.Mutex @@ -138,6 +141,8 @@ func New(policy Evaluator, secrets Secrets, ca *CA, router Router, audit AuditFu pools.mitm = pools.tr.Clone() pools.mitm.TLSClientConfig = &tls.Config{MinVersion: tls.VersionTLS12} p.leaves = map[string]*tls.Certificate{} + p.tickets = make([][32]byte, 1) + _, _ = rand.Read(p.tickets[0][:]) } p.pools.Store(pools) return p From c159b747822c6588c5eef0a540f39832e4faefd3 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:46:45 +0800 Subject: [PATCH 4/6] config: the cluster digest covers the egress layer Egress classes, pool egress, secret names, headers and value envs, internal allow and the upstream allow list must match across nodes, or a redirected claim reaches different hosts and its credentials are refused or read from another env there; the digest left them out, so drift was silent. They now feed the digest in an order-free form without secret values, and the manager computes it from the config as last reloaded and republishes it after every reload. --- docs/cluster.md | 6 +++-- sandboxd/config/config.go | 37 ++++++++++++++++++++++++--- sandboxd/config/config_test.go | 46 +++++++++++++++++++++++++++++++--- sandboxd/main.go | 14 +++++++---- sandboxd/pool/pool.go | 8 ++++++ sandboxd/pool/reload_test.go | 13 ++++++++++ 6 files changed, 111 insertions(+), 13 deletions(-) diff --git a/docs/cluster.md b/docs/cluster.md index bb86f8e0..7c95dcf3 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -331,13 +331,15 @@ Some config must match on every node or the cluster fails in confusing ways: | `preview_secret` | a preview URL signed on one node fails verification on another | | `mesh.cluster_key` | nodes cannot join / decrypt gossip at all | | `egress_ca` cluster root | a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception | +| egress: `egress_classes`, each pool's `egress` and `egress_upstream_env`, `secrets` names, headers and `value_env`, `egress_internal_allow`, `egress_upstream` | a claim or tenant redirected to a divergent node reaches different hosts, and its [claim credentials](egress.md#claim-credentials) are refused or not injected there | | the engine root path (cocoon's `root_dir`) | a checkpoint or template export pins the base image blobs' absolute path under the root that captured it; a node whose root sits at another path refuses to clone a record it healed or pulled (`untrusted storage path in snapshot metadata`) — the heal succeeds, the branch fails. Not part of the gossiped digest below | Each node gossips a digest of these over its live tenant set and quotas (HMAC-keyed by `cluster_key` when set, covering each tenant token's SHA-256; otherwise a token-free digest of tenant names and quotas + the CA root + `checkpoint_ttl_hours`, so -nothing brute-forceable rides cleartext gossip), and republishes it after every -tenant change. A mismatch logs a warning at the moment +nothing brute-forceable rides cleartext gossip) plus the egress config above in +an order-free form, never a secret value, and republishes it after every tenant +change and every config reload. A mismatch logs a warning at the moment the divergent node appears — not at the first unlucky redirect — and raises the `sandboxd_config_digest_mismatch` gauge. It is warn-only: a rolling credential rotation is a legitimate transient mismatch, so a divergence never partitions diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index 61a103e9..31d053f9 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -272,13 +272,14 @@ func (c *Config) HasEgress() bool { return len(c.Bridges) > 0 || len(c.Networks) > 0 } -// ClusterDigest fingerprints the must-match config and live tenant records; without cluster_key it omits token hashes. +// ClusterDigest fingerprints the must-match config, its egress layer and the live tenant records; without cluster_key it omits token hashes. func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) string { tenants = slices.Clone(tenants) slices.SortFunc(tenants, func(a, b TenantRecord) int { return strings.Compare(a.Name, b.Name) }) + eg := c.egressFingerprint() if c.Mesh != nil { if key, _ := c.Mesh.DecodedKey(); key != nil { - raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours}) + raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours, eg}) mac := hmac.New(sha256.New, key) mac.Write(raw) return hex.EncodeToString(mac.Sum(nil)) @@ -287,7 +288,37 @@ func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) str for i := range tenants { tenants[i].TokenSHA256 = "" } - return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours}) + return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours, eg}) +} + +type egressFingerprint struct { + Classes []EgressClass `json:"classes,omitempty"` + Secrets []egress.SecretSpec `json:"secrets,omitempty"` + Internal []string `json:"internal,omitempty"` + Upstream *EgressUpstreamConfig `json:"upstream,omitempty"` + Pools map[string]PoolSpec `json:"pools,omitempty"` +} + +// egressFingerprint is the egress config every node must share, in an order-free form; secret values stay out. +func (c *Config) egressFingerprint() egressFingerprint { + fp := egressFingerprint{ + Classes: slices.SortedFunc(slices.Values(c.EgressClasses), func(a, b EgressClass) int { return strings.Compare(a.Name, b.Name) }), + Internal: slices.Sorted(slices.Values(c.EgressInternalAllow)), + Pools: map[string]PoolSpec{}, + } + for _, s := range c.Secrets { + fp.Secrets = append(fp.Secrets, egress.SecretSpec{Name: s.Name, Header: s.Header, ValueEnv: s.ValueEnv}) + } + slices.SortFunc(fp.Secrets, func(a, b egress.SecretSpec) int { return strings.Compare(a.Name, b.Name) }) + if u := c.EgressUpstream; u != nil { + fp.Upstream = &EgressUpstreamConfig{ClaimEnv: u.ClaimEnv, Allow: slices.Sorted(slices.Values(u.Allow))} + } + for _, p := range c.Pools { + if p.Egress != nil || p.EgressUpstreamEnv != "" { + fp.Pools[poolLabel(p.PoolKey)] = PoolSpec{Egress: p.Egress, EgressUpstreamEnv: p.EgressUpstreamEnv} + } + } + return fp } func (c *Config) guardsEgressLane() bool { diff --git a/sandboxd/config/config_test.go b/sandboxd/config/config_test.go index ae34f5f7..54387ee2 100644 --- a/sandboxd/config/config_test.go +++ b/sandboxd/config/config_test.go @@ -9,6 +9,7 @@ import ( "strings" "testing" + "github.com/cocoonstack/sandbox/sandboxd/egress" "github.com/cocoonstack/sandbox/sandboxd/types" ) @@ -73,9 +74,9 @@ func TestClusterDigestBytesArePinned(t *testing.T) { fp string want string }{ - {"keyless", &Config{APIToken: "tk", PreviewSecret: "p&s", CheckpointTTLHours: 24}, tenants, "ca<&>\u2028fp", "889dd395d66341236c2e073bea4b9455e48a48eb7ea79a69f60bda9b914f40f9"}, - {"keyed", &Config{APIToken: "tk&\u2029", PreviewSecret: "p&s<", CheckpointTTLHours: 24, Mesh: &MeshConfig{ClusterKey: key}}, tenants, "ca<&>fp", "1bf87485f6d3a2e097ffa0835127601520725185d06ed960b441888e2538a6c8"}, - {"no tenants", &Config{}, nil, "fp", "38fc616b12f612c2c5c3f83af9d4c6caa771e1d3ffca39a6c835e522d3db49f8"}, + {"keyless", &Config{APIToken: "tk", PreviewSecret: "p&s", CheckpointTTLHours: 24}, tenants, "ca<&>\u2028fp", "ad397b4d6752991094c96a38e2e3a18bc220a5e68b315a625236f5147d9d0924"}, + {"keyed", &Config{APIToken: "tk&\u2029", PreviewSecret: "p&s<", CheckpointTTLHours: 24, Mesh: &MeshConfig{ClusterKey: key}}, tenants, "ca<&>fp", "848ffb9bbcaa14b59074f0753cc2aff33f72494722059f27587d6fe61eb74ad2"}, + {"no tenants", &Config{}, nil, "fp", "4bf6489cb7c229d99dad45b7c8eb8a070a4fd7040cf43c1b49f103ae3107b10b"}, } { t.Run(tt.name, func(t *testing.T) { if got := tt.cfg.ClusterDigest(tt.fp, tenantRecords(tt.tenants)); got != tt.want { @@ -85,6 +86,45 @@ func TestClusterDigestBytesArePinned(t *testing.T) { } } +func TestClusterDigestCoversTheEgressLayerInAnyOrder(t *testing.T) { + key := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} + desk := EgressClass{Name: "desk", Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.example.com"}}}} + ops := EgressClass{Name: "ops", Egress: &egress.Policy{Allow: []egress.Rule{{Host: "ops.example.com"}}}} + base := func() *Config { + return &Config{ + EgressClasses: []EgressClass{desk, ops}, + Secrets: []egress.SecretSpec{{Name: "gh", Header: "Authorization", ValueEnv: "GH_A"}, {Name: "gw", Header: "X-Key"}}, + EgressInternalAllow: []string{"10.0.0.0/8", "172.16.0.0/12"}, + EgressUpstream: &EgressUpstreamConfig{ClaimEnv: "EGRESS_UPSTREAM", Allow: []string{"a.example", "b.example"}}, + Pools: []PoolSpec{{PoolKey: key, Warm: 2, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.example.com", Intercept: true}}}}}, + } + } + d := base().ClusterDigest("fp", nil) + same := base() + same.EgressClasses = []EgressClass{ops, desk} + same.Secrets = []egress.SecretSpec{{Name: "gw", Header: "X-Key"}, {Name: "gh", Header: "Authorization", ValueEnv: "GH_A"}} + same.EgressInternalAllow = []string{"172.16.0.0/12", "10.0.0.0/8"} + same.EgressUpstream.Allow = []string{"b.example", "a.example"} + same.Pools[0].Warm = 9 + if same.ClusterDigest("fp", nil) != d { + t.Error("order or a pool target changed the digest") + } + for name, change := range map[string]func(c *Config){ + "class rule": func(c *Config) { c.EgressClasses[0].Egress = &egress.Policy{} }, + "secret header": func(c *Config) { c.Secrets[0].Header = "X-Other" }, + "secret env": func(c *Config) { c.Secrets[0].ValueEnv = "GH_B" }, + "internal allow": func(c *Config) { c.EgressInternalAllow = nil }, + "upstream allow": func(c *Config) { c.EgressUpstream.Allow = []string{"a.example"} }, + "pool egress": func(c *Config) { c.Pools[0].Egress = &egress.Policy{} }, + } { + c := base() + change(c) + if c.ClusterDigest("fp", nil) == d { + t.Errorf("a %s change left the digest unchanged", name) + } + } +} + func TestLoadAppliesDefaults(t *testing.T) { path := writeConfig(t, `{"pools":[{"template":"rt:24.04","net":"none","size":"small"}]}`) cfg, err := Load(path) diff --git a/sandboxd/main.go b/sandboxd/main.go index d569b138..5ad1513b 100644 --- a/sandboxd/main.go +++ b/sandboxd/main.go @@ -98,6 +98,7 @@ func main() { var placer server.Placer var prober server.CheckpointProber var probeKey []byte + var republish func() if cfg.Mesh != nil { msh, err := startMesh(ctx, cfg, mgr) if err != nil { @@ -108,9 +109,8 @@ func main() { mgr.SetTemplateNotifier(func() { msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes(), mgr.VolumeNames()) }) - mgr.OnTenants(func() { - msh.UpdateDigest(ctx, cfg.ClusterDigest(mgr.EgressCAFingerprint(), mgr.TenantRecords())) - }) + republish = func() { msh.UpdateDigest(ctx, mgr.ClusterDigest()) } + mgr.OnTenants(republish) clusterKey, err := cfg.Mesh.DecodedKey() if err != nil { logger.Fatalf(ctx, err, "decode mesh cluster key") @@ -138,7 +138,11 @@ func main() { if err != nil { return pool.ReloadResult{}, fmt.Errorf("%w: %w", pool.ErrBadConfig, err) } - return mgr.ReloadConfig(ctx, next) + res, err := mgr.ReloadConfig(ctx, next) + if err == nil && republish != nil { + republish() + } + return res, err } go reloadOnHangup(ctx, hangup, reload) srv := server.New(cfg.APIToken, cmp.Or(cfg.ClientAdvertise, cfg.AdvertiseAddr), mgr, eng, placer, prober, probeKey, preview) @@ -199,7 +203,7 @@ func startMesh(ctx context.Context, cfg *config.Config, mgr *pool.Manager) (*mes if err != nil { return nil, err } - self := mesh.NodeState{NodeID: cmp.Or(mc.NodeID, mc.Bind), Addr: cfg.AdvertiseAddr, ClientAddr: cfg.ClientAdvertise, Digest: cfg.ClusterDigest(mgr.EgressCAFingerprint(), mgr.TenantRecords())} + self := mesh.NodeState{NodeID: cmp.Or(mc.NodeID, mc.Bind), Addr: cfg.AdvertiseAddr, ClientAddr: cfg.ClientAdvertise, Digest: mgr.ClusterDigest()} msh, err := mesh.New(ctx, mlCfg, self, key, cfg.DataDir) if err != nil { return nil, err diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 4f68884e..d1f7248c 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -503,6 +503,14 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg return m, nil } +// ClusterDigest fingerprints this node's must-match config as last loaded, with its egress root and tenant set. +func (m *Manager) ClusterDigest() string { + m.reloadMu.Lock() + cfg := m.cfg + m.reloadMu.Unlock() + return cfg.ClusterDigest(m.EgressCAFingerprint(), m.TenantRecords()) +} + // EgressCAFingerprint is the egress root's fingerprint, or "" when the node intercepts nothing. func (m *Manager) EgressCAFingerprint() string { if m.egressCA == nil { diff --git a/sandboxd/pool/reload_test.go b/sandboxd/pool/reload_test.go index a47859d3..0f84c16b 100644 --- a/sandboxd/pool/reload_test.go +++ b/sandboxd/pool/reload_test.go @@ -276,6 +276,19 @@ func TestReloadIsAudited(t *testing.T) { } } +func TestAReloadOfTheEgressLayerChangesTheClusterDigest(t *testing.T) { + m := newTestManager(t, newFakeEngine()) + before := m.ClusterDigest() + next := *m.cfg + next.EgressInternalAllow = []string{"10.8.0.0/16"} + if _, err := m.ReloadConfig(t.Context(), &next); err != nil { + t.Fatalf("reload: %v", err) + } + if m.ClusterDigest() == before { + t.Error("the digest still covers the boot config after an egress reload") + } +} + func TestAnInterruptedReloadStillDestroysTheRetiredWarmVMs(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) From 5090043f438582467a59d50c6ad32c7716892ec5 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:55:18 +0800 Subject: [PATCH 5/6] review: drop comments the names already carry --- sandboxd/egress/policy.go | 1 - sandboxd/egress/proxy.go | 2 +- sandboxd/pool/env.go | 1 - sandboxd/pool/reload.go | 1 - sandboxd/pool/view.go | 1 - 5 files changed, 1 insertion(+), 5 deletions(-) diff --git a/sandboxd/egress/policy.go b/sandboxd/egress/policy.go index 980302a8..81c69489 100644 --- a/sandboxd/egress/policy.go +++ b/sandboxd/egress/policy.go @@ -208,7 +208,6 @@ func (c composite) ServesSocks() bool { return c.pool.ServesSocks() } -// InterceptsHost needs the pool to intercept the pattern and the tenant to allow it. func (c composite) InterceptsHost(pattern string) bool { return c.pool.InterceptsHost(pattern) && slices.ContainsFunc(c.tenant.Allow, func(r Rule) bool { return r.Covers(pattern) }) } diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index fc566c50..c3863c45 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -396,7 +396,7 @@ func (p *Proxy) reportTransfer(ev Event, sent, received int64) { } } -// inject sets the rule's secret header, then on an intercepted request each claim credential for host on a header still unset by this hop; guest values are overwritten. +// inject sets the rule's secret, then on an intercepted request each claim credential whose header is still unset; guest values are overwritten. func (p *Proxy) inject(rule Rule, h http.Header, host string, mitm bool) string { if p.secrets == nil { return "" diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index ffd1e879..d5d39ffd 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -146,7 +146,6 @@ func (m *Manager) checkEnv(env types.Env) error { return m.checkUpstreamEnv(env) } -// checkInjects refuses a claim batch whose inject hosts a claim's egress does not intercept. func (m *Manager) checkInjects(v *configView, sbs []*types.Sandbox) error { for _, sb := range sbs { if !sb.Env.HasInject() { diff --git a/sandboxd/pool/reload.go b/sandboxd/pool/reload.go index c991fe01..5aed63a9 100644 --- a/sandboxd/pool/reload.go +++ b/sandboxd/pool/reload.go @@ -79,7 +79,6 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload return res, nil } -// warnLostInjects names each claim whose inject hosts the reloaded egress no longer intercepts; the proxy stops sending them. func (m *Manager) warnLostInjects(ctx context.Context, v *configView, sbs []*types.Sandbox) { for _, sb := range sbs { m.mu.Lock() diff --git a/sandboxd/pool/view.go b/sandboxd/pool/view.go index 2d18a1b3..9e630200 100644 --- a/sandboxd/pool/view.go +++ b/sandboxd/pool/view.go @@ -97,7 +97,6 @@ func (v *configView) injectGap(eval egress.Evaluator, ok bool, sb *types.Sandbox return gap } -// secretSets reports whether a pool intercept rule covering host injects a secret into header. func (v *configView) secretSets(pool *egress.Policy, host, header string) bool { if pool == nil { return false From dba251a9a41798b0a918275b8a8da8560c5343d9 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 3 Oct 2026 01:58:15 +0800 Subject: [PATCH 6/6] e2e: claim-credential acceptance (credsmoke, credentials-e2e.sh) --- e2e/cmd/credsmoke/main.go | 212 +++++++++++++++++++++++++++++++++++++ scripts/credentials-e2e.sh | 123 +++++++++++++++++++++ 2 files changed, 335 insertions(+) create mode 100644 e2e/cmd/credsmoke/main.go create mode 100755 scripts/credentials-e2e.sh diff --git a/e2e/cmd/credsmoke/main.go b/e2e/cmd/credsmoke/main.go new file mode 100644 index 00000000..eda6fb98 --- /dev/null +++ b/e2e/cmd/credsmoke/main.go @@ -0,0 +1,212 @@ +// credsmoke is the claim-credential acceptance: claims on one intercept pool each +// send their own env-injected header to an HTTPS echo through the proxy, the guest +// never holds the value, and rotation, refusal, clearing and forks behave as the API says. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "net/http" + "os" + "strconv" + "strings" + "time" + + "github.com/cocoonstack/sandbox/e2e/internal/harness" + sandbox "github.com/cocoonstack/sandbox/sdk/go" +) + +const ( + proxy = "http://127.0.0.1:3128" + header = "X-Api-Key" + entry = "API_KEY" +) + +func main() { + addr := flag.String("addr", "127.0.0.1:7882", "sandboxd address") + token := flag.String("token", "", "node api token") + template := flag.String("template", "rt:24.04", "template of the intercept pool") + echo := flag.String("echo", "postman-echo.com", "HTTPS host that echoes request headers at /headers") + secret := flag.String("secret", "", "the pool secret's value the origin should also observe") + reattach := flag.String("reattach", "", "id:token:value of a claim to recheck after a restart") + sse := flag.String("sse", "", "plain HTTP event stream that sends one event, then holds") + flag.Parse() + var err error + if *reattach != "" { + err = recheck(*addr, *token, *echo, *reattach) + } else { + err = run(*addr, *token, *template, *echo, *secret, *sse) + } + if err != nil { + fmt.Fprintln(os.Stderr, "credsmoke:", err) + os.Exit(1) + } +} + +func run(addr, token, template, echo, secret, sse string) error { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + stamp := strconv.FormatInt(time.Now().UnixNano(), 10) + valueA, valueB := "key-a-"+stamp, "key-b-"+stamp + + claim := func(env map[string]sandbox.EnvVar) (*sandbox.Sandbox, error) { + opts := []sandbox.Option{sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.Small)} + if env != nil { + opts = append(opts, sandbox.WithEnv(env)) + } + _, sb, err := harness.Claim(ctx, addr, token, template, opts...) + return sb, err + } + a, err := claim(credential(echo, valueA)) + if err != nil { + return err + } + b, err := claim(credential(echo, valueB)) + if err != nil { + return err + } + defer func() { _ = b.Close() }() + plain, err := claim(nil) + if err != nil { + return err + } + defer func() { _ = plain.Close() }() + fmt.Printf(" claimed %s (A), %s (B), %s (plain) on one intercept pool\n", a.ID, b.ID, plain.ID) + + for _, tc := range []struct { + name string + sb *sandbox.Sandbox + want, not string + }{ + {"A", a, valueA, valueB}, + {"B", b, valueB, valueA}, + {"plain", plain, "placeholder", "key-"}, + } { + if err = expectHeader(ctx, tc.sb, echo, tc.want, tc.not, secret); err != nil { + return fmt.Errorf("%s: %w", tc.name, err) + } + } + fmt.Println(" each claim's origin saw only its own credential, the plain claim the guest's placeholder, all the pool secret") + if sse != "" { + out, _ := plain.Exec(ctx, "sh", "-c", fmt.Sprintf("timeout 4 curl -sSN --noproxy '' -x %s %s 2>&1; true", proxy, sse)) + if !strings.Contains(out, "data: one") { + return fmt.Errorf("the first event did not reach the guest while the stream stayed open: %q", out) + } + fmt.Println(" an open event stream reached the guest event by event through the proxy") + } + + leak, err := a.Exec(ctx, "sh", "-c", fmt.Sprintf("grep -rls %q /proc/[0-9]*/environ /run /etc 2>/dev/null; env | grep -c %q; true", valueA, valueA)) + if err != nil { + return fmt.Errorf("leak probe: %w", err) + } + if strings.TrimSpace(leak) != "0" { + return fmt.Errorf("the credential reached the guest: %q", leak) + } + env, err := a.Env(ctx) + if err != nil { + return fmt.Errorf("read env: %w", err) + } + if v := env[entry]; v.Value != "" || v.Inject == nil || v.Inject.Header != header { + return fmt.Errorf("GET env %+v, want the inject named with the value blanked", v) + } + fmt.Println(" value absent from the guest (env, /proc/*/environ, /run, /etc) and blanked in GET env") + + rotated := valueA + "-r1" + if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil { + return fmt.Errorf("rotate: %w", err) + } + if err = expectHeader(ctx, a, echo, rotated, valueA+"\"", secret); err != nil { + return fmt.Errorf("after a rotation: %w", err) + } + if err = a.Hibernate(ctx); err != nil { + return fmt.Errorf("hibernate: %w", err) + } + rotated += "-r2" + if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil { + return fmt.Errorf("rotate while hibernated: %w", err) + } + if err = expectHeader(ctx, a, echo, rotated, "-r1\"", secret); err != nil { + return fmt.Errorf("after a rotation while hibernated: %w", err) + } + fmt.Println(" rotation applied on the next request, a hibernated claim rotated without waking") + + for name, patch := range map[string]*sandbox.EnvVar{ + "an uncovered host": {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{"example.org"}, Header: header}}, + "the pool secret's header": {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Egress-Token"}}, + "a guest entry": {Value: "x", Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Other"}}, + } { + err = b.PatchEnv(ctx, map[string]*sandbox.EnvVar{"BAD": patch}) + if apiErr, ok := errors.AsType[*sandbox.APIError](err); !ok || apiErr.Status != http.StatusBadRequest { + return fmt.Errorf("inject on %s: %v, want 400", name, err) + } + } + if env, err = b.Env(ctx); err != nil || len(env) != 1 { + return fmt.Errorf("env after refused patches %v %v, want only the original entry", env, err) + } + if err = b.SetEnv(ctx, map[string]sandbox.EnvVar{}); err != nil { + return fmt.Errorf("clear: %w", err) + } + if err = expectHeader(ctx, b, echo, "placeholder", valueB, secret); err != nil { + return fmt.Errorf("after a clear: %w", err) + } + fmt.Println(" uncovered host, pool secret header and guest entry refused with 400; a cleared env stops injecting") + + children, err := a.Fork(ctx, 1, time.Minute) + if err != nil { + return fmt.Errorf("fork: %w", err) + } + defer func() { _ = children[0].Close() }() + if err = expectHeader(ctx, children[0], echo, "placeholder", rotated, secret); err != nil { + return fmt.Errorf("fork child: %w", err) + } + fmt.Println(" a fork child carries no credential") + fmt.Printf("REATTACH %s:%s:%s\n", a.ID, a.Token(), rotated) + fmt.Println("CREDSMOKE PASS") + return nil +} + +func recheck(addr, token, echo, reattach string) error { + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + parts := strings.SplitN(reattach, ":", 3) + if len(parts) != 3 { + return fmt.Errorf("-reattach %q, want id:token:value", reattach) + } + client, err := sandbox.Connect(addr, sandbox.WithAPIToken(token)) + if err != nil { + return err + } + sb, err := client.Lookup(ctx, parts[0], parts[1]) + if err != nil { + return fmt.Errorf("lookup after restart: %w", err) + } + defer func() { _ = sb.Close() }() + if err = expectHeader(ctx, sb, echo, parts[2], "placeholder", ""); err != nil { + return fmt.Errorf("after a restart: %w", err) + } + fmt.Println(" the credential survived a sandboxd restart") + fmt.Println("CREDSMOKE REATTACH PASS") + return nil +} + +func credential(echo, value string) map[string]sandbox.EnvVar { + return map[string]sandbox.EnvVar{entry: {Value: value, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: header}}} +} + +func expectHeader(ctx context.Context, sb *sandbox.Sandbox, echo, want, not, secret string) error { + out, err := sb.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sS -x %s -H '%s: placeholder' https://%s/headers", proxy, header, echo)) + if err != nil { + return fmt.Errorf("proxied HTTPS exec: %w (%s)", err, out) + } + switch { + case !strings.Contains(out, want): + return fmt.Errorf("origin did not see %q:\n%s", want, out) + case not != "" && strings.Contains(out, not): + return fmt.Errorf("origin saw %q:\n%s", not, out) + case secret != "" && !strings.Contains(out, secret): + return fmt.Errorf("origin did not see the pool secret:\n%s", out) + } + return nil +} diff --git a/scripts/credentials-e2e.sh b/scripts/credentials-e2e.sh new file mode 100755 index 00000000..eaf14822 --- /dev/null +++ b/scripts/credentials-e2e.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Bare-metal acceptance for claim credentials: claims on one intercept pool each +# send their own env-injected header to an HTTPS echo, the guest never holds the +# value, an open event stream reaches the guest event by event, and a credential +# survives a sandboxd restart. Runs a dedicated sandboxd on $ADDR. +# Needs outbound to $ECHO and a $TEMPLATE whose silkd binds the egress relay. +set -euo pipefail + +TEMPLATE=${TEMPLATE:-rt:24.04} +ADDR=${ADDR:-127.0.0.1:7882} +TOKEN=${TOKEN:-e2e} +ECHO=${ECHO:-postman-echo.com} +SSE_PORT=${SSE_PORT:-18998} +REPO=$(cd "$(dirname "$0")/.." && pwd) +PROBE=${EGRESS_PROBE_TOKEN:-probe-$(date +%s)} +export EGRESS_PROBE_TOKEN=$PROBE + +DATA=$(mktemp -d /tmp/credentials-e2e.XXXXXX) +DAEMON_PID="" +SSE_PID="" + +cleanup() { + status=$? + if [[ $status -ne 0 && -f "$DATA/daemon.log" ]]; then + echo "== daemon log tail" + tail -30 "$DATA/daemon.log" + fi + for pid in $DAEMON_PID $SSE_PID; do + kill "$pid" 2>/dev/null || true + done + wait 2>/dev/null || true + cocoon vm list --format json 2>/dev/null | + jq -r '.[] | select(.config.name | startswith("sbx-")) | .config.name' | + while read -r vm; do + cocoon vm stop --force "$vm" >/dev/null 2>&1 || true + cocoon vm rm --force "$vm" >/dev/null 2>&1 || true + done || true + rm -rf "$DATA" + exit "$status" +} +trap cleanup EXIT + +echo "== build" +if [[ -n ${SANDBOXD_BIN:-} && -n ${CREDSMOKE_BIN:-} ]]; then + cp "$SANDBOXD_BIN" "$DATA/sandboxd" && cp "$CREDSMOKE_BIN" "$DATA/credsmoke" +else + (cd "$REPO/sandboxd" && GOWORK=off go build -o "$DATA/sandboxd" .) + (cd "$REPO/e2e" && GOWORK=off go build -o "$DATA/credsmoke" ./cmd/credsmoke) +fi + +echo "== provision cluster root + this node's intermediate" +"$DATA/sandboxd" ca init -out "$DATA/ca" -cn "e2e cluster root" >/dev/null +"$DATA/sandboxd" ca issue-intermediate -root-cert "$DATA/ca/root.crt" \ + -root-key "$DATA/ca/root.key" -node e2e-node -out "$DATA/node" >/dev/null + +echo "== event stream origin on 127.0.0.1:$SSE_PORT (one event, then holds)" +python3 -c ' +import http.server, sys, time +class H(http.server.BaseHTTPRequestHandler): + def do_GET(self): + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.end_headers() + self.wfile.write(b"data: one\n\n") + self.wfile.flush() + time.sleep(30) + def log_message(self, *a): + pass +http.server.ThreadingHTTPServer(("127.0.0.1", int(sys.argv[1])), H).serve_forever() +' "$SSE_PORT" & +SSE_PID=$! + +cat >"$DATA/config.json" <>"$DATA/daemon.log" 2>&1 & + DAEMON_PID=$! + for _ in $(seq 1 40); do curl -sf "http://$ADDR/healthz" >/dev/null 2>&1 && break; sleep 0.5; done +} + +echo "== start sandboxd (intercept pool)" +start +for i in $(seq 1 300); do + curl -sf -H "Authorization: Bearer $TOKEN" "http://$ADDR/v1/info" 2>/dev/null | + jq -e '.pools[0].warm >= 1' >/dev/null 2>&1 && break + [[ $i == 300 ]] && { echo "pool never became warm"; exit 1; } + sleep 1 +done + +echo "== credsmoke" +"$DATA/credsmoke" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" -echo "$ECHO" -secret "$PROBE" \ + -sse "http://127.0.0.1:$SSE_PORT/" | tee "$DATA/smoke.log" +reattach=$(awk '/^REATTACH / {print $2}' "$DATA/smoke.log") +value=${reattach##*:} + +echo "== audit: claim credentials named, values absent" +grep -q '"secret":"probe,claim:API_KEY"' "$DATA/state/audit.jsonl" || { echo "audit names no claim:API_KEY"; exit 1; } +if grep -q "$value" "$DATA/state/audit.jsonl" "$DATA/state/usage.jsonl" 2>/dev/null; then + echo "a credential value reached the audit or usage journal" + exit 1 +fi +grep -q "$value" "$DATA/state/claims.json" || { echo "claims.json lost the credential"; exit 1; } + +echo "== restart sandboxd, recheck the surviving claim" +kill "$DAEMON_PID" +wait "$DAEMON_PID" 2>/dev/null || true +start +"$DATA/credsmoke" -addr "$ADDR" -token "$TOKEN" -echo "$ECHO" -reattach "$reattach" +echo "PASS"