Skip to content

sandboxd: per-claim credential injection (#301), streamed egress responses, TLS resumption, egress in the cluster digest - #302

Merged
CMGS merged 6 commits into
mainfrom
feat/claim-credentials
Oct 2, 2026
Merged

CMGS merged 6 commits into
mainfrom
feat/claim-credentials

Conversation

@CMGS

@CMGS CMGS commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Closes #301.

A control plane can now bind each sandbox's own credential to an upstream host per claim. The node config stays the envelope; the claim fills it.

Claim credentials (#301)

A guest: false env entry may carry inject:

{"env": {"API_TOKEN": {"value": "Bearer …", "guest": false,
                       "inject": {"hosts": ["api.example.com", "*.git.example.com"], "header": "Authorization"}}}}

It rides the claim request and the existing /env verbs. There is no new resource: rotation is the same PATCH, removal is null, GET blanks the value and shows the hosts, the entry persists with the claim, forks don't inherit it, and the env limits apply.

Envelope. The proxy sets the header only on intercepted (MITM) requests to those hosts, never on a plaintext forward request. Every host must be covered by an intercept: true rule of the claim's pool and allowed by its tenant class, and a header the pool rule's own secret sets on that host is refused. A violation answers 400 and stores nothing; at claim time the VM is released. Env writes only re-check inject entries that are new or changed, so a later narrowing does not block unrelated env edits.

Runtime. The pool secret is set first and wins. Among claim credentials, the first by entry name wins per header. An empty value injects nothing. A reload that drops the covering intercept stops the injection on the next request and logs the affected claims. The audit records claim:<entry> next to the pool secret's name, never a value.

Hot path.

  • Sandbox.SetEnv publishes an immutable env snapshot that the proxy reads without the manager lock.
  • Before this change, every secret-bearing request and every upstream-routed request took m.mu. Both claimSecrets.Header and claimRouter.Route now read the snapshot.
  • Credentials() for a claim without injects costs 2 ns and 0 allocs; with one inject, 28 ns and 1 alloc. Claim, wake and fork pay nothing extra.

Egress fixes and performance

  • Streamed responses (bug). relay copied upstream bodies with io.Copy and never flushed, so an SSE or chunked response stayed in the buffer, headers included, until it filled or the origin ended. A scratch probe showed headers and the first event arriving only when the origin closed, 2 s later. text/event-stream and unsized bodies are now flushed per write, headers first, on both the forward and the intercepted path, matching httputil.ReverseProxy.
  • TLS session resumption.
    • serveIntercept built a fresh tls.Config per CONNECT. Each had its own auto-generated ticket keys, so no guest session ever resumed: 0 of 1,000 in the benchmark.
    • One ticket key per proxy lets a caching client resume.
    • In an interleaved A/B (6 runs per arm, order swapped), connect plus one request went from 1.195 ms to 0.962 ms. Clients without a session cache are unchanged.
  • What was not changed.
    • The proxy's own code is about 2% of an intercepted request's CPU on the profile. The rest is TLS hops and syscalls.
    • Each egress request appends one usage line under a node-wide journal lock. I left it, since buffering would weaken the billing write.

Cluster digest covers egress

Egress classes, pool egress, secret names, headers and value_env, egress_internal_allow and egress_upstream now feed the gossiped digest, in an order-free form and without values, so drift shows in sandboxd_config_digest_mismatch. The digest is computed from the config as last reloaded and is republished after every reload; before this change it used the boot config. Mixed versions report a mismatch during the rollout, since no backward compatibility is kept.

Numbers

added removed
prod 435 55
tests 534 10
e2e + scripts 335 0
docs 63 10

The prod additions are the new feature: env inject, coverage checks, the snapshot, flush, tickets and the digest. Comment lines are +27/−6, all one-line godocs or WHYs.

Gates

  • Lint: GOWORK=off make go-lint printed 10× 0 issues. on darwin and on linux.
  • asl: clean on both GOOS for all modules.
  • Tests: make go-test passed with the PostgreSQL suite enabled, and go test -race passed in a non-root linux/arm64 container for sandboxd, e2e and sdk/go.
  • Mutation checks: disabling each guard made its test fail. The guards checked were the claim-time coverage check, the secret-header clash, the tenant layer, the stream flush and ticket sharing.
  • Hardware, cocoon-test1 (kit e2e-1001, cocoon 85c93ef, CH v54, FC 1.18.0-dev, rt:24.04-90230c0):
    • All 9 existing legs passed: sandboxd 23.4 s, egress-none 3.9, egress 7.1, socks 13.8, intercept 3.9, archive 41.3, port 4.1, ring 4.7, round 5.2. These match the previous round.
    • The new scripts/credentials-e2e.sh leg passed: each claim's credential reached postman-echo, the plain claim kept the guest's placeholder, and the pool secret was present.
    • In the same leg, the value was absent from the guest's env, /proc/*/environ, /run and /etc, and was blanked in GET /env.
    • Rotation took effect on the next request, including while hibernated.
    • The uncovered host, the clashing header and the guest entry were each refused with 400. A clear stopped the injection, and a fork child carried no credential.
    • An open event stream reached the guest event by event, the audit named the claim credential without its value, and the credential survived a sandboxd restart.

CMGS added 6 commits October 3, 2026 01:45
A guest: false env entry may name hosts and a header; the egress proxy sets its value there on intercepted requests only. Every host must be intercepted by the claim's pool and allowed by its tenant class, and may not take a header the pool rule's own secret sets, or the claim or env write answers 400. The proxy reads a per-sandbox env snapshot published on each env write instead of taking the manager lock per request.
relay copied the upstream body with io.Copy and never flushed, so an SSE or chunked response, headers included, stayed in the response buffer until it filled or the origin ended the stream. Like httputil.ReverseProxy, a text/event-stream or unsized response is now flushed after each write, headers first, on both the forward and the intercepted path.
serveIntercept built a fresh tls.Config per CONNECT, each with its own auto ticket keys, so no guest session ever resumed and every new connection paid a full handshake. One ticket key per proxy lets a caching client resume: connect plus one request went from 1.195 ms to 0.962 ms in an interleaved A/B (6 runs each, order swapped); clients without a session cache are unchanged.
Egress classes, pool egress, secret names, headers and value envs, internal allow and the upstream allow list must match across nodes, or a redirected claim reaches different hosts and its credentials are refused or read from another env there; the digest left them out, so drift was silent. They now feed the digest in an order-free form without secret values, and the manager computes it from the config as last reloaded and republishes it after every reload.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

egress: per-claim credential injection rules, set through the API and never visible to the guest

1 participant