Skip to content

egress: intercept a pool host only for claims that inject on it, and substitute placeholders in the query string #308

Closed

Description

@CMGS

Problem

A control plane that keeps users' static API keys host-side wants each box to call any API the user names, with the key injected at egress (#301, #307). Since #302 the claim chooses the credential, but the node still chooses which hosts are intercepted, and interception applies to every claim of the pool.

At 858a80e:

  • The tunnel decision reads only the policy: EvalHost plus rule.Intercept (sandboxd/egress/proxy.go:275-283). No claim data takes part.
  • Every inject host must be covered by an intercepting pool rule, or the write is 400 (sandboxd/pool/view.go:81-98).
  • Only pool rules in node config may intercept (sandboxd/config/config.go:551-552), and PUT /v1/pools cannot carry egress (sandboxd/pool/setpools.go:25-27).
  • Inside an intercepted tunnel, only HTTP/1.1 is offered (sandboxd/egress/intercept.go:34) and Upgrade is stripped (sandboxd/egress/proxy.go:37).
  • Substitution covers one header, by exact equality (sandboxd/egress/proxy.go:415-421).

So an operator who allows per-claim credentials on arbitrary hosts has two choices:

  • list every host in node config, which puts a control plane's catalog into infra and changes on every new vendor; or
  • intercept *, which TLS-terminates all traffic of every claim in the pool: browsers lose HTTP/2 and WebSocket everywhere, and pinned clients fail on every host.

And a key that a client sends as a query parameter (?key=, ?api_key=) cannot be injected at all.

Proposal

The operator still decides which hosts may be intercepted. The claim's inject entries decide which of them are intercepted for that claim.

{"host": "*", "intercept": "inject"}

Conditional interception.

  • intercept accepts true (today), false, or "inject".
  • With "inject", a CONNECT to a covered host is intercepted only when the claim has at least one inject entry whose hosts match it. Otherwise it is spliced, exactly as a non-intercepting rule.
  • The decision is made once per CONNECT at tunnelDecision (sandboxd/egress/proxy.go:229,275-284), where the proxy's claim secrets are already reachable. A boolean HasCredential(host) avoids the slice alloc of Credentials(host).
  • SOCKS5 keeps refusing a tunnel that would be intercepted for this claim, and splices one that would not (sandboxd/egress/socks.go:63-64).
  • An env write that adds an entry affects new CONNECTs only. An open spliced tunnel stays spliced until the guest reconnects. Keep-alive pools in clients make this visible, so the docs say it.
  • The envelope check (injectGap) treats an "inject" rule as covering its hosts.
  • Validation: "inject" needs egress_ca like true. A reload that turns it on for a served pool key is refused like true (sandboxd/pool/reload.go:94-108), since guests lack the root.

Query substitution.

  • inject gains query, a parameter name, as an alternative to header:

    {"hosts": ["serpapi.com"], "query": "api_key", "placeholder": "@SERP@"}
  • On an intercepted request, a parameter of that name whose value equals the placeholder is replaced by value. Other parameters and their order are kept byte-exact.

  • No buffering: the URL is already parsed.

  • The value never appears in sandboxd's audit or logs; the proxy logs no paths today, and that stays.

  • Exactly one of header and query is set. A query entry without a placeholder is refused, so a key is never appended blind.

Telemetry.

  • The egress usage event carries the injected names (claim:<name>, never values), as the audit line does (sandboxd/pool/telemetry.go:136-137). A control plane can then show "last used" through the usage API without reading audit.jsonl.

Hot path.

Optional extras (separate; can be split into their own issues)

  1. WebSocket on intercepted hosts. Inject on the Upgrade handshake as on any request. After the upstream answers 101, splice the two connections raw. Today the strip at sandboxd/egress/proxy.go:37 breaks every WebSocket on an intercepted host (OpenAI Realtime, streaming speech APIs, some CLI transports).
  2. Bounded body substitution. For Content-Type application/json or application/x-www-form-urlencoded with Content-Length ≤ 64 KiB, replace the placeholder string in the body and fix the length. Chunked or larger bodies pass unchanged. It needs one buffer per matching request; opt-in per entry ("body": true).
  3. Response scrubbing. Replace an exact echo of value in response headers with the placeholder. Bodies are not scrubbed; they stream.

Acceptance

Tests on one node, with a pool rule {host: "*", intercept: "inject"} and two test origins that echo request headers and the query:

  • A claim with no inject entries reaches both origins spliced: the guest sees the origins' own certificates, and HTTP/2 is negotiated.
  • A claim with an entry for origin A gets the egress leaf on A and the origin's certificate on B, and A sees the injected header only when the guest sends the placeholder.
  • Two claims on the same pool, one with and one without an entry for A, each see their own behavior at the same time.
  • PATCH adding an entry on a running claim: the next new CONNECT to A is intercepted; an already open spliced tunnel stays spliced.
  • PATCH removing the last entry for A: the next CONNECT to A is spliced.
  • A query entry replaces api_key=<placeholder> and leaves api_key=other, the other parameters and their order unchanged. A request without the parameter is unchanged.
  • No value appears in guest env, /proc/*/environ, GET env, a checkpoint, the audit line or the usage event. The usage event names claim:<name>.
  • intercept: "inject" without egress_ca fails config load. A reload turning it on for a served pool key is refused.
  • For the extras, if they are taken:
    • a WebSocket echo through an intercepted host sees the injected header on the handshake and then echoes frames;
    • a 10 KiB JSON body with the placeholder arrives with the value and a correct Content-Length, and a 1 MiB body arrives unchanged;
    • a response header echoing the value reaches the guest as the placeholder.

Docs: docs/egress.md (interception, inject), docs/sandboxd-api.md (the env inject shape).


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions