egress: an "inject" rule intercepts only the claims that inject on the host; query and body credentials

docs/egress.md

@@ -53,14 +53,15 @@ through the same code: a rule with a nonempty `methods` list that omits
 `CONNECT` denies it, and a rule with a `secret` opens it without injecting anything, on
 both ports. The one difference is `intercept`: on 3128 such a rule terminates
 the TLS and filters the requests inside, on 1080 there is no HTTP to filter, so
-the tunnel is refused. The door is opt-in: the pool policy sets
+the tunnel is refused. An `"inject"` rule is a plain one on 1080: the door never
+injects, so it splices whatever credentials the claim holds. The door is opt-in: the pool policy sets
 `"socks5": true`, otherwise the host side stays unwired and the guest's dial
 is refused like the HTTP one with no policy. The tenant policy's rules gate
 every tunnel through the door as they gate `CONNECT`, so the tenant does not
 opt in separately; its own `socks5` counts only on a claim outside any
 configured pool, where the tenant policy is the whole policy. A policy that
 opts in with rules that all carry a nonempty `methods` list without
-`CONNECT`, or all `intercept`, is rejected at load. A pool whose policy does
+`CONNECT`, or all `intercept: true`, is rejected at load. A pool whose policy does
 not opt in pays
 nothing for the door on the claim path, whatever its tenants' policies say.
 Audit lines (with `audit_log` on) carry `"method":"SOCKS5"`.
@@ -242,11 +243,16 @@ woken sandbox binds at arm time.
   injection needs `intercept`; an absolute-form `https://` request on the
   forward door needs none — the proxy dials the origin over verified TLS
   itself, checks the method, and injects.
-- `intercept`: terminate a matched HTTPS CONNECT so the request is filtered by
-  method and the secret injected (see below). Only a pool rule may set it.
+- `intercept`: `true` terminates a matched HTTPS CONNECT so the request is
+  filtered by method and the secret injected (see below). `"inject"` terminates
+  it only for a claim with a [credential](#claim-credentials) for that host; for
+  every other claim the rule is a plain one, so its tunnels splice and HTTP/2,
+  WebSocket and pinned clients keep working. An `"inject"` rule cannot carry a
+  `secret`. Only a pool rule may set `intercept`.
 - No policy on a claim ⇒ no egress at all (the proxy is not started).
 
-Each decision is metered as an `egress` usage event and, when `audit_log` is
+Each decision is metered as an `egress` usage event, which names the injected
+secrets in `secret` as the audit line does, and, when `audit_log` is
 on, written to `audit.jsonl` (`op:"egress"`, `dest`, `port`, `method`,
 `decision`, and the secret **name** in `secret`); an intercepted `CONNECT` is
 recorded as a decision of its own before its inner requests.
@@ -287,7 +293,8 @@ gateway key per user, while the guest holds a placeholder.
 
 The proxy reads a snapshot of the claim's env that each env write publishes,
 without a lock, once per request whose matched rule names a secret or whose
-host has a claim credential; other requests pay nothing.
+host has a claim credential, and once per `CONNECT` an `"inject"` rule covers;
+other requests pay nothing.
 
 ### Claim credentials
 
@@ -317,23 +324,68 @@ request while a browser on the same host gets nothing:
 git config --global http.https://github.com/.extraHeader "Authorization: Basic @GH_GIT@"
 ```
 
+A key that a client sends as a query parameter takes `query` instead of
+`header`, and a placeholder is required. On an intercepted request, the first
+`<query>=<placeholder>` pair of the URL gets the value, URL-escaped; the
+placeholder may arrive percent-encoded, and every other byte of the query
+stays as sent. With `body: true` and no placeholder in the URL, the first such
+pair of an `application/x-www-form-urlencoded` body, or the first top-level
+string member of that name holding the placeholder in an `application/json`
+body, is filled instead, when the body has a known length of at most 64 KiB;
+any other body streams unchanged. One fill per credential bounds what a
+request can grow by:
+
+```jsonc
+"FB": { "value": "EAAB…", "guest": false,
+        "inject": { "hosts": ["graph.facebook.com"], "query": "access_token", "body": true, "placeholder": "@FB@" } }
+```
+
+```sh
+curl "https://graph.facebook.com/me?access_token=@FB@"
+curl https://graph.facebook.com/me/feed -d access_token=@FB@ -d message=hi
+```
+
+Pair the credential with an `"inject"` pool rule, for example
+`{ "host": "*", "intercept": "inject" }`, and only the claims that hold a
+credential for a host have it intercepted: the operator decides which hosts may
+be intercepted, the claim's entries decide which of them are. The decision is
+taken per CONNECT, so a write that adds or removes an entry applies to the
+guest's next connection; a tunnel the guest keeps open stays as it was.
+
 The placeholder is not a secret: it stays in the guest and in `GET /env`.
 The entry rides the claim request or any `/env` write, and rotation is the
 same `PATCH`, accepted on a hibernated or archived claim without waking it.
 
 - **Envelope.** The operator still decides which hosts can carry a
-  credential: every host must be covered by an `intercept: true` rule of the
-  claim's pool and allowed by its tenant class, or the write answers 400 and
+  credential: every host must be covered by an `intercept` rule (`true` or
+  `"inject"`) of the claim's pool and allowed by its tenant class, or the write answers 400 and
   nothing is stored. A credential never widens egress and never rides a
-  plaintext forward request.
+  plaintext forward request. Keep a credential's `hosts` to the vendor's own
+  names: under a `*` rule, a pattern such as `*.amazonaws.com` or
+  `*.vercel.app` also matches subdomains a guest can own.
 - **Precedence.** The pool rule's own `secret` is set first and wins: a
   credential naming that header for a host the rule covers is a 400. Among
   credentials for the same header, the first by entry name whose placeholder
   (if any) the request carries wins. An empty value injects nothing.
+- **Echo.** Origins reflect the request URL: a redirect's `Location`, a
+  paging link, an echoed query. On a request that carried a `query`
+  credential, its value, as sent, query- or path-escaped, or with `/` written
+  `\/` as JSON may, reaches the guest as the placeholder in every
+  response header and in the response body. The proxy drops the guest's
+  `Accept-Encoding`, `Range` and `If-Range` on such a request, so it reads the
+  whole body uncompressed, and streams it holding back only a tail that may
+  begin a match, so event streams stay live. A value the origin transforms
+  otherwise is beyond the scrub, such as the JSON escape of a quote, backslash
+  or tab inside a value, and so is a value the origin stored earlier
+  and returns on a later request that carries no credential: the scrub covers
+  only the response to the request that sent the value. `header` credentials, which origins do not reflect, are
+  not scrubbed and add no work to the response.
 - **Shape.** `hosts` holds 1 to 8 lowercase names or `*.suffix` patterns
-  (no bare `*`, no port); `header` is any header name except `Host`,
-  `Content-Length` and the hop-by-hop ones; `placeholder`, when set, is at most
-  256 bytes of header value without control characters or surrounding space.
+  (no bare `*`, no port); exactly one of `header` and `query`; `header` is any
+  header name except `Host`, `Content-Length` and the hop-by-hop ones; `query`
+  needs a `placeholder`, and `body` needs `query`; `placeholder`, when set, is
+  at most 256 bytes of header value without control characters or surrounding
+  space.
   `GET /env` returns `inject` with the value blanked.
 - **Reload.** A reload that drops the covering intercept rule stops that
   injection on the next request and logs which claims lost which hosts.
@@ -389,8 +441,8 @@ usage-journal append per connection, which is why it is opt-in.
 
 ## HTTPS interception
 
-A rule with `intercept: true` makes the proxy terminate that host's TLS with a
-leaf it signs, so it sees the request's method and path and can inject the
+A rule with `intercept: true`, or `"inject"` for a claim with a credential for
+the host, makes the proxy terminate that host's TLS with a leaf it signs, so it sees the request's method and path and can inject the
 secret into HTTPS — the same guarantees plaintext already has. Upstream is
 re-originated and verified against the host's real root store; the proxy never
 trusts an unverified origin. A guest that caches TLS sessions resumes them on
@@ -501,5 +553,6 @@ Swapping the intermediate while old-root guests are alive breaks interception
 for them, with no re-claim path to fix it.
 
 Limitations: interception is HTTP/1.1 only and breaks clients that pin
-certificates, so scope it to hosts you control. A host that speaks a non-HTTP
+certificates, so scope it to hosts you control, or use `"inject"` so only the
+claims that inject on a host are affected. A host that speaks a non-HTTP
 protocol over TLS must not be given an intercept rule.

docs/sandboxd-api.md

@@ -73,8 +73,9 @@ Auth: `Authorization: Bearer <api_token>` (when configured).
   host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a
   `guest: false` entry never enters it and only feeds the node's egress
   [secrets](egress.md#claim-env-and-secrets), or, with `inject`
-  (`{"hosts": [...], "header": "...", "placeholder": "..."}`, `placeholder`
-  optional), becomes a
+  (`{"hosts": [...], "header": "...", "placeholder": "..."}` with `placeholder`
+  optional, or `{"hosts": [...], "query": "...", "body": true, "placeholder": "..."}`
+  with `body` optional), becomes a
   [claim credential](egress.md#claim-credentials) the proxy sends to those
   hosts. A bad entry answers 400, and an `inject` host the claim's egress does
   not intercept answers 400 with the claim's VM released; see
@@ -730,7 +731,7 @@ values:
 {"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}}
 ```
 
-An entry with `inject` comes back with its hosts, header and placeholder, value blanked.
+An entry with `inject` comes back with its hosts, header or query, body flag and placeholder, value blanked.
 An `inject` host the claim's pool does not intercept, or its tenant class does
 not allow, and a header the pool rule's own secret already sets on that host,
 answer 400 and store nothing.
@@ -788,7 +789,8 @@ Always on: every lifecycle transition appends one JSONL event to
 "id": "sb_…", "vm": "sbx-…"}` plus `key` (the pool key's stable hash, claim
 events), `tenant` (the owning tenant, on claim, egress, archive, unarchive and
 archive_delete), `children` (fork) and `ref` (the promoted
-template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event routed through an
+template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event that injected credentials names
+them in `secret` (`gh,claim:<entry>`, never a value). An egress event routed through an
 [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with
 `egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. A volume claim also carries
 `volumes`, the applied catalog names, and — omitted when empty — `volumes_rw`,

docs/security.md

@@ -134,7 +134,8 @@ Facts to plan around, stated so the boundary is honest:
   cocoonstack/cocoon#83). Compensate with dedicated sandbox nodes and a
   minimal host.
 - **HTTPS interception is HTTP/1.1 only and breaks certificate-pinning
-  clients** — scope `intercept` rules to hosts you control
+  clients** — scope `intercept` rules to hosts you control, or use
+  `"inject"` so only claims holding a credential for a host are intercepted
   ([egress](egress.md#https-interception)).
 - **The audit and usage journals are local JSONL** with size rotation —
   no tamper-evidence. Ship them off-node if integrity against a host

e2e/cmd/credsmoke/main.go

@@ -1,6 +1,6 @@
-// credsmoke is the claim-credential acceptance: claims on one intercept pool each
-// send their own env-injected header to an HTTPS echo through the proxy, the guest
-// never holds the value, and rotation, refusal, clearing and forks behave as the API says.
+// credsmoke is the claim-credential acceptance: claims each send their own env-injected header,
+// query or body credential through the proxy (-inject-size: on an "inject" pool), the guest never
+// holds the value, and rotation, refusal, clearing and forks behave as the API says.
 package main
 
 import (
@@ -9,6 +9,7 @@ import (
 	"flag"
 	"fmt"
 	"net/http"
+	"net/url"
 	"os"
 	"strconv"
 	"strings"
@@ -22,6 +23,9 @@ const (
 	proxy  = "http://127.0.0.1:3128"
 	header = "X-Api-Key"
 	entry  = "API_KEY"
+	tsPH   = "@TS@"
+	serpPH = "@SERP@"
+	fbPH   = "@FB@"
 )
 
 func main() {
@@ -32,22 +36,27 @@ func main() {
 	secret := flag.String("secret", "", "the pool secret's value the origin should also observe")
 	reattach := flag.String("reattach", "", "id:token:value of a claim to recheck after a restart")
 	sse := flag.String("sse", "", "plain HTTP event stream that sends one event, then holds")
+	injectSize := flag.String("inject-size", "", "size of the template's pool whose only rule is intercept \"inject\"; empty skips that leg")
+	other := flag.String("other", "example.com", "a second HTTPS host the inject pool must splice")
 	flag.Parse()
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
 	var err error
-	if *reattach != "" {
-		err = recheck(*addr, *token, *echo, *reattach)
-	} else {
-		err = run(*addr, *token, *template, *echo, *secret, *sse)
-	}
+	switch {
+	case *reattach != "":
+		err = recheck(ctx, *addr, *token, *echo, *reattach)
+	case *injectSize != "":
+		err = conditional(ctx, *addr, *token, *template, sandbox.Size(*injectSize), *echo, *other)
+	default:
+		err = run(ctx, *addr, *token, *template, *echo, *secret, *sse)
+	}
+	cancel()
 	if err != nil {
 		fmt.Fprintln(os.Stderr, "credsmoke:", err)
 		os.Exit(1)
 	}
 }
 
-func run(addr, token, template, echo, secret, sse string) error {
-	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
-	defer cancel()
+func run(ctx context.Context, addr, token, template, echo, secret, sse string) error {
 	stamp := strconv.FormatInt(time.Now().UnixNano(), 10)
 	valueA, valueB := "key-a-"+stamp, "key-b-"+stamp
 
@@ -171,9 +180,7 @@ func run(addr, token, template, echo, secret, sse string) error {
 	return nil
 }
 
-func recheck(addr, token, echo, reattach string) error {
-	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
-	defer cancel()
+func recheck(ctx context.Context, addr, token, echo, reattach string) error {
 	parts := strings.SplitN(reattach, ":", 3)
 	if len(parts) != 3 {
 		return fmt.Errorf("-reattach %q, want id:token:value", reattach)
@@ -195,6 +202,87 @@ func recheck(addr, token, echo, reattach string) error {
 	return nil
 }
 
+func conditional(ctx context.Context, addr, token, template string, size sandbox.Size, echo, other string) error {
+	claim := func() (*sandbox.Sandbox, error) {
+		_, sb, err := harness.Claim(ctx, addr, token, template, sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(size))
+		return sb, err
+	}
+	vault, err := claim()
+	if err != nil {
+		return err
+	}
+	defer func() { _ = vault.Close() }()
+	bare, err := claim()
+	if err != nil {
+		return err
+	}
+	defer func() { _ = bare.Close() }()
+	fmt.Printf("  claimed %s (vault) and %s (bare) on the inject pool\n", vault.ID, bare.ID)
+	for _, host := range []string{echo, other} {
+		if err = expectTunnel(ctx, vault, host, false); err != nil {
+			return fmt.Errorf("before any credential: %w", err)
+		}
+	}
+	fmt.Println("  with no credential both hosts splice: origin certificates, HTTP/2")
+
+	stamp := strconv.FormatInt(time.Now().UnixNano(), 10)
+	serp, fb := "serp "+stamp+"/+", "fb-"+stamp
+	if err = vault.PatchEnv(ctx, map[string]*sandbox.EnvVar{
+		"TS":   {Value: "2016-10-10", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Query: "timestamp", Placeholder: tsPH}},
+		"SERP": {Value: serp, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Query: "api_key", Placeholder: serpPH}},
+		"FB":   {Value: fb, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Query: "access_token", Body: true, Placeholder: fbPH}},
+	}); err != nil {
+		return fmt.Errorf("add credentials: %w", err)
+	}
+	for _, tt := range []struct {
+		sb          *sandbox.Sandbox
+		host        string
+		intercepted bool
+	}{{vault, echo, true}, {vault, other, false}, {bare, echo, false}} {
+		if err = expectTunnel(ctx, tt.sb, tt.host, tt.intercepted); err != nil {
+			return fmt.Errorf("after the credentials: %w", err)
+		}
+	}
+	fmt.Println("  the claim with credentials is intercepted on their host only; the other claim still splices it")
+
+	base := "https://" + echo
+	form, doc := "access_token="+fb+"&m=hi", `{"access_token":"`+fb+`"}`
+	for _, tt := range []struct {
+		name, args, want, not string
+	}{
+		{"query", "'" + base + "/time/valid?timestamp=" + url.QueryEscape(tsPH) + "'", `"valid":true`, "2016"},
+		{"other query value", "'" + base + "/time/valid?timestamp=bogus'", `"valid":false`, "2016"},
+		{"echoed query", "'" + base + "/get?b=2&api_key=" + url.QueryEscape(serpPH) + "&a=1'", serpPH, stamp},
+		{"form body", base + "/post -d access_token=" + fbPH + " -d m=hi", fmt.Sprintf(`"content-length":"%d"`, len(form)), fb},
+		{"json body", base + `/post -H 'Content-Type: application/json' -d '{"access_token":"` + fbPH + `"}'`, fmt.Sprintf(`"content-length":"%d"`, len(doc)), fb},
+		{"response header echo", "-D - -o /dev/null '" + base + "/response-headers?api_key=" + url.QueryEscape(serpPH) + "'", serpPH, stamp},
+	} {
+		out, execErr := vault.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sS -x %s %s", proxy, tt.args))
+		if execErr != nil {
+			return fmt.Errorf("%s exec: %w (%s)", tt.name, execErr, out)
+		}
+		if !strings.Contains(out, tt.want) || strings.Contains(out, tt.not) {
+			return fmt.Errorf("%s: guest saw %q, want %q and not %q", tt.name, out, tt.want, tt.not)
+		}
+	}
+	fmt.Println("  query and form/JSON body placeholders filled at the origin; every echo of a value reached the guest as its placeholder")
+
+	leak, err := vault.Exec(ctx, "sh", "-c", fmt.Sprintf("grep -rlsF %q /proc/[0-9]*/environ /run /etc 2>/dev/null; env | grep -cF %q; true", fb, fb))
+	if err != nil || strings.TrimSpace(leak) != "0" {
+		return fmt.Errorf("the credential reached the guest: %q %v", leak, err)
+	}
+	if err = vault.SetEnv(ctx, map[string]sandbox.EnvVar{}); err != nil {
+		return fmt.Errorf("clear: %w", err)
+	}
+	if err = expectTunnel(ctx, vault, echo, false); err != nil {
+		return fmt.Errorf("after a clear: %w", err)
+	}
+	fmt.Println("  value absent from the guest; a cleared env splices the next connection")
+	fmt.Printf("VAULT %s\nVAULT %s\n", serp, fb)
+	fmt.Println("CREDSMOKE INJECT PASS")
+	return nil
+}
+
 func credential(echo, value string) map[string]sandbox.EnvVar {
 	return map[string]sandbox.EnvVar{entry: {Value: value, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: header}}}
 }
@@ -220,6 +308,22 @@ func expectPlaceholder(ctx context.Context, sb *sandbox.Sandbox, echo, value str
 	return nil
 }
 
+// expectTunnel reads the issuer and HTTP version the guest gets from host; an intercepted tunnel shows the e2e CA and HTTP/1.1.
+func expectTunnel(ctx context.Context, sb *sandbox.Sandbox, host string, intercepted bool) error {
+	out, err := sb.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sSv -o /dev/null -w 'version=%%{http_version}\\n' -x %s https://%s/ 2>&1", proxy, host))
+	if err != nil {
+		return fmt.Errorf("%s exec: %w (%s)", host, err, out)
+	}
+	_, rest, found := strings.Cut(out, "issuer:")
+	issuer, _, _ := strings.Cut(rest, "\n")
+	mitm := strings.Contains(issuer, "e2e")
+	h2 := strings.Contains(out, "version=2")
+	if !found || mitm != intercepted || h2 == intercepted {
+		return fmt.Errorf("%s on %s: issuer %q, HTTP/2 %t, want intercepted %t:\n%s", host, sb.ID, issuer, h2, intercepted, out)
+	}
+	return nil
+}
+
 func expectHeader(ctx context.Context, sb *sandbox.Sandbox, echo, want, not, secret string) error {
 	out, err := sb.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sS -x %s -H '%s: placeholder' https://%s/headers", proxy, header, echo))
 	if err != nil {