diff --git a/docs/cluster.md b/docs/cluster.md index bb86f8e0..7c95dcf3 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -331,13 +331,15 @@ Some config must match on every node or the cluster fails in confusing ways: | `preview_secret` | a preview URL signed on one node fails verification on another | | `mesh.cluster_key` | nodes cannot join / decrypt gossip at all | | `egress_ca` cluster root | a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception | +| egress: `egress_classes`, each pool's `egress` and `egress_upstream_env`, `secrets` names, headers and `value_env`, `egress_internal_allow`, `egress_upstream` | a claim or tenant redirected to a divergent node reaches different hosts, and its [claim credentials](egress.md#claim-credentials) are refused or not injected there | | the engine root path (cocoon's `root_dir`) | a checkpoint or template export pins the base image blobs' absolute path under the root that captured it; a node whose root sits at another path refuses to clone a record it healed or pulled (`untrusted storage path in snapshot metadata`) — the heal succeeds, the branch fails. Not part of the gossiped digest below | Each node gossips a digest of these over its live tenant set and quotas (HMAC-keyed by `cluster_key` when set, covering each tenant token's SHA-256; otherwise a token-free digest of tenant names and quotas + the CA root + `checkpoint_ttl_hours`, so -nothing brute-forceable rides cleartext gossip), and republishes it after every -tenant change. A mismatch logs a warning at the moment +nothing brute-forceable rides cleartext gossip) plus the egress config above in +an order-free form, never a secret value, and republishes it after every tenant +change and every config reload. A mismatch logs a warning at the moment the divergent node appears — not at the first unlucky redirect — and raises the `sandboxd_config_digest_mismatch` gauge. It is warn-only: a rolling credential rotation is a legitimate transient mismatch, so a divergence never partitions diff --git a/docs/egress.md b/docs/egress.md index f5a36965..c1f47ebb 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -285,8 +285,46 @@ gateway key per user, while the guest holds a placeholder. - **Audit.** An injection is recorded as the secret's name; the value is never logged, listed or returned. -The proxy reads the claim's env under the node's manager lock, once per -request whose matched rule names a secret; other requests pay nothing. +The proxy reads a snapshot of the claim's env that each env write publishes, +without a lock, once per request whose matched rule names a secret or whose +host has a claim credential; other requests pay nothing. + +### Claim credentials + +A `guest: false` env entry can also name where its value goes, so a control +plane binds each sandbox's own credential to a host without editing the node +config: + +```jsonc +{ "env": { + "API_TOKEN": { "value": "Bearer …", "guest": false, + "inject": { "hosts": ["api.example.com", "*.git.example.com"], "header": "Authorization" } } +} } +``` + +On an intercepted request to one of `hosts`, the proxy sets `header` to the +entry's value, overwriting whatever the guest sent, so the guest can hold a +placeholder. The entry rides the claim request or any `/env` write, and +rotation is the same `PATCH`, accepted on a hibernated or archived claim +without waking it. + +- **Envelope.** The operator still decides which hosts can carry a + credential: every host must be covered by an `intercept: true` rule of the + claim's pool and allowed by its tenant class, or the write answers 400 and + nothing is stored. A credential never widens egress and never rides a + plaintext forward request. +- **Precedence.** The pool rule's own `secret` is set first and wins: a + credential naming that header for a host the rule covers is a 400. Among + credentials for the same header, the first by entry name wins. An empty + value injects nothing. +- **Shape.** `hosts` holds 1 to 8 lowercase names or `*.suffix` patterns + (no bare `*`, no port); `header` is any header name except `Host`, + `Content-Length` and the hop-by-hop ones. `GET /env` returns `inject` with + the value blanked. +- **Reload.** A reload that drops the covering intercept rule stops that + injection on the next request and logs which claims lost which hosts. +- **Audit.** An injection is recorded as `claim:` next to the + pool secret's name, comma-separated. ## Upstream proxies @@ -341,7 +379,11 @@ A rule with `intercept: true` makes the proxy terminate that host's TLS with a leaf it signs, so it sees the request's method and path and can inject the secret into HTTPS — the same guarantees plaintext already has. Upstream is re-originated and verified against the host's real root store; the proxy never -trusts an unverified origin. +trusts an unverified origin. A guest that caches TLS sessions resumes them on +its next CONNECT through the same sandbox, which skips the leaf signature and +chain check. A response that is an event stream or has no length +reaches the guest as each chunk arrives, on intercepted and forward requests +alike. Trust is a two-tier PKI. One **cluster root CA** is the trust anchor: its certificate (public) is baked into every interception guest's store, so a leaf diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 49105414..c185c488 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -68,12 +68,16 @@ Auth: `Authorization: Bearer ` (when configured). cold-booting an image named like the template - `env` is the claim's own environment, at most 64 entries: each name a shell identifier of at most 128 bytes, each value at most 8 KiB with no control - characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value` and `guest`; any other + characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other member, or `"guest": null`, answers 400, so a mistyped flag never lands a host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a `guest: false` entry never enters it and only feeds the node's egress - [secrets](egress.md#claim-env-and-secrets). A bad entry answers 400 before - any VM is touched; see [claim env](#getputpatch-v1sandboxesidenv) + [secrets](egress.md#claim-env-and-secrets), or, with `inject` + (`{"hosts": [...], "header": "..."}`), becomes a + [claim credential](egress.md#claim-credentials) the proxy sends to those + hosts. A bad entry answers 400, and an `inject` host the claim's egress does + not intercept answers 400 with the claim's VM released; see + [claim env](#getputpatch-v1sandboxesidenv) - `egress` `false` claims with no [egress policy](egress.md), whatever the pool's (or, for a promoted template, its source pool's): the guest's doors are never bound. Omitted or `true` keeps the policy the claim would get @@ -725,6 +729,11 @@ values: {"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}} ``` +An entry with `inject` comes back with its hosts and header, value blanked. +An `inject` host the claim's pool does not intercept, or its tenant class does +not allow, and a header the pool rule's own secret already sets on that host, +answer 400 and store nothing. + The env is persisted with the claim. A `guest: false` change takes effect on the egress proxy's next request at any time, a hibernated or archived claim included, and never touches the guest. A change to the guest entries is @@ -738,8 +747,8 @@ result does not rewrite the claims journal. A 500 after the store step means the env is stored and its `guest: false` values are already live, but the guest write failed; resend the same request to deliver it. The guest side is described in [silkd](silkd.md#claim-env). -`PUT`/`PATCH`/`DELETE` answer 204, `GET` 200; 400 a bad entry or an unknown body -field; 404 unknown id or another tenant's claim; 409 a guest change on a +`PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body +field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a paused or failed sandbox. ## PUT /v1/sandboxes/{id}/instance-metadata diff --git a/e2e/cmd/credsmoke/main.go b/e2e/cmd/credsmoke/main.go new file mode 100644 index 00000000..eda6fb98 --- /dev/null +++ b/e2e/cmd/credsmoke/main.go @@ -0,0 +1,212 @@ +// credsmoke is the claim-credential acceptance: claims on one intercept pool each +// send their own env-injected header to an HTTPS echo through the proxy, the guest +// never holds the value, and rotation, refusal, clearing and forks behave as the API says. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "net/http" + "os" + "strconv" + "strings" + "time" + + "github.com/cocoonstack/sandbox/e2e/internal/harness" + sandbox "github.com/cocoonstack/sandbox/sdk/go" +) + +const ( + proxy = "http://127.0.0.1:3128" + header = "X-Api-Key" + entry = "API_KEY" +) + +func main() { + addr := flag.String("addr", "127.0.0.1:7882", "sandboxd address") + token := flag.String("token", "", "node api token") + template := flag.String("template", "rt:24.04", "template of the intercept pool") + echo := flag.String("echo", "postman-echo.com", "HTTPS host that echoes request headers at /headers") + secret := flag.String("secret", "", "the pool secret's value the origin should also observe") + reattach := flag.String("reattach", "", "id:token:value of a claim to recheck after a restart") + sse := flag.String("sse", "", "plain HTTP event stream that sends one event, then holds") + flag.Parse() + var err error + if *reattach != "" { + err = recheck(*addr, *token, *echo, *reattach) + } else { + err = run(*addr, *token, *template, *echo, *secret, *sse) + } + if err != nil { + fmt.Fprintln(os.Stderr, "credsmoke:", err) + os.Exit(1) + } +} + +func run(addr, token, template, echo, secret, sse string) error { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + stamp := strconv.FormatInt(time.Now().UnixNano(), 10) + valueA, valueB := "key-a-"+stamp, "key-b-"+stamp + + claim := func(env map[string]sandbox.EnvVar) (*sandbox.Sandbox, error) { + opts := []sandbox.Option{sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.Small)} + if env != nil { + opts = append(opts, sandbox.WithEnv(env)) + } + _, sb, err := harness.Claim(ctx, addr, token, template, opts...) + return sb, err + } + a, err := claim(credential(echo, valueA)) + if err != nil { + return err + } + b, err := claim(credential(echo, valueB)) + if err != nil { + return err + } + defer func() { _ = b.Close() }() + plain, err := claim(nil) + if err != nil { + return err + } + defer func() { _ = plain.Close() }() + fmt.Printf(" claimed %s (A), %s (B), %s (plain) on one intercept pool\n", a.ID, b.ID, plain.ID) + + for _, tc := range []struct { + name string + sb *sandbox.Sandbox + want, not string + }{ + {"A", a, valueA, valueB}, + {"B", b, valueB, valueA}, + {"plain", plain, "placeholder", "key-"}, + } { + if err = expectHeader(ctx, tc.sb, echo, tc.want, tc.not, secret); err != nil { + return fmt.Errorf("%s: %w", tc.name, err) + } + } + fmt.Println(" each claim's origin saw only its own credential, the plain claim the guest's placeholder, all the pool secret") + if sse != "" { + out, _ := plain.Exec(ctx, "sh", "-c", fmt.Sprintf("timeout 4 curl -sSN --noproxy '' -x %s %s 2>&1; true", proxy, sse)) + if !strings.Contains(out, "data: one") { + return fmt.Errorf("the first event did not reach the guest while the stream stayed open: %q", out) + } + fmt.Println(" an open event stream reached the guest event by event through the proxy") + } + + leak, err := a.Exec(ctx, "sh", "-c", fmt.Sprintf("grep -rls %q /proc/[0-9]*/environ /run /etc 2>/dev/null; env | grep -c %q; true", valueA, valueA)) + if err != nil { + return fmt.Errorf("leak probe: %w", err) + } + if strings.TrimSpace(leak) != "0" { + return fmt.Errorf("the credential reached the guest: %q", leak) + } + env, err := a.Env(ctx) + if err != nil { + return fmt.Errorf("read env: %w", err) + } + if v := env[entry]; v.Value != "" || v.Inject == nil || v.Inject.Header != header { + return fmt.Errorf("GET env %+v, want the inject named with the value blanked", v) + } + fmt.Println(" value absent from the guest (env, /proc/*/environ, /run, /etc) and blanked in GET env") + + rotated := valueA + "-r1" + if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil { + return fmt.Errorf("rotate: %w", err) + } + if err = expectHeader(ctx, a, echo, rotated, valueA+"\"", secret); err != nil { + return fmt.Errorf("after a rotation: %w", err) + } + if err = a.Hibernate(ctx); err != nil { + return fmt.Errorf("hibernate: %w", err) + } + rotated += "-r2" + if err = a.PatchEnv(ctx, map[string]*sandbox.EnvVar{entry: new(credential(echo, rotated)[entry])}); err != nil { + return fmt.Errorf("rotate while hibernated: %w", err) + } + if err = expectHeader(ctx, a, echo, rotated, "-r1\"", secret); err != nil { + return fmt.Errorf("after a rotation while hibernated: %w", err) + } + fmt.Println(" rotation applied on the next request, a hibernated claim rotated without waking") + + for name, patch := range map[string]*sandbox.EnvVar{ + "an uncovered host": {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{"example.org"}, Header: header}}, + "the pool secret's header": {Value: "x", Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Egress-Token"}}, + "a guest entry": {Value: "x", Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: "X-Other"}}, + } { + err = b.PatchEnv(ctx, map[string]*sandbox.EnvVar{"BAD": patch}) + if apiErr, ok := errors.AsType[*sandbox.APIError](err); !ok || apiErr.Status != http.StatusBadRequest { + return fmt.Errorf("inject on %s: %v, want 400", name, err) + } + } + if env, err = b.Env(ctx); err != nil || len(env) != 1 { + return fmt.Errorf("env after refused patches %v %v, want only the original entry", env, err) + } + if err = b.SetEnv(ctx, map[string]sandbox.EnvVar{}); err != nil { + return fmt.Errorf("clear: %w", err) + } + if err = expectHeader(ctx, b, echo, "placeholder", valueB, secret); err != nil { + return fmt.Errorf("after a clear: %w", err) + } + fmt.Println(" uncovered host, pool secret header and guest entry refused with 400; a cleared env stops injecting") + + children, err := a.Fork(ctx, 1, time.Minute) + if err != nil { + return fmt.Errorf("fork: %w", err) + } + defer func() { _ = children[0].Close() }() + if err = expectHeader(ctx, children[0], echo, "placeholder", rotated, secret); err != nil { + return fmt.Errorf("fork child: %w", err) + } + fmt.Println(" a fork child carries no credential") + fmt.Printf("REATTACH %s:%s:%s\n", a.ID, a.Token(), rotated) + fmt.Println("CREDSMOKE PASS") + return nil +} + +func recheck(addr, token, echo, reattach string) error { + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + parts := strings.SplitN(reattach, ":", 3) + if len(parts) != 3 { + return fmt.Errorf("-reattach %q, want id:token:value", reattach) + } + client, err := sandbox.Connect(addr, sandbox.WithAPIToken(token)) + if err != nil { + return err + } + sb, err := client.Lookup(ctx, parts[0], parts[1]) + if err != nil { + return fmt.Errorf("lookup after restart: %w", err) + } + defer func() { _ = sb.Close() }() + if err = expectHeader(ctx, sb, echo, parts[2], "placeholder", ""); err != nil { + return fmt.Errorf("after a restart: %w", err) + } + fmt.Println(" the credential survived a sandboxd restart") + fmt.Println("CREDSMOKE REATTACH PASS") + return nil +} + +func credential(echo, value string) map[string]sandbox.EnvVar { + return map[string]sandbox.EnvVar{entry: {Value: value, Guest: new(false), Inject: &sandbox.EnvInject{Hosts: []string{echo}, Header: header}}} +} + +func expectHeader(ctx context.Context, sb *sandbox.Sandbox, echo, want, not, secret string) error { + out, err := sb.Exec(ctx, "sh", "-c", fmt.Sprintf("curl -sS -x %s -H '%s: placeholder' https://%s/headers", proxy, header, echo)) + if err != nil { + return fmt.Errorf("proxied HTTPS exec: %w (%s)", err, out) + } + switch { + case !strings.Contains(out, want): + return fmt.Errorf("origin did not see %q:\n%s", want, out) + case not != "" && strings.Contains(out, not): + return fmt.Errorf("origin saw %q:\n%s", not, out) + case secret != "" && !strings.Contains(out, secret): + return fmt.Errorf("origin did not see the pool secret:\n%s", out) + } + return nil +} diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index 61a103e9..31d053f9 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -272,13 +272,14 @@ func (c *Config) HasEgress() bool { return len(c.Bridges) > 0 || len(c.Networks) > 0 } -// ClusterDigest fingerprints the must-match config and live tenant records; without cluster_key it omits token hashes. +// ClusterDigest fingerprints the must-match config, its egress layer and the live tenant records; without cluster_key it omits token hashes. func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) string { tenants = slices.Clone(tenants) slices.SortFunc(tenants, func(a, b TenantRecord) int { return strings.Compare(a.Name, b.Name) }) + eg := c.egressFingerprint() if c.Mesh != nil { if key, _ := c.Mesh.DecodedKey(); key != nil { - raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours}) + raw, _ := utils.DigestJSON([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants, c.CheckpointTTLHours, eg}) mac := hmac.New(sha256.New, key) mac.Write(raw) return hex.EncodeToString(mac.Sum(nil)) @@ -287,7 +288,37 @@ func (c *Config) ClusterDigest(caFingerprint string, tenants []TenantRecord) str for i := range tenants { tenants[i].TokenSHA256 = "" } - return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours}) + return utils.DigestHex([]any{caFingerprint, tenants, c.CheckpointTTLHours, eg}) +} + +type egressFingerprint struct { + Classes []EgressClass `json:"classes,omitempty"` + Secrets []egress.SecretSpec `json:"secrets,omitempty"` + Internal []string `json:"internal,omitempty"` + Upstream *EgressUpstreamConfig `json:"upstream,omitempty"` + Pools map[string]PoolSpec `json:"pools,omitempty"` +} + +// egressFingerprint is the egress config every node must share, in an order-free form; secret values stay out. +func (c *Config) egressFingerprint() egressFingerprint { + fp := egressFingerprint{ + Classes: slices.SortedFunc(slices.Values(c.EgressClasses), func(a, b EgressClass) int { return strings.Compare(a.Name, b.Name) }), + Internal: slices.Sorted(slices.Values(c.EgressInternalAllow)), + Pools: map[string]PoolSpec{}, + } + for _, s := range c.Secrets { + fp.Secrets = append(fp.Secrets, egress.SecretSpec{Name: s.Name, Header: s.Header, ValueEnv: s.ValueEnv}) + } + slices.SortFunc(fp.Secrets, func(a, b egress.SecretSpec) int { return strings.Compare(a.Name, b.Name) }) + if u := c.EgressUpstream; u != nil { + fp.Upstream = &EgressUpstreamConfig{ClaimEnv: u.ClaimEnv, Allow: slices.Sorted(slices.Values(u.Allow))} + } + for _, p := range c.Pools { + if p.Egress != nil || p.EgressUpstreamEnv != "" { + fp.Pools[poolLabel(p.PoolKey)] = PoolSpec{Egress: p.Egress, EgressUpstreamEnv: p.EgressUpstreamEnv} + } + } + return fp } func (c *Config) guardsEgressLane() bool { diff --git a/sandboxd/config/config_test.go b/sandboxd/config/config_test.go index ae34f5f7..54387ee2 100644 --- a/sandboxd/config/config_test.go +++ b/sandboxd/config/config_test.go @@ -9,6 +9,7 @@ import ( "strings" "testing" + "github.com/cocoonstack/sandbox/sandboxd/egress" "github.com/cocoonstack/sandbox/sandboxd/types" ) @@ -73,9 +74,9 @@ func TestClusterDigestBytesArePinned(t *testing.T) { fp string want string }{ - {"keyless", &Config{APIToken: "tk", PreviewSecret: "p&s", CheckpointTTLHours: 24}, tenants, "ca<&>\u2028fp", "889dd395d66341236c2e073bea4b9455e48a48eb7ea79a69f60bda9b914f40f9"}, - {"keyed", &Config{APIToken: "tk&\u2029", PreviewSecret: "p&s<", CheckpointTTLHours: 24, Mesh: &MeshConfig{ClusterKey: key}}, tenants, "ca<&>fp", "1bf87485f6d3a2e097ffa0835127601520725185d06ed960b441888e2538a6c8"}, - {"no tenants", &Config{}, nil, "fp", "38fc616b12f612c2c5c3f83af9d4c6caa771e1d3ffca39a6c835e522d3db49f8"}, + {"keyless", &Config{APIToken: "tk", PreviewSecret: "p&s", CheckpointTTLHours: 24}, tenants, "ca<&>\u2028fp", "ad397b4d6752991094c96a38e2e3a18bc220a5e68b315a625236f5147d9d0924"}, + {"keyed", &Config{APIToken: "tk&\u2029", PreviewSecret: "p&s<", CheckpointTTLHours: 24, Mesh: &MeshConfig{ClusterKey: key}}, tenants, "ca<&>fp", "848ffb9bbcaa14b59074f0753cc2aff33f72494722059f27587d6fe61eb74ad2"}, + {"no tenants", &Config{}, nil, "fp", "4bf6489cb7c229d99dad45b7c8eb8a070a4fd7040cf43c1b49f103ae3107b10b"}, } { t.Run(tt.name, func(t *testing.T) { if got := tt.cfg.ClusterDigest(tt.fp, tenantRecords(tt.tenants)); got != tt.want { @@ -85,6 +86,45 @@ func TestClusterDigestBytesArePinned(t *testing.T) { } } +func TestClusterDigestCoversTheEgressLayerInAnyOrder(t *testing.T) { + key := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} + desk := EgressClass{Name: "desk", Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.example.com"}}}} + ops := EgressClass{Name: "ops", Egress: &egress.Policy{Allow: []egress.Rule{{Host: "ops.example.com"}}}} + base := func() *Config { + return &Config{ + EgressClasses: []EgressClass{desk, ops}, + Secrets: []egress.SecretSpec{{Name: "gh", Header: "Authorization", ValueEnv: "GH_A"}, {Name: "gw", Header: "X-Key"}}, + EgressInternalAllow: []string{"10.0.0.0/8", "172.16.0.0/12"}, + EgressUpstream: &EgressUpstreamConfig{ClaimEnv: "EGRESS_UPSTREAM", Allow: []string{"a.example", "b.example"}}, + Pools: []PoolSpec{{PoolKey: key, Warm: 2, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.example.com", Intercept: true}}}}}, + } + } + d := base().ClusterDigest("fp", nil) + same := base() + same.EgressClasses = []EgressClass{ops, desk} + same.Secrets = []egress.SecretSpec{{Name: "gw", Header: "X-Key"}, {Name: "gh", Header: "Authorization", ValueEnv: "GH_A"}} + same.EgressInternalAllow = []string{"172.16.0.0/12", "10.0.0.0/8"} + same.EgressUpstream.Allow = []string{"b.example", "a.example"} + same.Pools[0].Warm = 9 + if same.ClusterDigest("fp", nil) != d { + t.Error("order or a pool target changed the digest") + } + for name, change := range map[string]func(c *Config){ + "class rule": func(c *Config) { c.EgressClasses[0].Egress = &egress.Policy{} }, + "secret header": func(c *Config) { c.Secrets[0].Header = "X-Other" }, + "secret env": func(c *Config) { c.Secrets[0].ValueEnv = "GH_B" }, + "internal allow": func(c *Config) { c.EgressInternalAllow = nil }, + "upstream allow": func(c *Config) { c.EgressUpstream.Allow = []string{"a.example"} }, + "pool egress": func(c *Config) { c.Pools[0].Egress = &egress.Policy{} }, + } { + c := base() + change(c) + if c.ClusterDigest("fp", nil) == d { + t.Errorf("a %s change left the digest unchanged", name) + } + } +} + func TestLoadAppliesDefaults(t *testing.T) { path := writeConfig(t, `{"pools":[{"template":"rt:24.04","net":"none","size":"small"}]}`) cfg, err := Load(path) diff --git a/sandboxd/egress/bench_test.go b/sandboxd/egress/bench_test.go index b43daa11..a1c9d45f 100644 --- a/sandboxd/egress/bench_test.go +++ b/sandboxd/egress/bench_test.go @@ -47,6 +47,28 @@ func BenchmarkInterceptedHandshake(b *testing.B) { } } +func BenchmarkInterceptedConnection(b *testing.B) { + for _, arm := range []string{"fresh", "session-cache"} { + b.Run(arm, func(b *testing.B) { + addr, roots := benchFront(b, true) + var cache tls.ClientSessionCache + if arm == "session-cache" { + cache = tls.NewLRUClientSessionCache(8) + } + resumed := 0 + for b.Loop() { + tc := connectTLSCached(b, addr, "example.com:443", "example.com", roots, cache) + benchRoundTrip(b, tc, bufio.NewReader(tc)) + if tc.ConnectionState().DidResume { + resumed++ + } + _ = tc.Close() + } + b.ReportMetric(float64(resumed)/float64(b.N), "resumed/op") + }) + } +} + func BenchmarkSplicedHandshake(b *testing.B) { addr, roots := benchFront(b, false) for b.Loop() { diff --git a/sandboxd/egress/intercept.go b/sandboxd/egress/intercept.go index b05c53e0..8b09fc3d 100644 --- a/sandboxd/egress/intercept.go +++ b/sandboxd/egress/intercept.go @@ -32,6 +32,7 @@ func (p *Proxy) serveIntercept(w http.ResponseWriter, r *http.Request, host stri NextProtos: []string{"http/1.1"}, Certificates: []tls.Certificate{*leaf}, } + cfg.SetSessionTicketKeys(p.tickets) client, brw, err := http.NewResponseController(w).Hijack() if err != nil { http.Error(w, "egress: connection cannot be hijacked", http.StatusInternalServerError) diff --git a/sandboxd/egress/intercept_test.go b/sandboxd/egress/intercept_test.go index f4449c61..8acd09fd 100644 --- a/sandboxd/egress/intercept_test.go +++ b/sandboxd/egress/intercept_test.go @@ -246,21 +246,113 @@ func TestInterceptRejectsUntrustedUpstream(t *testing.T) { } } +func TestInterceptInjectsClaimCredentialsBelowThePoolSecret(t *testing.T) { + events := make(chan Event, 4) + secrets := credSecrets{fakeSecrets: fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}}, creds: map[string][]Credential{"example.com": { + {Name: "API", Header: "X-Api-Key", Value: "k1"}, + {Name: "AUTH", Header: "authorization", Value: "claim"}, + {Name: "EMPTY", Header: "X-Empty", Value: ""}, + {Name: "LATER", Header: "X-Api-Key", Value: "k2"}, + }}} + p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Secret: "gh", Intercept: true}}}, secrets, events) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + front := httptest.NewServer(p) + defer front.Close() + + tc := connectTLS(t, front.Listener.Addr().String(), "Example.com:443", "example.com", guestRoots) + defer func() { _ = tc.Close() }() + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "https://example.com/x", nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + req.Header.Set("X-Api-Key", "placeholder") + req.Header.Set("X-Empty", "guest") + if err = req.Write(tc); err != nil { + t.Fatalf("write request: %v", err) + } + resp, err := http.ReadResponse(bufio.NewReader(tc), req) + if err != nil { + t.Fatalf("read response: %v", err) + } + defer func() { _ = resp.Body.Close() }() + + if got := resp.Header.Get("X-Key-Seen"); got != "k1" { + t.Errorf("upstream saw X-Api-Key %q, want the first claim credential in order", got) + } + if got := resp.Header.Get("X-Auth-Seen"); got != "Bearer SECRET" { + t.Errorf("upstream saw Authorization %q, want the pool secret kept over the claim's", got) + } + if got := resp.Header.Get("X-Empty-Seen"); got != "guest" { + t.Errorf("upstream saw X-Empty %q, want the guest's value kept for an empty credential", got) + } + _ = recvEvent(t, events) + if ev := recvEvent(t, events); ev.Injected != "gh,claim:API" { + t.Errorf("audit Injected %q, want gh,claim:API", ev.Injected) + } +} + +func TestInterceptStreamsAnEventStreamAsItArrives(t *testing.T) { + release := make(chan struct{}) + upstream := httptest.NewTLSServer(sseHandler(release)) + defer upstream.Close() + defer close(release) + ca, _ := testCA(t) + p := New(Policy{Allow: []Rule{{Host: "example.com", Intercept: true}}}, nil, ca, fixedDial(upstream.Listener.Addr().String()), nil, nil) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + guestRoots := x509.NewCertPool() + guestRoots.AppendCertsFromPEM(ca.CertPEM()) + front := httptest.NewServer(p) + defer front.Close() + + tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) + defer func() { _ = tc.Close() }() + _ = tc.SetReadDeadline(time.Now().Add(2 * time.Second)) + resp := roundTripTLS(t, tc, http.MethodGet) + defer func() { _ = resp.Body.Close() }() + if line, err := bufio.NewReader(resp.Body).ReadString('\n'); err != nil || line != "data: one\n" { + t.Errorf("first event %q, %v; want it before the origin ends the stream", line, err) + } +} + +func TestInterceptResumesTheGuestsTLSSession(t *testing.T) { + p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Intercept: true}, nil) + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + front := httptest.NewServer(p) + defer front.Close() + cache := tls.NewLRUClientSessionCache(4) + for i := range 2 { + tc := connectTLSCached(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots, cache) + resp := roundTripTLS(t, tc, http.MethodGet) + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + if resumed := tc.ConnectionState().DidResume; resumed != (i == 1) { + t.Errorf("connection %d resumed=%v, want only the second to resume", i, resumed) + } + _ = tc.Close() + } +} + func interceptProxy(t *testing.T, rule Rule, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { return interceptProxyPolicy(t, Policy{Allow: []Rule{rule}}, events) } func interceptProxyPolicy(t *testing.T, policy Policy, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { + t.Helper() + return interceptProxySecrets(t, policy, fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}}, events) +} + +func interceptProxySecrets(t *testing.T, policy Policy, secrets Secrets, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { t.Helper() upstream := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("X-Auth-Seen", r.Header.Get("Authorization")) + w.Header().Set("X-Key-Seen", r.Header.Get("X-Api-Key")) + w.Header().Set("X-Empty-Seen", r.Header.Get("X-Empty")) w.Header().Set("X-Host-Seen", r.Host) _, _ = io.WriteString(w, "hello") })) t.Cleanup(upstream.Close) ca, _ := testCA(t) - secrets := fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}} audit := func(ev Event) { if events != nil { events <- ev @@ -282,6 +374,11 @@ func trustUpstream(upstream *httptest.Server) *x509.CertPool { } func connectTLS(tb testing.TB, proxyAddr, target, serverName string, roots *x509.CertPool) *tls.Conn { + tb.Helper() + return connectTLSCached(tb, proxyAddr, target, serverName, roots, nil) +} + +func connectTLSCached(tb testing.TB, proxyAddr, target, serverName string, roots *x509.CertPool, cache tls.ClientSessionCache) *tls.Conn { tb.Helper() conn, err := net.Dial("tcp", proxyAddr) if err != nil { @@ -293,7 +390,7 @@ func connectTLS(tb testing.TB, proxyAddr, target, serverName string, roots *x509 if status := readPreamble(tb, conn); !strings.Contains(status, "200") { tb.Fatalf("CONNECT status = %q, want 200", status) } - tc := tls.Client(conn, &tls.Config{ServerName: serverName, RootCAs: roots}) + tc := tls.Client(conn, &tls.Config{ServerName: serverName, RootCAs: roots, ClientSessionCache: cache}) if err := tc.Handshake(); err != nil { tb.Fatalf("guest tls handshake: %v", err) } diff --git a/sandboxd/egress/policy.go b/sandboxd/egress/policy.go index ff7dd257..81c69489 100644 --- a/sandboxd/egress/policy.go +++ b/sandboxd/egress/policy.go @@ -25,6 +25,16 @@ type Rule struct { Intercept bool `json:"intercept,omitzero"` } +// Covers reports whether the rule matches every host the pattern, exact or "*." suffix, matches. +func (r Rule) Covers(pattern string) bool { + suffix, wild := strings.CutPrefix(pattern, "*") + if !wild { + return r.matchHost(pattern) + } + rule, ruleWild := strings.CutPrefix(strings.ToLower(r.Host), "*") + return ruleWild && strings.HasSuffix(suffix, rule) +} + // matches expects host already lowercased by Eval. func (r Rule) matches(host, method string, port uint16) bool { return r.matchHost(host) && r.matchMethod(method) && r.matchPort(port) @@ -133,6 +143,11 @@ func (p Policy) ServesSocks() bool { return p.Socks5 } +// InterceptsHost reports whether an intercept rule covers the host pattern, so a claim credential for it can be injected. +func (p Policy) InterceptsHost(pattern string) bool { + return slices.ContainsFunc(p.Allow, func(r Rule) bool { return r.Intercept && r.Covers(pattern) }) +} + func (p Policy) admitsTunnel() bool { return slices.ContainsFunc(p.Allow, func(r Rule) bool { return !r.Intercept && r.matchMethod(http.MethodConnect) }) } @@ -143,6 +158,7 @@ type Evaluator interface { EvalHost(host string, port uint16) (Rule, Decision) EvalInner(host, method string, port uint16) (Rule, Decision) ServesSocks() bool + InterceptsHost(pattern string) bool } type composite struct { @@ -191,3 +207,7 @@ func (c composite) EvalInner(host, method string, port uint16) (Rule, Decision) func (c composite) ServesSocks() bool { return c.pool.ServesSocks() } + +func (c composite) InterceptsHost(pattern string) bool { + return c.pool.InterceptsHost(pattern) && slices.ContainsFunc(c.tenant.Allow, func(r Rule) bool { return r.Covers(pattern) }) +} diff --git a/sandboxd/egress/policy_test.go b/sandboxd/egress/policy_test.go index 9d6835db..2d6af474 100644 --- a/sandboxd/egress/policy_test.go +++ b/sandboxd/egress/policy_test.go @@ -95,6 +95,34 @@ func TestCompositeEvalInnerIntersectsTenant(t *testing.T) { } } +func TestInterceptsHostCoversPatterns(t *testing.T) { + pool := Policy{Allow: []Rule{ + {Host: "api.example.com", Intercept: true}, + {Host: "*.Corp.Test", Intercept: true}, + {Host: "plain.example.com"}, + }} + for pattern, want := range map[string]bool{ + "api.example.com": true, + "*.example.com": false, + "git.corp.test": true, + "*.corp.test": true, + "*.eu.corp.test": true, + "corp.test": false, + "plain.example.com": false, + } { + if got := pool.InterceptsHost(pattern); got != want { + t.Errorf("pool InterceptsHost(%q) = %v, want %v", pattern, got, want) + } + } + if !(Policy{Allow: []Rule{{Host: "*", Intercept: true}}}).InterceptsHost("*.anything.test") { + t.Error("a bare * intercept rule does not cover a wildcard pattern") + } + composed := Compose(pool, Policy{Allow: []Rule{{Host: "*.corp.test"}}}) + if composed.InterceptsHost("api.example.com") || !composed.InterceptsHost("git.corp.test") { + t.Error("a composed policy must need the pool to intercept and the tenant to allow the pattern") + } +} + func TestWildcardApexIsNotMatched(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "*.example.com"}}} if _, d := policy.Eval("example.com", "GET", 443); d != DecisionDeny { diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index 9ad1ba70..c3863c45 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -4,6 +4,7 @@ import ( "bufio" "cmp" "context" + "crypto/rand" "crypto/tls" "fmt" "io" @@ -29,11 +30,15 @@ const ( maxIdleConns = 64 ) -// hopHeaders are hop-by-hop and proxy-scoped headers this hop owns, never passed on. -var hopHeaders = []string{ - "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization", - "Proxy-Connection", "TE", "Trailer", "Transfer-Encoding", "Upgrade", -} +var ( + // hopHeaders are hop-by-hop and proxy-scoped headers this hop owns, never passed on. + hopHeaders = []string{ + "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization", + "Proxy-Connection", "TE", "Trailer", "Transfer-Encoding", "Upgrade", + } + + copyBufs = sync.Pool{New: func() any { b := make([]byte, 32<<10); return &b }} +) // TransferFunc receives the payload bytes an allowed tunnel or request moved, once it ends. type TransferFunc func(ev Event, sent, received int64) @@ -41,9 +46,17 @@ type TransferFunc func(ev Event, sent, received int64) // AuditFunc receives every egress decision the proxy takes. type AuditFunc func(Event) -// Secrets resolves a rule's Secret name to the header it injects. +// Credential is a claim-supplied header the proxy sets on an intercepted request. +type Credential struct { + Name string + Header string + Value string +} + +// Secrets resolves a rule's Secret name to the header it injects, and a host to the claim's own credentials for it. type Secrets interface { Header(name string) (header, value string, ok bool) + Credentials(host string) []Credential } // Holder is held for the life of every request and tunnel the proxy serves. @@ -52,7 +65,7 @@ type Holder interface { Unhold() } -// Event is one audited egress attempt; Injected names the credential and Upstream the proxy host, never their secrets. +// Event is one audited egress attempt; Injected lists the credentials set, comma-separated, and Upstream the proxy host, never their secrets. type Event struct { Method string Host string @@ -99,6 +112,8 @@ type Proxy struct { // leaves caches this sandbox's interception leaves; per-proxy, never shared. leafMu sync.Mutex leaves map[string]*tls.Certificate + // tickets lets a guest resume an intercepted TLS session on its next CONNECT. + tickets [][32]byte // conns tracks both halves of every tunnel and SOCKS5 connection, which http.Server.Close does not reach. connMu sync.Mutex @@ -126,6 +141,8 @@ func New(policy Evaluator, secrets Secrets, ca *CA, router Router, audit AuditFu pools.mitm = pools.tr.Clone() pools.mitm.TLSClientConfig = &tls.Config{MinVersion: tls.VersionTLS12} p.leaves = map[string]*tls.Certificate{} + p.tickets = make([][32]byte, 1) + _, _ = rand.Read(p.tickets[0][:]) } p.pools.Store(pools) return p @@ -307,7 +324,7 @@ func (p *Proxy) relay(w http.ResponseWriter, r *http.Request, ev Event, rule Rul out.RequestURI = "" out.Close = false // the guest's Connection: close is its own; the upstream pool keeps the conn stripHop(out.Header) - ev.Injected = p.inject(rule, out.Header) + ev.Injected = p.inject(rule, out.Header, ev.Host, mitm) var body *countingBody if p.transfer != nil && out.Body != nil && out.Body != http.NoBody { body = &countingBody{ReadCloser: out.Body} @@ -335,6 +352,10 @@ func (p *Proxy) relay(w http.ResponseWriter, r *http.Request, ev Event, rule Rul maps.Copy(w.Header(), resp.Header) stripHop(w.Header()) w.WriteHeader(resp.StatusCode) + if streamed(resp) { + received = copyFlushing(w, resp.Body) + return + } received, _ = io.Copy(w, resp.Body) } @@ -375,17 +396,29 @@ func (p *Proxy) reportTransfer(ev Event, sent, received int64) { } } -// inject sets the rule's secret header, overwriting any guest-supplied value. -func (p *Proxy) inject(rule Rule, h http.Header) string { - if rule.Secret == "" || p.secrets == nil { +// inject sets the rule's secret, then on an intercepted request each claim credential whose header is still unset; guest values are overwritten. +func (p *Proxy) inject(rule Rule, h http.Header, host string, mitm bool) string { + if p.secrets == nil { return "" } - header, value, ok := p.secrets.Header(rule.Secret) - if !ok || value == "" { - return "" + var names, set []string + if rule.Secret != "" { + if header, value, ok := p.secrets.Header(rule.Secret); ok && value != "" { + h.Set(header, value) + names, set = append(names, rule.Secret), append(set, header) + } + } + if !mitm { + return strings.Join(names, ",") } - h.Set(header, value) - return rule.Secret + for _, c := range p.secrets.Credentials(strings.ToLower(host)) { + if c.Value == "" || slices.ContainsFunc(set, func(s string) bool { return strings.EqualFold(s, c.Header) }) { + continue + } + h.Set(c.Header, c.Value) + names, set = append(names, "claim:"+c.Name), append(set, c.Header) + } + return strings.Join(names, ",") } func (p *Proxy) record(ev Event) { @@ -471,6 +504,35 @@ func spliceable(c net.Conn) net.Conn { return c } +func streamed(resp *http.Response) bool { + mediaType, _, _ := strings.Cut(resp.Header.Get("Content-Type"), ";") + return resp.ContentLength < 0 || strings.EqualFold(strings.TrimSpace(mediaType), "text/event-stream") +} + +func copyFlushing(w http.ResponseWriter, body io.Reader) int64 { + rc := http.NewResponseController(w) + if rc.Flush() != nil { + n, _ := io.Copy(w, body) + return n + } + buf := copyBufs.Get().(*[]byte) + defer copyBufs.Put(buf) + var total int64 + for { + n, err := body.Read(*buf) + if n > 0 { + written, werr := w.Write((*buf)[:n]) + total += int64(written) + if werr != nil || rc.Flush() != nil { + return total + } + } + if err != nil { + return total + } + } +} + func denied(w http.ResponseWriter, host string) { http.Error(w, fmt.Sprintf("egress denied: %s", host), http.StatusForbidden) } diff --git a/sandboxd/egress/proxy_test.go b/sandboxd/egress/proxy_test.go index fc1b9d2c..8b8b56fe 100644 --- a/sandboxd/egress/proxy_test.go +++ b/sandboxd/egress/proxy_test.go @@ -174,6 +174,57 @@ func TestForwardNeverInjectsInterceptSecret(t *testing.T) { } } +func TestForwardNeverInjectsClaimCredentials(t *testing.T) { + var seen string + upstream := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + seen = r.Header.Get("X-Api-Key") + })) + defer upstream.Close() + events := make(chan Event, 1) + secrets := credSecrets{creds: map[string][]Credential{"api.internal": {{Name: "API", Header: "X-Api-Key", Value: "k1"}}}} + p := New(Policy{Allow: []Rule{{Host: "api.internal"}}}, secrets, nil, fixedDial(upstream.Listener.Addr().String()), func(ev Event) { events <- ev }, nil) + front := httptest.NewServer(p) + defer front.Close() + + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + req.Header.Set("X-Api-Key", "placeholder") + resp, err := proxyClient(t, front.URL).Do(req) + if err != nil { + t.Fatalf("proxied GET: %v", err) + } + _ = resp.Body.Close() + if seen != "placeholder" { + t.Errorf("upstream saw X-Api-Key %q, want the guest's value: a claim credential never rides a cleartext forward", seen) + } + if ev := recvEvent(t, events); ev.Injected != "" { + t.Errorf("audit Injected %q, want none", ev.Injected) + } +} + +func TestForwardStreamsAnEventStreamAsItArrives(t *testing.T) { + release := make(chan struct{}) + upstream := httptest.NewServer(sseHandler(release)) + defer upstream.Close() + defer close(release) + p := New(Policy{Allow: []Rule{{Host: "sse.internal"}}}, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) + front := httptest.NewServer(p) + defer front.Close() + + client := proxyClient(t, front.URL) + client.Timeout = 2 * time.Second + resp, err := client.Get("http://sse.internal/") + if err != nil { + t.Fatalf("headers did not arrive before the origin ended the stream: %v", err) + } + defer func() { _ = resp.Body.Close() }() + if line, err := bufio.NewReader(resp.Body).ReadString('\n'); err != nil || line != "data: one\n" { + t.Errorf("first event %q, %v; want it before the origin ends the stream", line, err) + } +} + func TestForwardStripsHopHeaders(t *testing.T) { var gotHop, gotConn string upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { @@ -525,12 +576,34 @@ func (f fakeSecrets) Header(name string) (string, string, bool) { return hv[0], hv[1], ok } +func (f fakeSecrets) Credentials(string) []Credential { + return nil +} + +type credSecrets struct { + fakeSecrets + creds map[string][]Credential +} + +func (c credSecrets) Credentials(host string) []Credential { + return c.creds[host] +} + type holdCounter struct{ atomic.Int32 } func (h *holdCounter) Hold() { h.Add(1) } func (h *holdCounter) Unhold() { h.Add(-1) } +func sseHandler(release <-chan struct{}) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: one\n\n") + w.(http.Flusher).Flush() + <-release + }) +} + func fixedDial(target string) DialFunc { return func(ctx context.Context, network, _ string) (net.Conn, error) { var d net.Dialer diff --git a/sandboxd/egress/secrets.go b/sandboxd/egress/secrets.go index 07825936..a85ac958 100644 --- a/sandboxd/egress/secrets.go +++ b/sandboxd/egress/secrets.go @@ -28,12 +28,11 @@ func (s SecretSpec) Validate() error { return fmt.Errorf("secret name must not be empty") case s.Value != nil: return fmt.Errorf("secret %q: value is not supported, use value_env", s.Name) - case !httpguts.ValidHeaderFieldName(s.Header): - return fmt.Errorf("secret %q: header %q is not a valid header name", s.Name, s.Header) case s.ValueEnv == "" && !types.EnvNameRe.MatchString(s.Name): return fmt.Errorf("secret %q: without value_env the name is the claim env to read, so it must match %s", s.Name, types.EnvNameRe) - case slices.ContainsFunc(nonInjectable, func(h string) bool { return strings.EqualFold(h, s.Header) }): - return fmt.Errorf("secret %q: header %s is not injectable", s.Name, s.Header) + } + if err := CheckInjectHeader(s.Header); err != nil { + return fmt.Errorf("secret %q: %w", s.Name, err) } return nil } @@ -79,3 +78,14 @@ func (s *SecretStore) Header(name string) (header, value string, ok bool) { func (s *SecretStore) EnvName(name string) string { return s.byName[name].env } + +// CheckInjectHeader rejects a header name the proxy cannot set: an invalid one, or one the transport owns or strips. +func CheckInjectHeader(header string) error { + switch { + case !httpguts.ValidHeaderFieldName(header): + return fmt.Errorf("header %q is not a valid header name", header) + case slices.ContainsFunc(nonInjectable, func(h string) bool { return strings.EqualFold(h, header) }): + return fmt.Errorf("header %s is not injectable", header) + } + return nil +} diff --git a/sandboxd/main.go b/sandboxd/main.go index d569b138..5ad1513b 100644 --- a/sandboxd/main.go +++ b/sandboxd/main.go @@ -98,6 +98,7 @@ func main() { var placer server.Placer var prober server.CheckpointProber var probeKey []byte + var republish func() if cfg.Mesh != nil { msh, err := startMesh(ctx, cfg, mgr) if err != nil { @@ -108,9 +109,8 @@ func main() { mgr.SetTemplateNotifier(func() { msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes(), mgr.VolumeNames()) }) - mgr.OnTenants(func() { - msh.UpdateDigest(ctx, cfg.ClusterDigest(mgr.EgressCAFingerprint(), mgr.TenantRecords())) - }) + republish = func() { msh.UpdateDigest(ctx, mgr.ClusterDigest()) } + mgr.OnTenants(republish) clusterKey, err := cfg.Mesh.DecodedKey() if err != nil { logger.Fatalf(ctx, err, "decode mesh cluster key") @@ -138,7 +138,11 @@ func main() { if err != nil { return pool.ReloadResult{}, fmt.Errorf("%w: %w", pool.ErrBadConfig, err) } - return mgr.ReloadConfig(ctx, next) + res, err := mgr.ReloadConfig(ctx, next) + if err == nil && republish != nil { + republish() + } + return res, err } go reloadOnHangup(ctx, hangup, reload) srv := server.New(cfg.APIToken, cmp.Or(cfg.ClientAdvertise, cfg.AdvertiseAddr), mgr, eng, placer, prober, probeKey, preview) @@ -199,7 +203,7 @@ func startMesh(ctx context.Context, cfg *config.Config, mgr *pool.Manager) (*mes if err != nil { return nil, err } - self := mesh.NodeState{NodeID: cmp.Or(mc.NodeID, mc.Bind), Addr: cfg.AdvertiseAddr, ClientAddr: cfg.ClientAdvertise, Digest: cfg.ClusterDigest(mgr.EgressCAFingerprint(), mgr.TenantRecords())} + self := mesh.NodeState{NodeID: cmp.Or(mc.NodeID, mc.Bind), Addr: cfg.AdvertiseAddr, ClientAddr: cfg.ClientAdvertise, Digest: mgr.ClusterDigest()} msh, err := mesh.New(ctx, mlCfg, self, key, cfg.DataDir) if err != nil { return nil, err diff --git a/sandboxd/pool/checkpoint.go b/sandboxd/pool/checkpoint.go index 9f0352f1..ff730394 100644 --- a/sandboxd/pool/checkpoint.go +++ b/sandboxd/pool/checkpoint.go @@ -201,7 +201,7 @@ func (m *Manager) publishCheckpoint(ctx context.Context, sb *types.Sandbox, ckID // claimLoaded re-fetches under the record lock, so a delete racing the pre-check cannot slip in. func (m *Manager) claimLoaded(ctx context.Context, ckpt types.Checkpoint, o ClaimOptions) (*types.Sandbox, error) { - if err := m.checkUpstreamEnv(o.Env); err != nil { + if err := m.checkEnv(o.Env); err != nil { return nil, err } l := m.recLock(ckpt.ID) diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 4bff597d..a1b1302b 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -42,7 +42,8 @@ type ClaimOptions struct { // apply stamps the options a claim carries for its life. func (o ClaimOptions) apply(sb *types.Sandbox) { sb.Tenant, sb.ClaimRef, sb.Metadata, sb.OnExpire, sb.NoEgress = o.Tenant, o.ClaimRef, o.Metadata, o.OnExpire.Or(""), o.NoEgress - sb.EgressClass, sb.Env = o.EgressClass, o.Env + sb.EgressClass = o.EgressClass + sb.SetEnv(o.Env) } // ClaimWarm transfers ownership of a warm sandbox without provisioning; ErrNoWarm means empty. @@ -419,7 +420,10 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t } sb.TouchAt(now) } - err := m.deliverEnv(ctx, sbs, inherited) + err := m.checkInjects(m.view.Load(), sbs) + if err == nil { + err = m.deliverEnv(ctx, sbs, inherited) + } m.mu.Lock() if err == nil { err = m.quotaErr(len(sbs), sbs[0].Tenant) @@ -717,7 +721,7 @@ func (m *Manager) admitOptions(ctx context.Context, key types.PoolKey, o ClaimOp if err := archivable(key, len(o.Volumes) > 0, o.OnExpire); err != nil { return nil, nil, err } - if err := m.checkUpstreamEnv(o.Env); err != nil { + if err := m.checkEnv(o.Env); err != nil { return nil, nil, err } volumeSpecs, err := m.resolveVolumes(ctx, o.Tenant, o.Volumes) diff --git a/sandboxd/pool/egress.go b/sandboxd/pool/egress.go index ea816d73..4b20c991 100644 --- a/sandboxd/pool/egress.go +++ b/sandboxd/pool/egress.go @@ -322,13 +322,17 @@ func (m *Manager) disarmEgress(id string, removed bool) { // effectivePolicy resolves pool ∩ tenant; root has no tenant layer, an unpooled key no pool one, a NoEgress claim none at all. func (m *Manager) effectivePolicy(v *configView, sb *types.Sandbox) (egress.Evaluator, bool) { + m.mu.Lock() + defer m.mu.Unlock() + return m.effectivePolicyLocked(v, sb) +} + +func (m *Manager) effectivePolicyLocked(v *configView, sb *types.Sandbox) (egress.Evaluator, bool) { if sb.NoEgress { return nil, false } poolPol := v.poolEgress[sb.PolicyKey()] tenantPol := v.classEgress[sb.EgressClass] - m.mu.Lock() - defer m.mu.Unlock() if poolPol == nil { pooled := sb.Layer == types.LayerPooled if sb.Layer == "" { diff --git a/sandboxd/pool/egress_test.go b/sandboxd/pool/egress_test.go index 050b0865..d1cde37f 100644 --- a/sandboxd/pool/egress_test.go +++ b/sandboxd/pool/egress_test.go @@ -1074,10 +1074,15 @@ func egressClient(path string) *http.Client { } func egressManager(t *testing.T, eng *fakeEngine, pools ...config.PoolSpec) *Manager { + t.Helper() + return egressManagerAt(t, eng, t.TempDir(), pools...) +} + +func egressManagerAt(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { t.Helper() t.Setenv("GH_TOKEN", "s3cr3t") secrets := testSecrets(t, egress.SecretSpec{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}) - cfg := &config.Config{DataDir: t.TempDir(), Bridges: []string{"sbxbr0"}, EgressCA: writeTestEgressCA(t), Pools: pools} + cfg := &config.Config{DataDir: dataDir, Bridges: []string{"sbxbr0"}, EgressCA: writeTestEgressCA(t), Pools: pools} m, err := NewManager(t.Context(), cfg, eng, secrets) if err != nil { t.Fatalf("manager: %v", err) diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index d5ca6148..d5d39ffd 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "slices" + "strings" "golang.org/x/sync/errgroup" @@ -59,10 +60,18 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu m.mu.Unlock() return fmt.Errorf("%w: %w", ErrBadEnv, err) } - if err := m.checkUpstreamEnv(env); err != nil { + if err := m.checkEnv(env); err != nil { m.mu.Unlock() return err } + if fresh := env.InjectsChanged(prev); fresh != nil { + v := m.view.Load() + eval, ok := m.effectivePolicyLocked(v, sb) + if gap := v.injectGap(eval, ok, sb, fresh); len(gap) > 0 { + m.mu.Unlock() + return fmt.Errorf("%w: inject %s", ErrBadEnv, strings.Join(gap, ", ")) + } + } changed := !env.SameGuest(prev) paused := !locked || sb.HibernateSnap != "" || sb.PendingSnap != "" || sb.ArchiveCk != "" || failed if changed && paused { @@ -76,14 +85,14 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu m.mu.Unlock() return m.deliverGuestEnv(ctx, verb, id, sock, env, !paused) } - sb.Env = env + sb.SetEnv(env) js := m.store.set(sb) m.mu.Unlock() if err := m.store.commit(js); err != nil { m.mu.Lock() var rb claimSnapshot if m.claimed[id] == sb && sb.Env.Equal(env) { - sb.Env = prev + sb.SetEnv(prev) rb = m.store.set(sb) } m.mu.Unlock() @@ -124,6 +133,32 @@ func (m *Manager) deliverEnv(ctx context.Context, sbs []*types.Sandbox, inherite return group.Wait() } +// checkEnv admits an env this node can serve: its upstream entry, and inject headers the proxy may set. +func (m *Manager) checkEnv(env types.Env) error { + for name, v := range env { + if v.Inject == nil { + continue + } + if err := egress.CheckInjectHeader(v.Inject.Header); err != nil { + return fmt.Errorf("%w: env %s: inject %w", ErrBadEnv, name, err) + } + } + return m.checkUpstreamEnv(env) +} + +func (m *Manager) checkInjects(v *configView, sbs []*types.Sandbox) error { + for _, sb := range sbs { + if !sb.Env.HasInject() { + continue + } + eval, ok := m.effectivePolicy(v, sb) + if gap := v.injectGap(eval, ok, sb, sb.Env); len(gap) > 0 { + return fmt.Errorf("%w: inject %s", ErrBadEnv, strings.Join(gap, ", ")) + } + } + return nil +} + // heldGuestEnv reports whether sb's guest holds an env file; the caller holds sb.Transition, so no guest write races its capture. func (m *Manager) heldGuestEnv(sb *types.Sandbox) bool { m.mu.Lock() @@ -144,12 +179,16 @@ func (c claimSecrets) Header(name string) (header, value string, ok bool) { if header, value, ok = secrets.Header(name); !ok { return "", "", false } - env := secrets.EnvName(name) - c.m.mu.Lock() - own, set := c.sb.Env.Hidden(env) - c.m.mu.Unlock() - if set { + if own, set := c.sb.HiddenEnv(secrets.EnvName(name)); set { value = own } return header, value, true } + +func (c claimSecrets) Credentials(host string) []egress.Credential { + var out []egress.Credential + for name, v := range c.sb.Injections(host) { + out = append(out, egress.Credential{Name: name, Header: v.Inject.Header, Value: v.Value}) + } + return out +} diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index 8fd0b4a6..9222e4c0 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -450,6 +450,150 @@ func TestPatchEnvMergesAndRepairsTheGuest(t *testing.T) { } } +func TestAnInjectNeedsAHostThePoolIntercepts(t *testing.T) { + pol := &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Secret: "gh", Intercept: true}, {Host: "*.corp.test", Intercept: true}, {Host: "plain.test"}}} + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: pol}) + sb := mustClaim(t, m, testKey) + for name, patch := range map[string]types.EnvPatch{ + "a plain host": injectPatch("X-Api-Key", "plain.test"), + "an unlisted host": injectPatch("X-Api-Key", "other.test"), + "the pool secret's header": injectPatch("authorization", "api.github.com"), + "a transport header": injectPatch("Content-Length", "git.corp.test"), + } { + if err := m.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, ErrBadEnv) { + t.Errorf("%s: %v, want ErrBadEnv", name, err) + } + } + if env, _ := m.Env(sb.ID, ""); env != nil { + t.Errorf("a refused inject left env %v", env) + } + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com", "*.corp.test"), ""); err != nil { + t.Fatalf("an intercepted inject: %v", err) + } + if got := credentialsFor(m, sb, "git.corp.test"); got != "KEY:X-Api-Key=k" { + t.Errorf("credentials for git.corp.test %q, want KEY:X-Api-Key=k", got) + } +} + +func TestAnInjectNeedsTheTenantClassToAllowTheHost(t *testing.T) { + pol := &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: true}, {Host: "*.corp.test", Intercept: true}}} + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: pol}) + editView(m, func(v *configView) { + v.classEgress = map[string]*egress.Policy{"desk": {Allow: []egress.Rule{{Host: "*.corp.test"}}}} + }) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "acme", EgressClass: "desk"}) + if err != nil { + t.Fatalf("claim: %v", err) + } + if err = m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com"), "acme"); !errors.Is(err, ErrBadEnv) { + t.Errorf("a host the class denies: %v, want ErrBadEnv", err) + } + if err = m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "git.corp.test"), "acme"); err != nil { + t.Errorf("a host the pool intercepts and the class allows: %v", err) + } +} + +func TestAClaimWithAnUninterceptedInjectIsRefused(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"other.test"}, Header: "X-Api-Key"}}} + if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}); !errors.Is(err, ErrBadEnv) { + t.Fatalf("claim: %v, want ErrBadEnv", err) + } + m.mu.Lock() + claimed := len(m.claimed) + m.mu.Unlock() + if claimed != 0 || len(eng.removedNames()) != 1 { + t.Errorf("%d claims left and %v removed, want none kept and the VM destroyed", claimed, eng.removedNames()) + } +} + +func TestEachClaimInjectsItsOwnCredentialAndAHostOnlyChangeNeedsNoGuest(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + claim := func(value string) *types.Sandbox { + t.Helper() + env := types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) + if err != nil { + t.Fatalf("claim: %v", err) + } + return sb + } + a, b, plain := claim("a"), claim("b"), mustClaim(t, m, testKey) + if got := credentialsFor(m, a, "api.github.com") + "|" + credentialsFor(m, b, "api.github.com") + "|" + credentialsFor(m, plain, "api.github.com"); got != "KEY:X-Api-Key=a|KEY:X-Api-Key=b|" { + t.Errorf("credentials %q, want each claim's own and none for the plain claim", got) + } + m.mu.Lock() + a.HibernateSnap = "snap" + m.mu.Unlock() + if err := m.PatchEnv(t.Context(), a.ID, types.EnvPatch{"KEY": {Value: "a2", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}}, ""); err != nil { + t.Fatalf("rotate on a hibernated claim: %v", err) + } + if got, wrote := eng.guestEnvs[a.VsockSocket]; wrote || credentialsFor(m, a, "api.github.com") != "KEY:X-Api-Key=a2" { + t.Errorf("guest file %q (written %v), credentials %q; want no guest write and the rotated value", got, wrote, credentialsFor(m, a, "api.github.com")) + } + if err := m.SetEnv(t.Context(), b.ID, types.Env{}, ""); err != nil || credentialsFor(m, b, "api.github.com") != "" { + t.Errorf("clear: %v, credentials %q; want none", err, credentialsFor(m, b, "api.github.com")) + } +} + +func TestInjectsSurviveARestartAndAReloadThatDropsTheInterceptStopsThem(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + dataDir := t.TempDir() + pool := config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: true}}}} + m := egressManagerAt(t, eng, dataDir, pool) + env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) + if err != nil { + t.Fatalf("claim: %v", err) + } + m2 := egressManagerAt(t, eng, dataDir, pool) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + if got := credentialsFor(m2, restored, "api.github.com"); got != "KEY:X-Api-Key=k" { + t.Errorf("credentials after a restart %q, want the persisted inject", got) + } + editView(m2, func(v *configView) { + v.poolEgress[testKey] = &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com"}}} + }) + v := m2.view.Load() + eval, ok := m2.effectivePolicy(v, restored) + if gap := v.injectGap(eval, ok, restored, env); len(gap) != 1 { + t.Errorf("gap after the intercept rule went %v, want the inject named", gap) + } +} + +func TestALostInterceptRefusesOnlyAChangedInject(t *testing.T) { + eng := newFakeEngine() + eng.sockRoot = sockRoot(t) + m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) + sb := mustClaim(t, m, testKey) + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Api-Key", "api.github.com"), ""); err != nil { + t.Fatalf("inject: %v", err) + } + editView(m, func(v *configView) { + v.poolEgress[testKey] = &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com"}}} + }) + if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"OTHER": {Value: "o", Guest: hostOnly}}, ""); err != nil { + t.Errorf("an unrelated patch after the intercept went: %v", err) + } + if err := m.PatchEnv(t.Context(), sb.ID, injectPatch("X-Other", "api.github.com"), ""); !errors.Is(err, ErrBadEnv) { + t.Errorf("a changed inject after the intercept went: %v, want ErrBadEnv", err) + } +} + func BenchmarkClaimSecretsHeader(b *testing.B) { b.Setenv("GH_TOKEN", "node-gh") store, err := egress.NewSecretStore([]egress.SecretSpec{{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}}) @@ -466,14 +610,38 @@ func BenchmarkClaimSecretsHeader(b *testing.B) { for _, arm := range []string{"no-claim-env", "claim-env"} { sb := &types.Sandbox{} if arm == "claim-env" { - sb.Env = types.Env{"GH_TOKEN": {Value: "Bearer claim", Guest: hostOnly}, "MODE": {Value: "on"}} + sb.SetEnv(types.Env{"GH_TOKEN": {Value: "Bearer claim", Guest: hostOnly}, "MODE": {Value: "on"}}) } b.Run(arm, func(b *testing.B) { for b.Loop() { _, _, _ = claimSecrets{m: m, sb: sb}.Header("gh") } }) + b.Run(arm+"/credentials", func(b *testing.B) { + for b.Loop() { + _ = claimSecrets{m: m, sb: sb}.Credentials("api.example.com") + } + }) + } + sb := &types.Sandbox{} + sb.SetEnv(types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.example.com"}, Header: "X-Key"}}}) + b.Run("inject/credentials", func(b *testing.B) { + for b.Loop() { + _ = claimSecrets{m: m, sb: sb}.Credentials("api.example.com") + } + }) +} + +func injectPatch(header string, hosts ...string) types.EnvPatch { + return types.EnvPatch{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: header}}} +} + +func credentialsFor(m *Manager, sb *types.Sandbox, host string) string { + var out []string + for _, c := range (claimSecrets{m: m, sb: sb}).Credentials(host) { + out = append(out, c.Name+":"+c.Header+"="+c.Value) } + return strings.Join(out, ",") } func envManager(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 4f68884e..d1f7248c 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -503,6 +503,14 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg return m, nil } +// ClusterDigest fingerprints this node's must-match config as last loaded, with its egress root and tenant set. +func (m *Manager) ClusterDigest() string { + m.reloadMu.Lock() + cfg := m.cfg + m.reloadMu.Unlock() + return cfg.ClusterDigest(m.EgressCAFingerprint(), m.TenantRecords()) +} + // EgressCAFingerprint is the egress root's fingerprint, or "" when the node intercepts nothing. func (m *Manager) EgressCAFingerprint() string { if m.egressCA == nil { diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index 728894d1..22e826c9 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -39,6 +39,7 @@ func (m *Manager) Reconcile(ctx context.Context) error { var adopted []*types.Sandbox m.mu.Lock() for id, sb := range claims { + sb.SetEnv(sb.Env) rec, ok := live[sb.VMName] switch { case sb.ArchiveCk != "": diff --git a/sandboxd/pool/reload.go b/sandboxd/pool/reload.go index 6dad5d1f..5aed63a9 100644 --- a/sandboxd/pool/reload.go +++ b/sandboxd/pool/reload.go @@ -43,9 +43,15 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload } var trim []string var doors []*egressListener + var injecting []*types.Sandbox m.mu.Lock() m.view.Store(view) live := slices.Collect(maps.Values(m.egressListeners)) + for _, sb := range m.claimed { + if sb.Env.HasInject() { + injecting = append(injecting, sb) + } + } if !view.guardedEgress { doors = slices.Collect(maps.Values(m.egressPrebound)) clear(m.egressPrebound) @@ -68,10 +74,23 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload m.recordAudit(ctx, "", auditFrame{Op: "config_reload", Changed: changed}) log.WithFunc("pool.ReloadConfig").Infof(ctx, "config reloaded: %s; %d warm VMs of retired goldens destroyed", strings.Join(changed, ", "), len(trim)) m.warnMissingClasses(ctx) + m.warnLostInjects(ctx, view, injecting) m.kickRefill() return res, nil } +func (m *Manager) warnLostInjects(ctx context.Context, v *configView, sbs []*types.Sandbox) { + for _, sb := range sbs { + m.mu.Lock() + env := sb.Env + m.mu.Unlock() + eval, ok := m.effectivePolicy(v, sb) + if gap := v.injectGap(eval, ok, sb, env); len(gap) > 0 { + log.WithFunc("pool.warnLostInjects").Warnf(ctx, "claim %s: inject no longer intercepted: %s", sb.ID, strings.Join(gap, ", ")) + } + } +} + // refuseInterceptOn turns intercept on only for a key whose guests can trust the CA: one the node has not served, with the CA loaded at boot. func (m *Manager) refuseInterceptOn(old, next *configView) error { for _, key := range slices.SortedFunc(maps.Keys(next.poolEgress), func(a, b types.PoolKey) int { return strings.Compare(a.Hash(), b.Hash()) }) { diff --git a/sandboxd/pool/reload_test.go b/sandboxd/pool/reload_test.go index a47859d3..0f84c16b 100644 --- a/sandboxd/pool/reload_test.go +++ b/sandboxd/pool/reload_test.go @@ -276,6 +276,19 @@ func TestReloadIsAudited(t *testing.T) { } } +func TestAReloadOfTheEgressLayerChangesTheClusterDigest(t *testing.T) { + m := newTestManager(t, newFakeEngine()) + before := m.ClusterDigest() + next := *m.cfg + next.EgressInternalAllow = []string{"10.8.0.0/16"} + if _, err := m.ReloadConfig(t.Context(), &next); err != nil { + t.Fatalf("reload: %v", err) + } + if m.ClusterDigest() == before { + t.Error("the digest still covers the boot config after an egress reload") + } +} + func TestAnInterruptedReloadStillDestroysTheRetiredWarmVMs(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) diff --git a/sandboxd/pool/upstream.go b/sandboxd/pool/upstream.go index 28f31b2b..93576e32 100644 --- a/sandboxd/pool/upstream.go +++ b/sandboxd/pool/upstream.go @@ -28,10 +28,7 @@ func (r claimRouter) Route() string { if view.upstreamEnv == "" { return "" } - r.m.mu.Lock() - v, set := r.sb.Env.Hidden(view.upstreamEnv) - r.m.mu.Unlock() - if set { + if v, set := r.sb.HiddenEnv(view.upstreamEnv); set { if v == upstreamDirect { return "" } diff --git a/sandboxd/pool/upstream_test.go b/sandboxd/pool/upstream_test.go index b07e2c58..4e8ad862 100644 --- a/sandboxd/pool/upstream_test.go +++ b/sandboxd/pool/upstream_test.go @@ -35,7 +35,8 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { {"claim value", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "socks5://127.0.0.1:3003", Guest: hostOnly}}, "socks5://127.0.0.1:3003"}, {"claim direct", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "direct", Guest: hostOnly}}, ""}, } { - sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tc.class, Env: tc.env} + sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tc.class} + sb.SetEnv(tc.env) if got := (claimRouter{m: m, sb: sb}).Route(); got != tc.want { t.Errorf("%s: route %q, want %q", tc.name, got, tc.want) } @@ -51,7 +52,7 @@ func TestClaimsOnOnePoolLeaveThroughTheirOwnUpstreams(t *testing.T) { claim := func(upstream string) *types.Sandbox { sb := &types.Sandbox{ID: "sb_" + upstream, Key: testKey} if upstream != "" { - sb.Env = types.Env{"EGRESS_UPSTREAM": {Value: "http://user:pw@" + upstream, Guest: hostOnly}} + sb.SetEnv(types.Env{"EGRESS_UPSTREAM": {Value: "http://user:pw@" + upstream, Guest: hostOnly}}) } return sb } @@ -82,7 +83,9 @@ func TestATenantClaimUnderAClassDialsItsOwnUpstreamLikeARootClaim(t *testing.T) own, classDefault := tunnelProxy(t), tunnelProxy(t) editView(m, func(v *configView) { v.classUpstream["desk"] = "http://" + classDefault.addr }) tenantClaim := func(upstream string) *types.Sandbox { - return &types.Sandbox{ID: "sb_desk", Key: testKey, Tenant: "acme", EgressClass: "desk", Env: types.Env{"EGRESS_UPSTREAM": {Value: upstream, Guest: hostOnly}}} + sb := &types.Sandbox{ID: "sb_desk", Key: testKey, Tenant: "acme", EgressClass: "desk"} + sb.SetEnv(types.Env{"EGRESS_UPSTREAM": {Value: upstream, Guest: hostOnly}}) + return sb } r := claimRouter{m: m, sb: tenantClaim("http://user:pw@" + own.addr)} conn, err := r.Dial(t.Context(), r.Route(), "tcp", origin) diff --git a/sandboxd/pool/view.go b/sandboxd/pool/view.go index 2fd54cfc..9e630200 100644 --- a/sandboxd/pool/view.go +++ b/sandboxd/pool/view.go @@ -1,7 +1,11 @@ package pool import ( + "fmt" + "maps" "os" + "slices" + "strings" "sync/atomic" "github.com/cocoonstack/sandbox/sandboxd/config" @@ -73,6 +77,39 @@ func newConfigView(cfg *config.Config, secrets *egress.SecretStore) *configView return v } +// injectGap names, in env order, each inject host sb's egress does not intercept and each header a covering pool secret already sets. +func (v *configView) injectGap(eval egress.Evaluator, ok bool, sb *types.Sandbox, env types.Env) []string { + var gap []string + pool := v.poolEgress[sb.PolicyKey()] + for _, name := range slices.Sorted(maps.Keys(env)) { + inj := env[name].Inject + if inj == nil { + continue + } + for _, h := range inj.Hosts { + if !ok || !eval.InterceptsHost(h) { + gap = append(gap, fmt.Sprintf("%s: %s is not intercepted", name, h)) + } else if v.secretSets(pool, h, inj.Header) { + gap = append(gap, fmt.Sprintf("%s: the pool's secret already sets %s on %s", name, inj.Header, h)) + } + } + } + return gap +} + +func (v *configView) secretSets(pool *egress.Policy, host, header string) bool { + if pool == nil { + return false + } + return slices.ContainsFunc(pool.Allow, func(r egress.Rule) bool { + if !r.Intercept || r.Secret == "" || !r.Covers(host) { + return false + } + h, _, known := v.secrets.Header(r.Secret) + return known && strings.EqualFold(h, header) + }) +} + type resolvedPolicy struct { view *configView eval egress.Evaluator @@ -109,6 +146,10 @@ func (l *livePolicy) ServesSocks() bool { return l.current().ServesSocks() } +func (l *livePolicy) InterceptsHost(pattern string) bool { + return l.current().InterceptsHost(pattern) +} + // current denies everything once a reload leaves the claim with no policy. func (l *livePolicy) current() egress.Evaluator { v := l.m.view.Load() diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index 9d7ad313..b202f8c4 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -83,6 +83,8 @@ func TestEnvVarDecodesStrictly(t *testing.T) { {`{"value":"s","host_only":true}`, false, false}, {`{"value":"s","guest":null}`, false, false}, {`{"value":"s","guest":"false"}`, false, false}, + {`{"value":"s","guest":false,"inject":{"hosts":["a.test"],"header":"X-Key"}}`, false, true}, + {`{"value":"s","guest":false,"inject":{"hosts":["a.test"],"heder":"X-Key"}}`, false, false}, } { var v EnvVar err := json.Unmarshal([]byte(tt.body), &v) @@ -92,6 +94,79 @@ func TestEnvVarDecodesStrictly(t *testing.T) { } } +func TestEnvInjectValidates(t *testing.T) { + inject := func(header string, hosts ...string) Env { + return Env{"K": {Value: "v", Guest: new(false), Inject: &EnvInject{Hosts: hosts, Header: header}}} + } + tests := []struct { + name string + env Env + want string + }{ + {"host-only exact and wildcard", inject("X-Api-Key", "api.example.com", "*.corp.test", "10.0.0.5"), ""}, + {"guest entry", Env{"K": {Value: "v", Inject: &EnvInject{Hosts: []string{"a.test"}, Header: "X-Key"}}}, "needs guest: false"}, + {"no hosts", inject("X-Key"), "1 to 8 hosts"}, + {"too many hosts", inject("X-Key", "a", "b", "c", "d", "e", "f", "g", "h", "i"), "1 to 8 hosts"}, + {"uppercase host", inject("X-Key", "API.example.com"), "lowercase host name"}, + {"bare wildcard", inject("X-Key", "*"), "lowercase host name"}, + {"host with a port", inject("X-Key", "a.test:443"), "lowercase host name"}, + {"bad header", inject("X Key", "a.test"), "not a valid header name"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := tt.env.Validate() + if (tt.want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), tt.want)) { + t.Errorf("Validate = %v, want %q", err, tt.want) + } + }) + } +} + +func TestEnvInjectMatchesAndCompares(t *testing.T) { + inj := &EnvInject{Hosts: []string{"api.example.com", "*.corp.test"}, Header: "X-Key"} + for host, want := range map[string]bool{"api.example.com": true, "git.corp.test": true, "corp.test": false, "example.com": false} { + if got := inj.Matches(host); got != want { + t.Errorf("Matches(%q) = %v, want %v", host, got, want) + } + } + a := Env{"K": {Value: "v", Guest: new(false), Inject: inj}} + b := Env{"K": {Value: "v", Guest: new(false), Inject: &EnvInject{Hosts: []string{"api.example.com"}, Header: "X-Key"}}} + if a.Equal(b) || !a.Equal(Env{"K": {Value: "v", Guest: new(false), Inject: inj}}) || !a.SameGuest(b) { + t.Error("Equal must see an inject change, and SameGuest must not") + } + if red := a.Redacted(); red["K"].Value != "" || red["K"].Inject != inj { + t.Errorf("Redacted = %+v, want the value dropped and the inject kept", red["K"]) + } +} + +func TestSandboxPublishesItsEnvToTheProxy(t *testing.T) { + sb := &Sandbox{} + if _, ok := sb.HiddenEnv("K"); ok { + t.Fatal("a sandbox without SetEnv served a hidden value") + } + to := func(hosts ...string) *EnvInject { return &EnvInject{Hosts: hosts, Header: "X-Key"} } + sb.SetEnv(Env{ + "B": {Value: "b", Guest: new(false), Inject: to("a.test")}, + "A": {Value: "a", Guest: new(false), Inject: to("*.test")}, + "OTHER": {Value: "o", Guest: new(false), Inject: to("b.example")}, + "PLAIN": {Value: "p", Guest: new(false)}, + }) + var names []string + for name := range sb.Injections("a.test") { + names = append(names, name) + } + if strings.Join(names, ",") != "A,B" { + t.Errorf("Injections(a.test) = %v, want A,B in name order", names) + } + if v, ok := sb.HiddenEnv("PLAIN"); !ok || v != "p" { + t.Errorf("HiddenEnv(PLAIN) = %q %v, want p", v, ok) + } + sb.SetEnv(nil) + if _, ok := sb.HiddenEnv("PLAIN"); ok { + t.Error("a cleared env still serves a hidden value") + } +} + func TestEnvSplitsGuestAndHostOnlyEntries(t *testing.T) { env := Env{"G": {Value: "g"}, "H": {Value: "h", Guest: new(false)}, "T": {Value: "t", Guest: new(true)}} if v, ok := env.Hidden("H"); !ok || v != "h" { diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index fd51cb06..a56b7f70 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -9,6 +9,7 @@ import ( "encoding/json/v2" "errors" "fmt" + "iter" "maps" "path/filepath" "regexp" @@ -69,6 +70,7 @@ const ( maxEnvVars = 64 maxEnvValueBytes = 8 << 10 maxEnvBytes = 64 << 10 + maxInjectHosts = 8 ) var ( @@ -78,6 +80,8 @@ var ( VolumeNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,19}$`) // EnvNameRe is the portable shell variable name. EnvNameRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]{0,127}$`) + // injectHostRe is a lowercase DNS name or IPv4 literal, optionally under a "*." wildcard. + injectHostRe = regexp.MustCompile(`^(\*\.)?([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)*[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$`) // guestFileEscaper covers the characters systemd unescapes inside a double-quoted EnvironmentFile value. guestFileEscaper = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "`", "\\`", `$`, `\$`) @@ -215,10 +219,58 @@ func (md Metadata) encodedSize() int { return size } +// EnvInject sends a host-only entry's value as Header on intercepted requests to Hosts, each exact or "*." suffix. +type EnvInject struct { + Hosts []string `json:"hosts"` + Header string `json:"header"` +} + +// Matches reports whether the lowercased host is one of the inject hosts. +func (i *EnvInject) Matches(host string) bool { + return slices.ContainsFunc(i.Hosts, func(h string) bool { + if suffix, wild := strings.CutPrefix(h, "*"); wild { + return strings.HasSuffix(host, suffix) + } + return host == h + }) +} + +func (i *EnvInject) validate(name string) error { + if len(i.Hosts) == 0 || len(i.Hosts) > maxInjectHosts { + return fmt.Errorf("env %s: inject needs 1 to %d hosts, got %d", name, maxInjectHosts, len(i.Hosts)) + } + if !httpguts.ValidHeaderFieldName(i.Header) { + return fmt.Errorf("env %s: inject header %q is not a valid header name", name, i.Header) + } + for _, h := range i.Hosts { + if len(h) > 253 || !injectHostRe.MatchString(h) { + return fmt.Errorf("env %s: inject host %q must be a lowercase host name or *.suffix", name, h) + } + } + return nil +} + +func (i *EnvInject) size() int { + n := len(i.Header) + for _, h := range i.Hosts { + n += len(h) + } + return n +} + +func (i *EnvInject) equal(other *EnvInject) bool { + if i == nil || other == nil { + return i == other + } + return i.Header == other.Header && slices.Equal(i.Hosts, other.Hosts) +} + // EnvVar is one entry of a claim's environment; a nil Guest means the entry reaches the guest. type EnvVar struct { Value string `json:"value"` Guest *bool `json:"guest,omitempty"` + // Inject, on a host-only entry, makes the egress proxy send Value to the named hosts. + Inject *EnvInject `json:"inject,omitempty"` } // InGuest reports whether the entry is delivered into the guest. @@ -227,13 +279,14 @@ func (v EnvVar) InGuest() bool { return v.Guest == nil || *v.Guest } // UnmarshalJSON rejects unknown members and a null guest, so a mistyped host-only flag never falls back to the guest. func (v *EnvVar) UnmarshalJSON(b []byte) error { var raw struct { - Value string `json:"value"` - Guest jsontext.Value `json:"guest,omitzero"` + Value string `json:"value"` + Guest jsontext.Value `json:"guest,omitzero"` + Inject *EnvInject `json:"inject,omitempty"` } if err := json.Unmarshal(b, &raw, json.RejectUnknownMembers(true)); err != nil { return fmt.Errorf("env entry: %w", err) } - *v = EnvVar{Value: raw.Value} + *v = EnvVar{Value: raw.Value, Inject: raw.Inject} if raw.Guest == nil { return nil } @@ -246,7 +299,7 @@ func (v *EnvVar) UnmarshalJSON(b []byte) error { } func (v EnvVar) same(other EnvVar) bool { - return v.Value == other.Value && v.InGuest() == other.InGuest() + return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Inject.equal(other.Inject) } // Env is a claim's own environment: guest entries reach the guest, the others stay host-side for egress secrets. @@ -269,9 +322,19 @@ func (e Env) Validate() error { if !httpguts.ValidHeaderFieldValue(v.Value) { return fmt.Errorf("env value of %s must not hold control characters other than tab", name) } + if v.Inject == nil { + continue + } + if v.InGuest() { + return fmt.Errorf("env %s: inject needs guest: false, or the guest holds the value", name) + } + if err := v.Inject.validate(name); err != nil { + return err + } + total += v.Inject.size() } if total > maxEnvBytes { - return fmt.Errorf("env names and values must total at most %d bytes, got %d", maxEnvBytes, total) + return fmt.Errorf("env names, values and injects must total at most %d bytes, got %d", maxEnvBytes, total) } return nil } @@ -285,6 +348,30 @@ func (e Env) Hidden(name string) (string, bool) { return v.Value, true } +// HasInject reports whether any entry injects into egress requests. +func (e Env) HasInject() bool { + for _, v := range e { + if v.Inject != nil { + return true + } + } + return false +} + +// InjectsChanged returns the entries of e that inject and that prev lacks or holds differently. +func (e Env) InjectsChanged(prev Env) Env { + var out Env + for name, v := range e { + if old, ok := prev[name]; v.Inject != nil && (!ok || !v.same(old)) { + if out == nil { + out = Env{} + } + out[name] = v + } + } + return out +} + // HasGuest reports whether any entry reaches the guest. func (e Env) HasGuest() bool { for _, v := range e { @@ -310,7 +397,8 @@ func (e Env) Redacted() Env { out := maps.Clone(e) for name, v := range out { if !v.InGuest() { - out[name] = EnvVar{Guest: v.Guest} + v.Value = "" + out[name] = v } } return out @@ -410,6 +498,12 @@ func (k PoolKey) Validate() error { return nil } +// envView is an immutable env with its inject entries named in order. +type envView struct { + env Env + injects []string +} + // Sandbox is the node-local record of one pooled or claimed VM. type Sandbox struct { ID string `json:"id"` @@ -441,7 +535,7 @@ type Sandbox struct { OnExpire ExpireAction `json:"on_expire,omitempty"` // Volumes records the volumes successfully applied to this claim. Volumes []Volume `json:"volumes,omitempty"` - // Env is replaced whole under the manager mutex; host-only values are never served back. + // Env is replaced whole through SetEnv under the manager mutex; host-only values are never served back. Env Env `json:"env,omitempty"` VsockSocket string `json:"vsock_socket,omitempty"` @@ -476,6 +570,8 @@ type Sandbox struct { lastActivity atomic.Int64 // open counts the data-plane connections held right now. open atomic.Int32 + // live is the env as of the last SetEnv, read by the egress proxy without the manager mutex. + live atomic.Pointer[envView] // Transition serializes hibernate/wake; lock it before (never under) the manager mutex. Transition sync.Mutex `json:"-"` @@ -505,6 +601,42 @@ func (s *Sandbox) UnholdIdle() { s.open.Add(-1) } // Busy reports whether a data-plane connection is held. func (s *Sandbox) Busy() bool { return s.open.Load() > 0 } +// SetEnv replaces the env and publishes it to the egress proxy. +func (s *Sandbox) SetEnv(e Env) { + s.Env = e + view := &envView{env: e} + for name, v := range e { + if v.Inject != nil && !v.InGuest() { + view.injects = append(view.injects, name) + } + } + slices.Sort(view.injects) + s.live.Store(view) +} + +// HiddenEnv returns a host-only entry's value as last published. +func (s *Sandbox) HiddenEnv(name string) (string, bool) { + if view := s.live.Load(); view != nil { + return view.env.Hidden(name) + } + return "", false +} + +// Injections yields, in name order, the published host-only entries that inject into host. +func (s *Sandbox) Injections(host string) iter.Seq2[string, EnvVar] { + return func(yield func(string, EnvVar) bool) { + view := s.live.Load() + if view == nil { + return + } + for _, name := range view.injects { + if v := view.env[name]; v.Inject.Matches(host) && !yield(name, v) { + return + } + } + } +} + // PolicyKey is the pool key whose egress policy applies: PolicySource when set, else Key. func (s *Sandbox) PolicyKey() PoolKey { return cmp.Or(s.PolicySource, s.Key) } diff --git a/scripts/credentials-e2e.sh b/scripts/credentials-e2e.sh new file mode 100755 index 00000000..eaf14822 --- /dev/null +++ b/scripts/credentials-e2e.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Bare-metal acceptance for claim credentials: claims on one intercept pool each +# send their own env-injected header to an HTTPS echo, the guest never holds the +# value, an open event stream reaches the guest event by event, and a credential +# survives a sandboxd restart. Runs a dedicated sandboxd on $ADDR. +# Needs outbound to $ECHO and a $TEMPLATE whose silkd binds the egress relay. +set -euo pipefail + +TEMPLATE=${TEMPLATE:-rt:24.04} +ADDR=${ADDR:-127.0.0.1:7882} +TOKEN=${TOKEN:-e2e} +ECHO=${ECHO:-postman-echo.com} +SSE_PORT=${SSE_PORT:-18998} +REPO=$(cd "$(dirname "$0")/.." && pwd) +PROBE=${EGRESS_PROBE_TOKEN:-probe-$(date +%s)} +export EGRESS_PROBE_TOKEN=$PROBE + +DATA=$(mktemp -d /tmp/credentials-e2e.XXXXXX) +DAEMON_PID="" +SSE_PID="" + +cleanup() { + status=$? + if [[ $status -ne 0 && -f "$DATA/daemon.log" ]]; then + echo "== daemon log tail" + tail -30 "$DATA/daemon.log" + fi + for pid in $DAEMON_PID $SSE_PID; do + kill "$pid" 2>/dev/null || true + done + wait 2>/dev/null || true + cocoon vm list --format json 2>/dev/null | + jq -r '.[] | select(.config.name | startswith("sbx-")) | .config.name' | + while read -r vm; do + cocoon vm stop --force "$vm" >/dev/null 2>&1 || true + cocoon vm rm --force "$vm" >/dev/null 2>&1 || true + done || true + rm -rf "$DATA" + exit "$status" +} +trap cleanup EXIT + +echo "== build" +if [[ -n ${SANDBOXD_BIN:-} && -n ${CREDSMOKE_BIN:-} ]]; then + cp "$SANDBOXD_BIN" "$DATA/sandboxd" && cp "$CREDSMOKE_BIN" "$DATA/credsmoke" +else + (cd "$REPO/sandboxd" && GOWORK=off go build -o "$DATA/sandboxd" .) + (cd "$REPO/e2e" && GOWORK=off go build -o "$DATA/credsmoke" ./cmd/credsmoke) +fi + +echo "== provision cluster root + this node's intermediate" +"$DATA/sandboxd" ca init -out "$DATA/ca" -cn "e2e cluster root" >/dev/null +"$DATA/sandboxd" ca issue-intermediate -root-cert "$DATA/ca/root.crt" \ + -root-key "$DATA/ca/root.key" -node e2e-node -out "$DATA/node" >/dev/null + +echo "== event stream origin on 127.0.0.1:$SSE_PORT (one event, then holds)" +python3 -c ' +import http.server, sys, time +class H(http.server.BaseHTTPRequestHandler): + def do_GET(self): + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.end_headers() + self.wfile.write(b"data: one\n\n") + self.wfile.flush() + time.sleep(30) + def log_message(self, *a): + pass +http.server.ThreadingHTTPServer(("127.0.0.1", int(sys.argv[1])), H).serve_forever() +' "$SSE_PORT" & +SSE_PID=$! + +cat >"$DATA/config.json" <>"$DATA/daemon.log" 2>&1 & + DAEMON_PID=$! + for _ in $(seq 1 40); do curl -sf "http://$ADDR/healthz" >/dev/null 2>&1 && break; sleep 0.5; done +} + +echo "== start sandboxd (intercept pool)" +start +for i in $(seq 1 300); do + curl -sf -H "Authorization: Bearer $TOKEN" "http://$ADDR/v1/info" 2>/dev/null | + jq -e '.pools[0].warm >= 1' >/dev/null 2>&1 && break + [[ $i == 300 ]] && { echo "pool never became warm"; exit 1; } + sleep 1 +done + +echo "== credsmoke" +"$DATA/credsmoke" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" -echo "$ECHO" -secret "$PROBE" \ + -sse "http://127.0.0.1:$SSE_PORT/" | tee "$DATA/smoke.log" +reattach=$(awk '/^REATTACH / {print $2}' "$DATA/smoke.log") +value=${reattach##*:} + +echo "== audit: claim credentials named, values absent" +grep -q '"secret":"probe,claim:API_KEY"' "$DATA/state/audit.jsonl" || { echo "audit names no claim:API_KEY"; exit 1; } +if grep -q "$value" "$DATA/state/audit.jsonl" "$DATA/state/usage.jsonl" 2>/dev/null; then + echo "a credential value reached the audit or usage journal" + exit 1 +fi +grep -q "$value" "$DATA/state/claims.json" || { echo "claims.json lost the credential"; exit 1; } + +echo "== restart sandboxd, recheck the surviving claim" +kill "$DAEMON_PID" +wait "$DAEMON_PID" 2>/dev/null || true +start +"$DATA/credsmoke" -addr "$ADDR" -token "$TOKEN" -echo "$ECHO" -reattach "$reattach" +echo "PASS" diff --git a/sdk/go/options.go b/sdk/go/options.go index 4a08b3e1..a2022df9 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -47,10 +47,17 @@ type VolumeInfo struct { Writable bool `json:"writable,omitzero"` } +// EnvInject makes the node's egress proxy send a host-only entry's value as Header on intercepted requests to Hosts, each exact or "*." suffix. +type EnvInject struct { + Hosts []string `json:"hosts"` + Header string `json:"header"` +} + // EnvVar is one entry of a claim's env; a nil Guest delivers it into the guest, false keeps it host-side for the node's egress. type EnvVar struct { - Value string `json:"value"` - Guest *bool `json:"guest,omitempty"` + Value string `json:"value"` + Guest *bool `json:"guest,omitempty"` + Inject *EnvInject `json:"inject,omitempty"` } // Option configures a New claim.