egress: MITM tunneled TLS with per-SNI certificate minting by haiyanmeng · Pull Request #871 · agent-substrate/substrate

haiyanmeng · 2026-08-11T20:45:10Z

New changes:

atenet sdsmint — an SDS server that mints a short-lived TLS leaf per SNI on demand. Runs as a native sidecar beside Envoy, sharing the atenet image but staying a separate container so the MITM signing key never lands on the data plane.
mitm_listener — the CONNECT route now hands the upgraded stream here instead of to the forward proxy. Two chains,
picked by tls_inspector: tls terminates using Envoy 1.37's on_demand_secret selector + sni cert mapper (hence the
1.34 → 1.37 bump), raw_buffer proxies plaintext http:// and enforces the allowlist as an :authority match, since
there's no mint on that path to refuse.
CA plumbing — internal/localca gains GenerateCA(GenerateOptions); kubectl-ate admin make-ca-pool gains --key-type,
--common-name; install-ate.sh creates the egress-mitm-ca-pool secret before the Deployment.
e2e (internal/e2e/suites/sdsmint) covers the tunnelled handshake, the leaf Envoy serves, and actor identity across the MITM, against the new egressprobe fixture.

Not included:

how to make actors trust egress-mitm-ca-pool

This is to address #823

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

Bowei Du (bowei)

Remove the intermediate CA code
Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.
Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()
I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.
Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

I mostly looked at the server code. Once you get the simplifications in, I will take a more detailed look at the rest of the change.

haiyanmeng · 2026-08-12T14:51:07Z

Remove the intermediate CA code

Done

Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.

Done

Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()

Done

I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.

minter.Forgetter() iis called through the optional interface in cmd/atenet/internal/sdsmint/deltastream.go:391:

  if f, ok := d.srv.minter.(Forgetter); ok {
      for _, name := range expired {
          f.Forget(name)
      }
  }

Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

Done

Taahir Ahmed (ahmedtd) · 2026-08-12T23:12:32Z

+
+	cmd.Flags().StringVar(&cfg.UDSPath, "uds-path", "", "unix socket to listen on; required, and the only transport offered, because leaf private keys transit this channel")
+	cmd.Flags().StringVar(&cfg.CAPoolPath, "ca-pool-path", "", "path to a localca pool JSON holding the MITM CA, the format substrate mounts its other CAs in")
+	cmd.Flags().StringVar(&cfg.CAID, "ca-id", "", "which CA in the pool to sign with; empty takes the first")


Let's leave the behavior to just take the first --- localca.Pool should be tracking which localca.CA is active for signing. I will send a PR to do this (and add rotation commands for the CA secrets).

Taahir Ahmed (@ahmedtd) , are you suggesting removing this flag and take the first CA from the pool by default?

Does your PR need to block this PR?

Can we default to first for now or it doesn't work without this flag.

It works as is. Here is how the sdsmint container in the egress gateway Pod sets it

args: - "sdsmint" - "--uds-path=/var/run/sdsmint/sdsmint.sock" - "--ca-pool-path=/run/ca-state/mitm-pool.json" - "--ca-id=mitm"

Yeah, we can just default to the first CA for now.

haiyanmeng · 2026-08-13T03:13:35Z

Also, mint is super cheap. The generation of the key pair is what takes time. so if we generate keypair at startup, every mint is super fast.

Agreed

remove all the state of the world code (SotW) and all of its StreamSecrets/FetchSecrets/buildSotW. We dont need it since we are doing delta xds.

remove all metrics - lets have a separate follow-up pr for metrics

I dont think we need the pproof endpoint for this PR

These can be done.

haiyanmeng · 2026-08-13T03:23:03Z

I think we should remove all caching for now, we should have a separate issue to add caching properly. For the first implementation we dont need to have cache in sds minter. Envoy is caching the certificate anyway. We can look into existing lru solutions later.

This can be done too.

All the withdrawal login in deltastreams.go - is it just a memory optimization? why do we need it right now?

This is to prevent Envoy from being OOM.

we dont need all the proactive rotation in deltastream.go + leafkey.go (rotateStale, ticker, reuseFraction/rotateFraction) - this is just because the PR is opinionated. In Delta xds - can do something like per-resource ttl - which would require envoy to re-request the secret. - but we dont need it right now

If we remove the rotation logic without adding an alternative like per-resource ttl, the UX may be bad. The secret Envoy holds expires, and Envoy keeps presenting it. Every client handshake to that host then fails on an expired cert. Let me test out this behavior.

Lior Lieberman (LiorLieberman) · 2026-08-13T03:25:06Z

If we remove the rotation logic without adding an alternative like per-resource ttl, the UX will be bad. The secret Envoy holds expires, and Envoy keeps presenting it. Every client handshake to that host then fails on an expired cert.

And would it retry and then fetch the new cert/secret again?

Taahir Ahmed (ahmedtd) · 2026-08-13T05:19:37Z

Also, mint is super cheap. The generation of the key pair is what takes time. so if we generate keypair at startup, every mint is super fast.

This isn't true for ECDSA as I understand it --- ECDSA keygen just reading random bytes from crypto/rand, whereas the signatures for creating the certificates require bigint math.

It's more performant than RSA, but I don't think we can get away without caching for a production system.

Bowei Du (bowei) · 2026-08-13T05:29:26Z

We should also be careful about exhausting randomness sources.

haiyanmeng · 2026-08-13T14:53:55Z

Bowei Du (@bowei) , Taahir Ahmed (@ahmedtd) , Lior Lieberman (@LiorLieberman) , I updated the PR:

Removed cache, metrics, pprof, SotW, cert rotation from sdsmint;
Added resource TTL. (I verified the resource TTL works as expected. See https://github.com/haiyanmeng/substrate/blob/sds-3/poc/sdsmint/expiry/README.md).

PTAL.

Lior Lieberman (LiorLieberman)

Thanks for all the hard work, sds server code mostly looks good to me.

Left some comments. Plus we need to think how to gurad this with a feature flag or something.

Also - any resolution on https://github.com/agent-substrate/substrate/pull/871/changes#r3771056979?

Lior Lieberman (LiorLieberman) · 2026-08-14T01:13:13Z

+	st := &deltaStream{
+		srv:      s,
+		stream:   stream,
+		sendCh:   make(chan *discovery.DeltaDiscoveryResponse, 8),


nit: why 8? whats 8?

concurrency limit?

8 was picked by Claude. I added a comment generated by Claude. We can tune it in the future if needed.

what is it for though? limiting concurrency?

sendDepth is just the capacity of sendCh. If we set it to 0, every response becomes a synchronous rendezvous with sendLoop, so the select loop stalls on each write. That's the thing the buffer exists to avoid.

Lior Lieberman (LiorLieberman) · 2026-08-14T01:26:00Z

dont we already have some egress testing go code machinery (if not, how do we test egress now?) maybe this can be integrated there vs creating a new package?

The only thing I am aware of https://github.com/agent-substrate/substrate/blob/main/internal/e2e/suites/networking/networking_test.go.

Let us look into how to converge these egress tests after actors trust egress-mitm-ca-pool.

Lior Lieberman (LiorLieberman) · 2026-08-14T01:28:17Z

are we going to have an internal listener now for all envoy deployments? we should likely guard it with a feature flag.

Also does this code/config right now only pass to the envoy internal listener when MITM is needed? (e.g actor dialed HTTPS) or every time? the impl should be the former.

are we going to have an internal listener now for all envoy deployments? we should likely guard it with a feature flag.

#926 adds --experimental-use-sdsmint to select the egress manifest. The plan is to submit #871 first, then submit #926 as a fast followup.

Also does this code/config right now only pass to the envoy internal listener when MITM is needed? (e.g actor dialed HTTPS) or every time? the impl should be the former.

mitm_listener has two filter chains. One for tls traffic (transport_protocol: tls), the other for plaintext traffic (transport_protocol: raw_buffer).

not sure I understand the response. if actor sends http traffic (meaning no need for mitm) - do we go to mitm listener?

it goes to mitm_listener. But it has two filter chain. Only the filter chain for https traffic talks to sdsmint.

For http traffic, it does not talk to sdsmint.

haiyanmeng · 2026-08-14T01:48:44Z

Thanks for all the hard work, sds server code mostly looks good to me.

Left some comments. Plus we need to think how to gurad this with a feature flag or something.

Also - any resolution on https://github.com/agent-substrate/substrate/pull/871/changes#r3771056979?

Not yet. Will follow with Bowei and Taahir on that.

yanavlasov

LGTM for Envoy config and minting server code.

The egress gateway terminated the actor's CONNECT tunnel and forwarded the bytes inside it opaquely, so nothing could be said about the TLS session an actor established through it. This intercepts that session: Envoy terminates the tunnelled TLS with a leaf minted for the SNI the client asked for, which puts the plaintext request on a filter chain where policy can be applied later. The data path, all inside the atenet-egress pod: atunnel client -> gateway front door (mTLS, socket listener) -> CONNECT to an IP:port -> mitm_internal cluster (envoy_internal_address) -> mitm_listener (internal, no socket anywhere) -> tls_inspector reads the SNI -> DownstreamTlsContext, on_demand_secret selector -> DELTA_GRPC SDS over a unix socket to sdsmint -> leaf minted for that SNI, handshake resumes -> dynamic forward proxy to the real destination Supporting changes: internal/localca: CA.SigningKey narrows from crypto.PrivateKey to crypto.Signer, so a key that parses but cannot sign (X25519 out of PKCS#8) is refused at load rather than at the first handshake. Adds CA.Validate, and GenerateCA/GenerateOptions with a key type, common name and lifetime. GenerateED25519CA stays as a wrapper -- every existing caller wants exactly what it produced. kubectl-ate admin make-ca-pool: --key-type and --common-name. hack/install-ate.sh: creates the egress-mitm-ca-pool secret, ecdsa-p256 rather than the ed25519 default. These leaves are validated by arbitrary clients inside actor sandboxes, where Ed25519 support cannot be assumed. internal/atunnel: ErrGatewayHandshake and ConnectRejectedError replace formatted strings, so a caller can tell a front-door TLS rejection from an authorization denial without matching on message text. A gateway that resets instead of alerting is folded into the former, since which of the two happens is a race. e2e: internal/e2e/suites/sdsmint covers the tunnelled handshake, the leaf Envoy serves, and actor identity across the MITM, against the new egressprobe fixture.

haiyanmeng · 2026-08-14T20:26:27Z

I rebased after #959 is merged. We should expect TestActorEgressHTTPS to fail.

haiyanmeng · 2026-08-14T21:19:38Z

https://github.com/agent-substrate/substrate/actions/runs/31837847684/job/94888039437?pr=871

The test TestActorEgressHTTPS indeed failed with {"error":"request failed: Get \"[https://example.com/\](https://example.com//)": tls: failed to verify certificate: x509: certificate signed by unknown authority"}:

=== RUN   TestActorEgressHTTPS
    networking_test.go:117: creating actor networking-e2e/egress-https-1786739670953880925
    networking_test.go:117: resumed actor networking-e2e/egress-https-1786739670953880925
    networking_test.go:128: Actor HTTPS egress fetch returned HTTP 502, want 200; body: {"error":"request failed: Get \"[https://example.com/\](https://example.com//)": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
--- FAIL: TestActorEgressHTTPS (1.31s)

Bowei Du (@bowei) , let us merge #926 first. Then I will rebase and update this PR.

Lior Lieberman (LiorLieberman)

Keith Mattix II (@keithmattix) has done a nice thing on #715 where he builds the listener in go (see xds.go in his PR) based on whether someone enabled connect (using a flag).

we should do something similar for MITM listener and other config changes.

Taahir Ahmed (ahmedtd) · 2026-08-14T22:47:10Z

We should also be careful about exhausting randomness sources.

Go's random sources cannot be exhausted --- they are a CSPRNG keyed with initial randomness read from /dev/urandom. /dev/urandom itself is a CSPRNG seeded with a few bytes of random derived from startup or the RDRAND instruction.

Linux does a bunch of accounting of entropy sources, and tries to reseed /dev/random and /dev/urandom once many values are read from them. This is widely considered to be superstitious hokum.

Taahir Ahmed (ahmedtd) · 2026-08-14T22:47:51Z

+
+	cmd.Flags().StringVar(&cfg.UDSPath, "uds-path", "", "unix socket to listen on; required, and the only transport offered, because leaf private keys transit this channel")
+	cmd.Flags().StringVar(&cfg.CAPoolPath, "ca-pool-path", "", "path to a localca pool JSON holding the MITM CA, the format substrate mounts its other CAs in")
+	cmd.Flags().StringVar(&cfg.CAID, "ca-id", "", "which CA in the pool to sign with; empty takes the first")


Yeah, we can just default to the first CA for now.

Taahir Ahmed (ahmedtd) · 2026-08-14T22:54:38Z

+				return
+			}
+			if err := d.stream.Send(resp); err != nil {
+				select {


Does this select do anything?

Taahir Ahmed (ahmedtd) · 2026-08-14T23:01:46Z

+	// derived from the leaf lifetime rather than configured separately, so the
+	// two cannot be set into an order that does not work. See pack, which
+	// stamps it, for what Envoy does with it.
+	resourceTTL time.Duration


It seems like the resourceTTL we hand back should be derived from the NotAfter timestamp of the cert we minted (for example, cert.NotAfter.Add(-10*time.Minute). Then we don't have a server-wide setting that can be misconfigured.

Taahir Ahmed (ahmedtd) · 2026-08-14T23:04:04Z

+// toSecret packs a minted cert into the Secret proto Envoy expects back. The
+// secret's name MUST equal the requested resource name (the SNI), or Envoy
+// will not match the response to its subscription.
+func toSecret(name string, c *certauth.MintedCert) *tlsv3.Secret {


This probably belongs in deltastream.go, since that's where it's used

Taahir Ahmed (ahmedtd) · 2026-08-14T23:11:31Z

LGTM with some nits

haiyanmeng marked this pull request as draft August 11, 2026 20:45

haiyanmeng force-pushed the sds-1 branch from 7cb6eb3 to 8fb773f Compare August 11, 2026 21:01

Eitan Yarmush (EItanya) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) August 11, 2026 23:04

haiyanmeng force-pushed the sds-1 branch 2 times, most recently from 195d6c4 to 7976e5a Compare August 11, 2026 23:41

haiyanmeng marked this pull request as ready for review August 11, 2026 23:42

haiyanmeng requested review from Eitan Yarmush (EItanya), Taahir Ahmed (ahmedtd) and Max Thompson (thompsonmax) August 11, 2026 23:43

haiyanmeng mentioned this pull request Aug 11, 2026

localca: accept external signers and make CA generation configurable #821

Closed

haiyanmeng commented Aug 11, 2026

View reviewed changes

Comment thread manifests/ate-install/atenet-egress.yaml Outdated

haiyanmeng force-pushed the sds-1 branch from 7976e5a to 6d731dd Compare August 11, 2026 23:54

haiyanmeng added the area/network label Aug 11, 2026

haiyanmeng force-pushed the sds-1 branch 3 times, most recently from 6080f7e to 7b74711 Compare August 12, 2026 01:08

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Bowei Du (bowei) self-assigned this Aug 12, 2026

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

haiyanmeng requested review from Taahir Ahmed (ahmedtd) and Bowei Du (bowei) August 12, 2026 15:07

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/cmd.go Outdated

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/leafkey.go Outdated

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/minter.go Outdated

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/cmd.go Outdated

Comment thread cmd/atenet/internal/sdsmint/deltastream.go

haiyanmeng force-pushed the sds-1 branch from 1b2dd23 to 6f76856 Compare August 13, 2026 11:39

Bowei Du (bowei) reviewed Aug 13, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/deltastream.go Outdated

haiyanmeng force-pushed the sds-1 branch from 1e6d359 to 1eb79f1 Compare August 13, 2026 14:51

haiyanmeng requested review from Taahir Ahmed (ahmedtd) and Bowei Du (bowei) August 13, 2026 14:51

haiyanmeng force-pushed the sds-1 branch from 1eb79f1 to 0da1713 Compare August 13, 2026 16:39

Lior Lieberman (LiorLieberman) self-assigned this Aug 13, 2026

haiyanmeng mentioned this pull request Aug 13, 2026

install: add --experimental-use-sdsmint to select the egress manifest #926

Open

2 tasks

This was referenced Aug 13, 2026

Add clusterTrustBundle as a SystemInfo volume data source #932

Open

[stacked on #803] Add clusterTrustBundle as a SystemInfo volume data source #941

Draft

Lior Lieberman (LiorLieberman) reviewed Aug 14, 2026

View reviewed changes

haiyanmeng force-pushed the sds-1 branch from d965e83 to 02c19fe Compare August 14, 2026 02:43

haiyanmeng requested a review from Lior Lieberman (LiorLieberman) August 14, 2026 02:44

yanavlasov approved these changes Aug 14, 2026

View reviewed changes

haiyanmeng added 2 commits August 14, 2026 16:25

Address comments

fbef520

haiyanmeng force-pushed the sds-1 branch from 02c19fe to fbef520 Compare August 14, 2026 20:25

Lior Lieberman (LiorLieberman) reviewed Aug 14, 2026

View reviewed changes

Taahir Ahmed (ahmedtd) reviewed Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

haiyanmeng commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Bowei Du (bowei) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haiyanmeng commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haiyanmeng commented Aug 13, 2026

Uh oh!

haiyanmeng commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 13, 2026

Uh oh!

Taahir Ahmed (ahmedtd) commented Aug 13, 2026

Uh oh!

haiyanmeng commented Aug 11, 2026 •

edited

Loading

Bowei Du (bowei) left a comment •

edited

Loading

haiyanmeng commented Aug 13, 2026 •

edited

Loading

haiyanmeng commented Aug 14, 2026 •

edited

Loading