Peer mTLS for inter-node gRPC

[bootjp]

Split out of #1250, where a review (CWE-319) correctly observed that the peer forward token travels in cleartext.

Current state

Every peer dial in the server process goes through internal.GRPCDialOptions (internal/grpc.go:21), which is insecure.NewCredentials() and nothing else. There is no peer-TLS option anywhere in the server. TLS exists only in the client tools — cmd/elastickv-admin and cmd/elastickv-backup both have loadTransportCredentials with CA file, server name, and skip-verify flags.

So the inter-node channel carries, in cleartext:

• Raft traffic itself (internal/raftengine/etcd/grpc_transport.go), including snapshot payloads
• the admin/peer bearer token on Internal.Forward and Internal.ForwardLeaseRead
• the S3 blob peer token on S3BlobFetch

An observer on the peer network can already read and inject the Raft traffic, so the token is not the weakest link — which is why #1250 keeps sending it rather than withholding it on insecure connections. Withholding it would disable the gate in every existing deployment rather than harden it.

What this needs

Confidentiality and peer authentication for the inter-node channel, not just for the token:

• dial-side credentials for peer connections (transport, and ideally client certs for mutual auth)
• server-side grpc.Creds on the internal listener
• operator surface: CA file, cert/key, server name, and a way to run mixed during a rollout
• a rollout story, since this has the same skew shape as kv: authenticate Internal.Forward and pin the TSO issuance invariants #1250's gate: a node presenting a client cert to a peer that does not yet require one, and vice versa

Because it adds an operator-facing surface and changes the wire, this wants a *_proposed_*.md design doc before implementation, per CLAUDE.md.

Related

• kv: authenticate Internal.Forward and pin the TSO issuance invariants #1250 — the Internal.Forward gate whose token this would protect; its kv/leader_proxy.go comment and docs/design/2026_08_29_partial_tso_batch_slot_claims.md §4a record the threat model this issue resolves.

https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE