design-proposal: tenant external network connectivity via a gateway VM (IPsec/WireGuard)#30
design-proposal: tenant external network connectivity via a gateway VM (IPsec/WireGuard)#30myasnikovdaniil wants to merge 2 commits into
Conversation
Adds a design proposal for tenant-managed site-to-site connectivity (IPsec or WireGuard) terminated in a KubeVirt VM that NAT-bridges an external site to the tenant's managed-app ClusterIPs, both directions, without granting tenants privileged host-cluster pods. Includes the end-to-end prototype validation results and the relationship to the ClusterMesh/Kilo proposal (#7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a design proposal for tenant-managed external network connectivity via a gateway VM, covering traffic flow design, deployment semantics, security boundaries, validation findings, rollout, and alternatives. ChangesTenant Site Gateway Design Proposal
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal for tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) in Cozystack, which terminates VPN tunnels inside a KubeVirt VM to isolate privileged networking. The review feedback highlights several key technical refinements for the proposal: ensuring the guest OS is configured to own virtual service-exposure addresses for inbound traffic, addressing potential port collisions for outbound traffic to multiple remote targets by mapping them to unique targetPorts, providing concrete KubeVirt API examples for the block size override, and noting that standard VRRP for high availability may be blocked by CNI-level IP/MAC spoofing prevention, requiring alternative failover mechanisms.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| ### Inbound path (external site → any tenant app) | ||
|
|
||
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. |
There was a problem hiding this comment.
For the inbound path, the remote peer targets a virtual "service-exposure" address. To ensure the gateway VM can receive and DNAT this traffic, the guest OS needs to configure these virtual IPs (e.g., as alias IPs or on a dummy interface) so that the guest kernel accepts packets destined for them.
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. | |
| The remote peer targets a virtual "service-exposure" address that the gateway VM owns (configured as an alias or dummy interface IP inside the guest OS via cloud-init) and maps to an app ClusterIP. |
There was a problem hiding this comment.
This isn't needed for the validated design: the tunnel is route-based (VTI), so the decrypted packet arrives on the tunnel interface and is DNAT'd in PREROUTING — which runs before the kernel's local-delivery check — so the gateway need not own the virtual service-exposure address on an interface. The dev-cluster prototype confirmed this: nat destination rule inbound-interface vti0 destination <vip> → <ClusterIP> worked with no alias/dummy IP assigned. Keeping the text as-is to avoid documenting a requirement that doesn't apply to the route-based path.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | ||
| 2. The node routes the `ClusterIP` to the gateway VM. | ||
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
When using a local ClusterIP Service to route outbound traffic to the gateway VM, the CNI/kube-proxy on the client node will perform DNAT, translating the Service's ClusterIP to the gateway VM's pod IP before the packet reaches the VM.\n\nIf a tenant has multiple remote targets that use the same destination port (e.g., two different remote databases on port 5432), the gateway VM will receive packets for both targets with the same destination IP (the VM's pod IP) and destination port (5432). It will have no way to distinguish which remote target was originally intended.\n\nTo resolve this, the design should explicitly specify that each remote target must map to a unique targetPort (local listener port) on the gateway VM. The local Services can still expose the standard port (e.g., 5432), but they must map to these unique targetPorts on the gateway VM (e.g., 5432 and 5433), which the gateway VM then DNATs to the respective remote IP:port.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | |
| 2. The node routes the `ClusterIP` to the gateway VM. | |
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. | |
| 1. The app connects to a local Service name the chart creates (e.g. remote-db.<tenant>.svc:port) — the only app-side change is the target hostname.\n2. The node translates the ClusterIP to the gateway VM's pod IP and a unique targetPort (local listener port) allocated on the VM. This unique port is required so the VM can distinguish between multiple remote targets that use the same destination port.\n3. **DNAT**: the unique local listener port on the VM -> the real remote IP:port behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
Good catch — addressed in 72db47e. The outbound path now states each remote target maps to a per-target-unique listener port on the gateway (the local Service still exposes the standard port, e.g. 5432), so the VM can disambiguate multiple targets sharing a destination port. The values schema's remoteTargets[].localPort already carries this.
| - **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. | ||
| - **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. | ||
| - **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. | ||
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
To make the KubeVirt blockSize override actionable and clear for the implementer, it would be beneficial to include a brief example of the exact KubeVirt API fields used to configure it.
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). | |
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt blockSize override (e.g., setting blockSize.custom.logical to 512 and blockSize.custom.physical to 4096 in the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
Done in 72db47e — added the concrete example (blockSize.custom.logical: 512 / physical: 4096 on the disk spec).
|
|
||
| - **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. | ||
| - **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. | ||
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. |
There was a problem hiding this comment.
In standard Kubernetes CNIs (including Cilium and kube-ovn), IP/MAC spoofing prevention is typically enabled by default. This means standard VRRP (keepalived) running inside guest VMs on the default pod network may fail because the CNI will block gratuitous ARPs or refuse to route the shared VIP to the standby VM.\n\nTo support HA, the platform might need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism instead of pure guest-level VRRP.
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. | |
| - **HA.** Is active/standby in scope for a later iteration? Note that standard VRRP (e.g., keepalived with a shared VIP) inside guest VMs on the default pod network may fail due to CNI-level IP/MAC spoofing prevention (which blocks gratuitous ARPs and unauthorized IPs). The HA design will likely need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism. |
There was a problem hiding this comment.
Agreed that guest-level VRRP with a shared VIP is problematic — the new High Availability section covers it (72db47e). One correction from validation: the blocker isn't IP/MAC anti-spoofing (that's solvable with a scoped kube-ovn allowed-address-pair — validated: the VIP moves through OVN with port-security kept on). The real blocker is that VRRP advertisements (IP proto 112) are dropped pod-to-pod by Cilium's conntrack (non-TCP/UDP/ICMP/SCTP; see Cilium CFP #39601, closed not-planned), so keepalived can't elect over the pod network. HA therefore uses Service/endpoint failover or a side-channel for the election, as documented.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@design-proposals/tenant-site-gateway/README.md`:
- Around line 157-158: The tenant-secret flow is still underspecified in the
README: define the concrete delivery path for PSKs/certs/WireGuard keys into the
guest without putting them in the VM spec or cloud-init, and document the
post-reconciliation read-access model clearly. Update the tenant-supplied
secrets section to name the component or mechanism that performs delivery and
the expected access boundaries so the security story is explicit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f6ce1e74-edfe-4394-9c0e-b857bcb66a81
📒 Files selected for processing (1)
design-proposals/tenant-site-gateway/README.md
Adds a High availability section (KubeVirt live-migration for planned maintenance; Service-fronted active/passive and kube-ovn allowed-address-pairs shared-VIP for unplanned failure), with the shared-VIP mechanism validated on a development cluster. Attributes the pod-to-pod drop of non-TCP/UDP/ICMP/SCTP IP protocols (VRRP proto 112, ESP proto 50) to Cilium's conntrack rather than the geneve tunnel, unifying it with the native-ESP finding. Also addresses review feedback: per-target-unique outbound listener ports, a concrete blockSize example, a note that SNAT keeps the gateway anti-spoofing clean, and a stronger secret-handling open question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil
requested review from
Andrei Kvapil (kvaps) and
Timofei Larkin (lllamnyp)
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Thanks — the e2e validation here (the ESP-in-UDP finding, the SNAT negative test, the AAP scoping proof) is exactly the groundwork we needed. We discussed the direction internally; three structural changes before this is ready, plus nits.
1. Routed (no-NAT) mode is the primary mechanism and belongs in Phase 1.
Tenants already run this pattern today (virtual routers: the ovn.kubernetes.io/routes namespace annotation + relaxed port security), so shipping NAT-first would regress their UX. Please restructure around two modes:
- routed (default): the routes annotation (kubeovn-webhook already inherits it onto pods at creation) provides the return path, so SNAT is unnecessary and the client source IP is preserved. The "missing SNAT → black-hole" finding holds only for NAT mode — scope it accordingly.
- NAT (fallback): keep the current design as an enumerated-targets mode for when routing is impossible (remote CIDR overlaps podCIDR/serviceCIDR/node networks) or the remote side only accepts narrow traffic selectors. State the limits honestly: per-host:port entries, no whole-subnet reachability, no ICMP. NETMAP-style 1:1 subnet mapping → future work.
Anti-spoofing for routed mode: disabling port_security on the gateway VM port is acceptable for v1. Containment comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program on the node-side veth, so the policy verdict doesn't depend on the packet's source IP; a spoofed source cannot widen the reachable set beyond the tenant tree + world. Document the residual risks in Security: identity spoofing within the tenant's own namespace (receiver-side identity is resolved from src IP via ipcache), and spoofed-source egress where SNAT is absent. Target state (follow-up, not v1): scoped port_security — keep it enabled and add the declared remote CIDRs to the port's allowed addresses; your AAP validation is the right mechanism, it only lacks CIDR support in kube-ovn (OVN itself accepts MAC IP/mask).
Also state explicitly: routed mode is transparent L3 for TCP/UDP/ICMP/SCTP only — per your own conntrack finding, ESP/GRE/OSPF/VRRP still won't cross the pod fabric.
2. Day-2 reconfiguration needs a section, with the mechanism agreed now.
Direction we settled on: VyOS HTTP API driven by a platform controller — render the target config from values and push it. VyOS commits are transactional, so unrelated changes don't touch the tunnel and key rotation only re-establishes the SA. The API key is delivered once at first boot via cloud-init; the API must be reachable by the platform only (define that boundary in Security). Implementation may land as a separate phase — v1 may temporarily fall back to "config change = VM reboot" if called out explicitly.
3. Add a "low-level entities & controller" section.
Tenants can't (and shouldn't) annotate their namespaces, so a platform controller mediates:
- user input stays in the catalog app values — no new CRD (the aggregated API server just mirrors HelmReleases); validation = values.schema plus controller-side checks at reconcile, surfaced as status conditions;
- the controller validates declared remote CIDRs (disjoint from podCIDR/serviceCIDR/join/node networks; overlap between tenants is fine — routes are namespace-scoped), sets the namespace routes annotation, relaxes/scopes port security on the gateway VM port, and cleans everything up on app deletion;
- routes apply only to newly created pods (the webhook mutates at Create) — no auto-restart of tenant workloads; the controller reports pods whose annotation lags the namespace ("pods pending route") and the user rolls them;
- keep the CNI-touching operations behind a small backend interface so a future CNI can implement them natively.
Nits:
- Inbound exposure model is inconsistent: the text uses virtual IPs (
10.200.0.10:5432), the values sketch uses{name, listenPort, targetService, port}— pick one. In routed mode the remote side hits ClusterIPs directly, so virtual exposure addresses are a NAT-mode-only concept. - HA: agreed with your recommendation (single VM + live-migration in v1). For the follow-up, prefer the Service-fronted active/passive path — CNI-agnostic, which matters since we don't want to deepen kube-ovn coupling.
- Sync with the portal-side implementation before building the image — it uses the same VyOS + cloud-init shape; image and config format should be shared.
- The tunnel LoadBalancer should align with the structured external exposure redesign (#29) rather than adding another ad-hoc LB path.
- Scope: list per-tenant egress IP as explicitly deferred future work — the gateway VM is its natural home later.
2 participants
Summary
Adds a design proposal for tenant-managed external network connectivity:
terminating a site-to-site VPN (IPsec or WireGuard) inside a KubeVirt VM that
NAT-bridges an external site to the tenant's managed apps, in both directions,
for any L4 protocol — without granting tenants privileged host-cluster pods.
The proposal covers:
terminated in a VM guest (contained privilege)
packages/apps/site-gatewaycatalog app with a values-schema sketchfills the tenant-namespace-apps + NAT-egress space design-proposal: cross-cluster mesh for tenant access to host services #7 defers
The design was prototyped and validated end-to-end (two gateway VMs + a real
managed Postgres); results are included, notably the finding that native ESP is
dropped by the Cilium/kube-ovn overlay (forced UDP encapsulation required for
IPsec; WireGuard sidesteps it).
Test plan
Design proposal; no code. Implementation testing (helm-unittest across backends,
e2e of the two-VM topology asserting inbound/outbound + SNAT-required + MSS
behavior) is scoped in the proposal and will follow in implementation PRs.
🤖 Generated with Claude Code
Summary by CodeRabbit