From 809babefdb1c910aa508cfddb9b11bebb10608f5 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Tue, 14 Jul 2026 14:32:18 -0400 Subject: [PATCH] fix(deploy): let the cluster own the Gateway, keep only the route (#518) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This repo cannot know the cluster it lands in, and gateway-listeners.yaml was trying to. Between them, the ListenerSet and its routes hardcoded four facts about cfp-live-cluster — the shared Gateway's name and namespace, the ClusterIssuer, the TLS Secret name, and the ListenerSet API itself — and three were wrong there: - cert-manager.io/cluster-issuer: letsencrypt-prod is dead. It solves ACME over ingress-nginx, which requires PROXY protocol on every connection and is therefore unreachable from inside the cluster. No certificate has issued through it since May. - certificateRefs: balancer-tls is the legacy Ingress Secret. Gateway certs are -gw-tls. - ListenerSet is not reconciled by Envoy Gateway v1.7.3. It watches XListenerSet and logs "XListenerSet CRD not found, skipping XListenerSet watch". The object applies cleanly and is then ignored — the app just has no listener, with nothing in any log to say why. None of that is a mistake anyone could have avoided from this repo. It isn't knowable here. So stop trying to know it. Split on what each side actually knows: this repo -> HTTPRoute: paths, backend Services, ports. App facts. They change when the app changes, in the same commit. cluster -> Gateway: listeners, hostnames, TLS, issuers. Cluster facts. They differ per environment and are shared with every other app. The route now attaches to a Gateway the cluster provides, named after the app, and declares no hostnames — a route that omits hostnames inherits them from the listeners it attaches to. So one file serves balancerproject.org in live and balancer.sandbox.k8s.phl.io in sandbox, with no patches and no placeholders. The HOSTNAME_PLACEHOLDER substitutions and the overlay routing patches are gone. Drops balancer-http-redirect too: the cluster already redirects every hostname reaching it on port 80. Worse than redundant — a route claiming an exact hostname on the shared HTTP listener outranks the catch-all redirect and can shadow cert-manager's ACME solver route, which is what produced the apex 404 that #517 set out to fix. That 404 was a stuck certificate, and it cleared the moment the cert issued. Adds a README stating the contract, so the next person doesn't have to infer it. Refs CodeForPhilly/cfp-live-cluster#166, #168 --- deploy/manifests/balancer/README.md | 60 +++++++++++++++++++ .../balancer/base/gateway-listeners.yaml | 27 --------- deploy/manifests/balancer/base/httproute.yaml | 41 ++++--------- .../balancer/base/kustomization.yaml | 1 - .../overlays/production/kustomization.yaml | 31 ++-------- .../overlays/sandbox/kustomization.yaml | 26 +------- 6 files changed, 80 insertions(+), 106 deletions(-) create mode 100644 deploy/manifests/balancer/README.md delete mode 100644 deploy/manifests/balancer/base/gateway-listeners.yaml diff --git a/deploy/manifests/balancer/README.md b/deploy/manifests/balancer/README.md new file mode 100644 index 00000000..05bd7fca --- /dev/null +++ b/deploy/manifests/balancer/README.md @@ -0,0 +1,60 @@ +# balancer deployment manifests + +Consumed by the Code for Philly cluster repos +([cfp-live-cluster](https://github.com/CodeForPhilly/cfp-live-cluster), +[cfp-sandbox-cluster](https://github.com/CodeForPhilly/cfp-sandbox-cluster)), which project +`base/` into their GitOps trees. Also usable directly against a local `kind` cluster +(`deploy/kind-config.yaml`). + +## Who owns what + +Routing is split along a single line: **this repo declares what the app serves; the +cluster declares where it's reachable.** + +| This repo owns | The cluster owns | +| --- | --- | +| `HTTPRoute` — paths, backend Services, ports | `Gateway` — listeners, hostnames, TLS | +| Deployment, Service, app config | Certificates, ClusterIssuers, `GatewayClass` | + +Paths and ports are facts about the application: they change when the app changes, and +they belong in the same commit as that change. Hostnames, TLS, and issuers are facts +about the cluster: they differ between live and sandbox, they're negotiated across every +app sharing the ingress, and they change when the *cluster* changes. + +## The contract + +**The cluster provides a `Gateway` named after the app, in the app's namespace.** That +sentence is the whole interface. + +`base/httproute.yaml` attaches to it by name and declares no hostnames: + +```yaml +spec: + parentRefs: + - name: balancer # the Gateway the cluster provides + # no hostnames — inherited from the Gateway's listeners +``` + +A route that omits `hostnames` inherits them from the listeners it attaches to. So the +same file serves `balancerproject.org` in live and `balancer.sandbox.k8s.phl.io` in +sandbox, with no patching and no placeholders. + +## What must NOT live in this repo + +Anything that names a cluster implementation detail: + +- **`Gateway` or `ListenerSet` resources.** Cluster-owned. +- **`cert-manager.io/cluster-issuer` annotations**, or any ClusterIssuer name. +- **TLS `certificateRefs` / cert Secret names.** +- **`gatewayClassName`**, or the name/namespace of any shared Gateway. +- **An HTTP→HTTPS redirect.** The cluster already redirects every hostname reaching it + on port 80. An app-level redirect is redundant, and a route claiming an exact hostname + on the shared HTTP listener can shadow cert-manager's ACME solver route. + +None of these are knowable from this repo, and getting one wrong fails in ways that are +hard to see. A `ListenerSet`, for instance, applies cleanly to the API server and is then +silently ignored by Envoy Gateway v1.7.3 — the app simply has no listener, with nothing +in any log to say why. + +If you need something from the cluster that isn't here, open an issue on the cluster repo +rather than guessing at its internals. diff --git a/deploy/manifests/balancer/base/gateway-listeners.yaml b/deploy/manifests/balancer/base/gateway-listeners.yaml deleted file mode 100644 index 66b32664..00000000 --- a/deploy/manifests/balancer/base/gateway-listeners.yaml +++ /dev/null @@ -1,27 +0,0 @@ -apiVersion: gateway.networking.k8s.io/v1 -kind: ListenerSet -metadata: - name: balancer-listeners - annotations: - cert-manager.io/cluster-issuer: letsencrypt-prod-gateway - hostname: HOSTNAME_PLACEHOLDER -spec: - parentRef: - name: main-gateway - namespace: envoy-gateway-system - group: gateway.networking.k8s.io - kind: Gateway - listeners: - - name: http - protocol: HTTP - port: 80 - hostname: HOSTNAME_PLACEHOLDER - - name: https - protocol: HTTPS - port: 443 - hostname: HOSTNAME_PLACEHOLDER - tls: - mode: Terminate - certificateRefs: - - name: balancer-tls - kind: Secret diff --git a/deploy/manifests/balancer/base/httproute.yaml b/deploy/manifests/balancer/base/httproute.yaml index be52bc16..e77e58e8 100644 --- a/deploy/manifests/balancer/base/httproute.yaml +++ b/deploy/manifests/balancer/base/httproute.yaml @@ -2,16 +2,20 @@ apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: balancer - annotations: - hostname: HOSTNAME_PLACEHOLDER spec: + # Attaches to the Gateway the cluster provides for this app. The cluster names + # it after the app, in the app's namespace — that convention is the entire + # contract. See ../README.md. parentRefs: - - name: balancer-listeners - kind: ListenerSet - group: gateway.networking.k8s.io - sectionName: https - hostnames: - - HOSTNAME_PLACEHOLDER + - name: balancer + + # No `hostnames` on purpose. + # + # A route that omits hostnames inherits the hostnames of the listeners it + # attaches to, so the cluster's Gateway decides where this app is reachable — + # balancerproject.org in live, balancer.sandbox.k8s.phl.io in sandbox — and + # this file stays identical in every environment. + rules: - matches: - path: @@ -20,24 +24,3 @@ spec: backendRefs: - name: balancer port: 80 ---- -apiVersion: gateway.networking.k8s.io/v1 -kind: HTTPRoute -metadata: - name: balancer-http-redirect - annotations: - hostname: HOSTNAME_PLACEHOLDER -spec: - parentRefs: - - name: balancer-listeners - kind: ListenerSet - group: gateway.networking.k8s.io - sectionName: http - hostnames: - - HOSTNAME_PLACEHOLDER - rules: - - filters: - - type: RequestRedirect - requestRedirect: - scheme: https - statusCode: 301 diff --git a/deploy/manifests/balancer/base/kustomization.yaml b/deploy/manifests/balancer/base/kustomization.yaml index 8df1e2e8..3541d3f1 100644 --- a/deploy/manifests/balancer/base/kustomization.yaml +++ b/deploy/manifests/balancer/base/kustomization.yaml @@ -5,7 +5,6 @@ resources: - namespace.yaml - deployment.yaml - service.yaml - - gateway-listeners.yaml - httproute.yaml labels: diff --git a/deploy/manifests/balancer/overlays/production/kustomization.yaml b/deploy/manifests/balancer/overlays/production/kustomization.yaml index 7a5c4af4..0fe80266 100644 --- a/deploy/manifests/balancer/overlays/production/kustomization.yaml +++ b/deploy/manifests/balancer/overlays/production/kustomization.yaml @@ -24,28 +24,9 @@ labels: environment: production app.kubernetes.io/instance: balancer-production -patches: - - target: - kind: ListenerSet - name: balancer-listeners - patch: | - - op: replace - path: /spec/listeners/0/hostname - value: balancerproject.org - - op: replace - path: /spec/listeners/1/hostname - value: balancerproject.org - - target: - kind: HTTPRoute - name: balancer - patch: | - - op: replace - path: /spec/hostnames/0 - value: balancerproject.org - - target: - kind: HTTPRoute - name: balancer-http-redirect - patch: | - - op: replace - path: /spec/hostnames/0 - value: balancerproject.org \ No newline at end of file +# No routing patches. The cluster's Gateway carries the hostname and TLS, and +# the HTTPRoute inherits it — see ../../README.md. +# +# HOSTNAME above is still set here because the *application* needs to know its +# own public name (Django ALLOWED_HOSTS / CSRF_TRUSTED_ORIGINS). That's app +# config, not routing. \ No newline at end of file diff --git a/deploy/manifests/balancer/overlays/sandbox/kustomization.yaml b/deploy/manifests/balancer/overlays/sandbox/kustomization.yaml index 633c5127..074f2918 100644 --- a/deploy/manifests/balancer/overlays/sandbox/kustomization.yaml +++ b/deploy/manifests/balancer/overlays/sandbox/kustomization.yaml @@ -25,31 +25,9 @@ labels: environment: sandbox app.kubernetes.io/instance: balancer-sandbox +# No routing patches — the cluster's Gateway carries the hostname and TLS, and +# the HTTPRoute inherits it. See ../../README.md. patches: - - target: - kind: ListenerSet - name: balancer-listeners - patch: | - - op: replace - path: /spec/listeners/0/hostname - value: sandbox.balancerproject.org - - op: replace - path: /spec/listeners/1/hostname - value: sandbox.balancerproject.org - - target: - kind: HTTPRoute - name: balancer - patch: | - - op: replace - path: /spec/hostnames/0 - value: sandbox.balancerproject.org - - target: - kind: HTTPRoute - name: balancer-http-redirect - patch: | - - op: replace - path: /spec/hostnames/0 - value: sandbox.balancerproject.org - target: kind: Deployment name: balancer