feat: add gateway-api support by l0wl3vel · Pull Request #299 · metal-stack/mini-lab

l0wl3vel · 2026-05-06T12:20:19Z

Description

Add kind-cloud-controller-manager to provide Type: Loadbalancer services
Introduce envoy-gateway as the Gateway API implementation
Move metal-stack control plane kind cluster into the mini_lab_external docker network
- can't select container IP in the default docker bridge, which we need for the pre-defined *.nip.io DNS records
Kept ingress-nginx for now. Still required for Dex, Thanos, Gardener, PowerDNS

WIPs

Certificates are a bit messed up still (using default-gateway cert for grcp termination)
~~Link metal-roles pr branch to run ci in pull request~~ metal-roles PR is merged
CI is failing -> Fix route reconciliation by enabling fpm #315

Closes: #297

Requires: metal-stack/helm-charts#156 and metal-stack/metal-roles#594

Tested configurations

Sonic
Dell Sonic
Gardener (looks good, deploys correctly with metal-stack on GWAPI and Gardener components still on ingress-nginx, further testing required)
Kamaji
- non-functional. So likely a wontfix, unless it gets integreated into mini-lab. Only usable in capi-lab, which uses an old pinned version of mini-lab.

Noteworthy

- We now create a dedicated internal mini-lab docker network (mini_lab_internal) to enable explicit reproducible IP assignments for Service type: LoadBalancer. We reserve the 172.42.0.1/16 subnet for the mini_lab_internal network. mini-lab will manage the lifecycle of this network and will destroy it during `make down`.

- We made changes to the certificates. If you deployed mini-lab before you need to run `make roll-certs` to update host names in the certificates.

Used AI-Tools ✨

none

ma-hartma · 2026-05-11T07:36:52Z

Dell Sonic does actually work, but you need credentials to pull from r.metal-stack.io.

vknabel · 2026-05-28T13:47:37Z

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

l0wl3vel · 2026-05-29T12:58:10Z

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

@vknabel fixed in 629cb02

l0wl3vel · 2026-05-29T13:09:32Z

@Sven-Ric Would you mind taking a look at the network changes?

Sven-Ric · 2026-06-05T12:14:47Z

It seems like the kind node always ends up in the default kind network on a clean first run. The kind network is read from .env, which is written by env.sh. However the Makefile reads .env before env.sh is invoked and the kind node network falls back to default. Because .env is persistent the bug is masked on all subsequent runs.

On initial run:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "6530b19e41b397d41d37f6a38d6b1bbd74c9ba2b7478df95f6a6270cc84c4d0e",
                    "EndpointID": "6d56f5f0fa83330b85e0b0ebbd04175a93d8586c48492c9b545beb7eeecce015",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "MacAddress": "12:98:42:c8:e4:ec",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::2",
                    "GlobalIPv6PrefixLen": 64,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "bd976835cec0"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

On all subsequent runs:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "mini_lab_internal": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "2734b8f942cae84d8693ecd43ab3bb9d5cd71905faf992fbfe5c3df17ddc376b",
                    "EndpointID": "62f8b2a6eb379bb65f13f6441a9249417fc9ce754218a29b699cd7511b393d29",
                    "Gateway": "172.42.0.1",
                    "IPAddress": "172.42.0.2",
                    "MacAddress": "66:e7:b9:9c:2e:39",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "5b12fbbedfdc"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

l0wl3vel · 2026-06-05T17:52:10Z

Thank you so much for checking it out @Sven-Ric. Fixed in baddf29. A few people checked this PR and it worked fine but CI was failing and I had no clue why. You saved me a lot of time 😅

Gerrit91

PR looks good to me. Thanks for the effort!

Would like to read the migration path somewhere. I guess for most operators it is sufficient to deploy a Gateway controller and then set metal_control_plane_gateway_dns: "{{ metal_control_plane_ingress_dns }} and when we remove the old metal_control_plane_ingress_dns then they need to replace the old variable in their deployment repository?

Still required for Dex, Thanos, Gardener, PowerDNS

Dex should not be used anymore, let's remove the role in another PR.

Gerrit91 · 2026-06-15T11:26:20Z

        grpc \
-        masterdata-api; do
+        masterdata-api \
+        metal-api; do


Wasn't the grpc component used for metal-api gRPC already? What is the cert used for?

pretty sure it was not. iirc we used http only before

But why are there variables like this then?

# metal.yaml metal_api_grpc_certs_server_key: "{{ lookup('file', 'certs/grpc/server-key.pem') }}" ...

I don't think it was unused?

You are probably right. I was looking at https://github.com/metal-stack/mini-lab/blob/master/Makefile#L14, which is metal-apiserver not metal-api. So, yeah we were using it, just not for the cli.

I tried to just make the certificate stuff work. What would you like to see here?

One certificate (client+server) for metal-api and metal-apiserver

Seperate certs for metal-api and metal-apiserver

Seperate certs for metal-api http, metal-api grpc and metal-apiserver

I think the certificates can remain unchanged. The metal-apiserver should not need a certificate because all clients can communicate via HTTP. We just need to use a certificate for the TCP Route of the metal-api, which exposes gRPC. I am not even sure why this one has a cert, but I would try to keep it as before (probably enforced by the server to provide a cert would be my guess).

To enable fixed IPs for the Gateway API, mini-lab needs a docker network that is not the docker default bridge. Create a dedicated `mini_lab_internal` network (172.42.0.0/16) and relocate everything off the old default network bridge (172.17.0.1 / 172.18.0.0). This shifts the control-plane ingress DNS from 172.17.0.1.nip.io to 172.42.0.1.nip.io. No Gateway API migration yet Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Add a metal-api server cert (api./v2. nip.io SANs) for the Gateway to terminate TLS with, add it to roll_certs.sh, and update the grpc cert host to the Gateway LoadBalancer IP. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

l0wl3vel · 2026-07-08T12:40:13Z

Tests currently do not pass until routing propagation fix lands: #315

Add the Gateway API plumbing without wiring any application to it yet: Deploy cloud-provider-kind to provide LoadBalancer IPs. Add a `gateway` role deploying the Envoy Gateway controller, a GatewayClass, an EnvoyProxy pinned to the fixed LB IP 172.42.0.42, and the metal-control-plane Gateway. Register the role in the control-plane playbook and expose metal_control_plane_gateway_dns Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Migrate zitadel, nsq, metal-api and metal-apiserver off ingress-nginx to Gateway API. Enable http/tcp routes, point their external URLs at the Gateway LoadBalancer (172.42.0.42 / gateway_dns). Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

metal-robot Bot added this to Development May 6, 2026

l0wl3vel force-pushed the feat/gatewayapi branch 2 times, most recently from 28079c5 to f84c000 Compare May 8, 2026 14:58

l0wl3vel mentioned this pull request May 26, 2026

Use Sonic VPP image #301

Draft

9 tasks

l0wl3vel requested review from Sven-Ric and vknabel May 29, 2026 13:10

l0wl3vel force-pushed the feat/gatewayapi branch from bf2ba5a to 5aa456d Compare June 1, 2026 06:53

l0wl3vel marked this pull request as ready for review June 1, 2026 06:53

l0wl3vel requested review from a team as code owners June 1, 2026 06:53

l0wl3vel force-pushed the feat/gatewayapi branch from 5aa456d to a67cb9d Compare June 2, 2026 12:14

l0wl3vel force-pushed the feat/gatewayapi branch from e88a693 to a67cb9d Compare June 5, 2026 17:20

l0wl3vel force-pushed the feat/gatewayapi branch 4 times, most recently from 8075d54 to ffc4120 Compare June 10, 2026 04:42

Gerrit91 reviewed Jun 15, 2026

View reviewed changes

l0wl3vel force-pushed the feat/gatewayapi branch 2 times, most recently from f80647f to 5819828 Compare July 3, 2026 12:09

l0wl3vel force-pushed the feat/gatewayapi branch from dae73d5 to a78c21e Compare July 6, 2026 11:13

l0wl3vel requested a review from Gerrit91 July 6, 2026 11:49

l0wl3vel added 2 commits July 7, 2026 12:36

feat: add metal-api server certificate

3046e21

Add a metal-api server cert (api./v2. nip.io SANs) for the Gateway to terminate TLS with, add it to roll_certs.sh, and update the grpc cert host to the Gateway LoadBalancer IP. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

l0wl3vel force-pushed the feat/gatewayapi branch from a78c21e to 3e98266 Compare July 7, 2026 10:40

l0wl3vel force-pushed the feat/gatewayapi branch 2 times, most recently from adeaa03 to 8da4fe3 Compare July 8, 2026 12:39

l0wl3vel added 3 commits July 8, 2026 14:44

feat: remove ingress-nginx exposed ports

86f560d

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

l0wl3vel force-pushed the feat/gatewayapi branch from 8da4fe3 to 86f560d Compare July 8, 2026 12:44

Uh oh!

Conversation

l0wl3vel commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

WIPs

Tested configurations

Noteworthy

Used AI-Tools ✨

Uh oh!

ma-hartma commented May 11, 2026

Uh oh!

vknabel commented May 28, 2026

Uh oh!

l0wl3vel commented May 29, 2026

Uh oh!

l0wl3vel commented May 29, 2026

Uh oh!

Sven-Ric commented Jun 5, 2026

Uh oh!

l0wl3vel commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gerrit91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gerrit91 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

l0wl3vel Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Gerrit91 Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

l0wl3vel Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gerrit91 Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

l0wl3vel commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

l0wl3vel commented May 6, 2026 •

edited

Loading

l0wl3vel commented Jun 5, 2026 •

edited

Loading

l0wl3vel Jul 7, 2026 •

edited

Loading

Gerrit91 Jul 7, 2026 •

edited

Loading