From bef6f46663217c2f0d5d969bfeb49b3f9cfc4afc Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 01:40:01 +0200 Subject: [PATCH 01/24] docs: restructure deployment guide --- .../03-deployment/01_guide.mdx | 27 +++ .../03-deployment/02_initial-cluster.mdx | 9 + .../03_control-plane.mdx} | 164 ++---------------- .../03-deployment/04_partition-bootstrap.mdx | 74 ++++++++ .../03-deployment/05_partition.mdx | 50 ++++++ 5 files changed, 174 insertions(+), 150 deletions(-) create mode 100644 docs/04-For Operators/03-deployment/01_guide.mdx create mode 100644 docs/04-For Operators/03-deployment/02_initial-cluster.mdx rename docs/04-For Operators/{03-deployment-guide.mdx => 03-deployment/03_control-plane.mdx} (60%) create mode 100644 docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx create mode 100644 docs/04-For Operators/03-deployment/05_partition.mdx diff --git a/docs/04-For Operators/03-deployment/01_guide.mdx b/docs/04-For Operators/03-deployment/01_guide.mdx new file mode 100644 index 00000000..c9695c2a --- /dev/null +++ b/docs/04-For Operators/03-deployment/01_guide.mdx @@ -0,0 +1,27 @@ +--- +slug: /deployment-guide +title: Deployment Guide +sidebar_position: 1 +--- + +# Deployment Guide + +We are bootstrapping the [metal control plane](../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. + +In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). + +In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. + +This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe you the way of how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. + +:::warning +Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). +::: + +:::info +If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. However, you will probably be able to re-use some of our contents from our [metal-roles](https://github.com/metal-stack/metal-roles) repository, e.g. the Helm chart for deploying the metal control plane. +::: + +:::tip +You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this document. +::: diff --git a/docs/04-For Operators/03-deployment/02_initial-cluster.mdx b/docs/04-For Operators/03-deployment/02_initial-cluster.mdx new file mode 100644 index 00000000..6a7768d3 --- /dev/null +++ b/docs/04-For Operators/03-deployment/02_initial-cluster.mdx @@ -0,0 +1,9 @@ +--- +slug: /deployment/initial-cluster +title: Deployment - Initial Cluster +sidebar_position: 2 +--- + +# Initial Cluster + +TODO diff --git a/docs/04-For Operators/03-deployment-guide.mdx b/docs/04-For Operators/03-deployment/03_control-plane.mdx similarity index 60% rename from docs/04-For Operators/03-deployment-guide.mdx rename to docs/04-For Operators/03-deployment/03_control-plane.mdx index ce58e0e0..29bff614 100644 --- a/docs/04-For Operators/03-deployment-guide.mdx +++ b/docs/04-For Operators/03-deployment/03_control-plane.mdx @@ -1,32 +1,10 @@ --- -slug: /deployment-guide -title: Deployment Guide +slug: /deployment/control-plane +title: Deployment - Metal Control Plane sidebar_position: 3 --- -# Deployment Guide - -We are bootstrapping the [metal control plane](../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. - -In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). - -In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. - -This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe you the way of how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. - -:::warning -Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). -::: - -:::info -If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. However, you will probably be able to re-use some of our contents from our [metal-roles](https://github.com/metal-stack/metal-roles) repository, e.g. the Helm chart for deploying the metal control plane. -::: - -:::tip -You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this document. -::: - -## Metal Control Plane Deployment +# Metal Control Plane Deployment The metal control plane is typically deployed in a Kubernetes cluster. Therefore, this document will assume that you have a Kubernetes cluster ready for getting deployed. Even though it is theoretically possible to deploy metal-stack without Kubernetes, we strongly advise you to use the described method because we believe that Kubernetes gives you a lot of benefits regarding the stability and maintainability of the application deployment. @@ -73,7 +51,7 @@ At the end of this section we are gonna end up with the following files and fold └── main.yaml ``` -### Releases and Ansible Role Dependencies +## Releases and Ansible Role Dependencies As metal-stack consists of many microservices all having individual versions, we have come up with a [releases](https://github.com/metal-stack/releases) repository. It contains a YAML file (we often call it release vector) describing the fitting versions of all components for every release of metal-stack. Ansible role dependencies are also part of a metal-stack release. Both the metal-stack release vector and the metal-stack ansible-roles are shipped as OCI artifacts following a specific format that's described [here](/community/oci-artifacts). These artifacts are signed with the CI token of the metal-stack Github organization and can be verified using [cosign](https://github.com/sigstore/cosign). @@ -109,7 +87,7 @@ The public key for the validation is attached to each metal-stack release. Optional further parametrization for this module can be found in its [module documentation](https://github.com/metal-stack/ansible-common/blob/a5977fc618cd34a6cebda262602b8cbc0632500d/library/metal_stack_release_vector.py). -### Inventory +## Inventory Then, there will be an inventory for the control plane deployment in `inventories/control-plane.yaml` that adds `localhost` to the `control-plane` host group: @@ -125,7 +103,7 @@ We do this since we are deploying to Kubernetes and do not need to SSH-connect t Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases. -### Control Plane Playbook +## Control Plane Playbook Next, we will define the actual deployment playbook in a file called `deploy_metal_control_plane.yaml`. You can start with the following lines: @@ -159,7 +137,7 @@ Basically, this playbook does the following: - Deploying the postgres database for the masterdata-api (wrapped in a backup-restore-sidecar) - Applying the metal control plane helm chart -### Setup an ingress-controller +## Setup an ingress-controller As a next step we write the `ingress-controller` role for deploying an ingress-controller into the cluster. [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) is what we use. If you want to use another ingress-controller, you need to parametrize the metal roles carefully. When you just use ingress-nginx, make sure to also deploy it to the default namespace ingress-nginx. @@ -180,7 +158,7 @@ This is how your `roles/ingress-controller/tasks/main.yaml` could look like: The [ansible-common](https://github.com/metal-stack/ansible-common) repository contains very general roles and modules that you can also use when extending your deployment further. ::: -### Deployment Parametrization +## Deployment Parametrization Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: @@ -189,7 +167,7 @@ Now you can parametrize the referenced roles to fit your environment. The role p metal_control_plane_ingress_dns: # if you do not have a DNS entry, you could also start with .nip.io ``` -### Providing Certificates +## Providing Certificates We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. @@ -404,7 +382,7 @@ metal_api_grpc_certs_ca_cert: "{{ lookup('file', 'certs/ca.pem') }}" For the actual communication between the metal-api and the user clients (REST API, runs over the ingress-controller you deployed before), you can simply deploy a tool like [cert-manager](https://github.com/cert-manager/cert-manager) into your Kubernetes cluster, which will automatically provide your ingress domains with Let's Encrypt certificates. ::: -### Running the Deployment +## Running the Deployment Finally, it should be possible to run the deployment through a Docker container. Make sure to have the [Kubeconfig file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) of your cluster and set the path in the following command accordingly: @@ -439,7 +417,7 @@ docker run --rm -it \ If you are having issues regarding the deployment take a look at the [troubleshoot document](./06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: -### Providing Images +## Providing Images After the deployment has finished (hopefully without any issues!), you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: @@ -465,7 +443,7 @@ Then, re-run the deployment to apply your changes. Our playbooks are idempotent. Image versions should be regularly checked for updates. ::: -### Setting up metalctl +## Setting up metalctl You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). The configuration for `metalctl` should look like this: @@ -496,7 +474,7 @@ ubuntu-19.10.20200331 Ubuntu 19.10 20200331 Ubuntu 19.10 202 The basic principles of how the metal control plane can be deployed should now be clear. It is now up to you to move the deployment execution into your CI and add things like certificates for the ingress-controller and NSQ. -### Setting Up the backup-restore-sidecar +## Setting Up the backup-restore-sidecar The backup-restore-sidecar can come in very handy when you want to add another layer of security to the metal-stack databases in your Kubernetes cluster. The sidecar takes backups of the metal databases in small time intervals and stores them in a blobstore of a cloud provider. For each database that will be backed up, a lifecycle rule is established. The backup mechanism is deactivated by default and must be activated by the operator. This way your metal-stack setup can even survive the deletion of your Kubernetes control plane cluster (including all volumes getting lost). After re-deploying metal-stack to another Kubernetes clusters, the databases come up with the latest backup data in a matter of seconds. @@ -504,7 +482,7 @@ Encryption can be enabled for the backups by providing an AES-256 encryption key Checkout the [role documentation](https://github.com/metal-stack/metal-roles/tree/master/control-plane) of the individual databases to find out how to configure the sidecar properly. You can also try out the mechanism from the [backup-restore-sidecar](https://github.com/metal-stack/backup-restore-sidecar) repository. -### Auth +## Auth metal-stack currently supports two authentication methods: @@ -532,117 +510,3 @@ metal_masterdata_api_tenants: namespace_max_length: 20 description: ``` - -## Bootstrapping a Partition - -### Out-Of-Band-Network - -To be able to deploy and maintain a metal-stack partition, you need to bootstrap the Out-Of-Band-Network first. -Some considerations must be made to fulfill the requirements of our infrastructure, a partition is designed to be: - -- secure -- fully routable (BGP) -- scalable -- resilient -- deployable via CI/CD jobs -- accessible from the internet from specific IPs - -In order to accomplish this task remotely and in a nearly automatic manner, you have to bootstrap the components in this order: - -1. management firewalls -2. management servers -3. management spines -4. management leaves -5. leaves, spines and exits - -This document assumes that all cabling is done. Here is a quick overview of the architecture: - -![Out-of-Band-Network](mgmt_net_layer3.png) - -### Management Firewalls - -As you can see, the management firewalls are the first bastion hosts in a partition to provide access to our infrastructure. There are two of them in each partition to guarantee high availability and load balancing. The very first configuration of these routers has to be done manually to solve the chicken and egg problem that you need the management firewalls in place to deploy the partition. Manually means that we generate a configuration template with ansible that we deploy with copy/paste, and the load, through the machine console. Once the management server has been deployed, we are able to deploy this configuration via CI runner and ansible. For this you need the user and the ssh-key, which is deployed with the configuration file mentioned above. -The Edgerouters has to fulfill some requirements including: - -- provide and restrict access to the Out-Of-Band-Network from the internet with a firewall ruleset -- provide destination NAT to the management server and its IPMI interface -- provide Onie Boot and ztp via DHCP options for the management spine -- provide DHCP management addresses for management spine, management server and ipmi interface of the management server -- Hairpin-NAT for the management server to access itself via its public IP, needed by the CI runner to delegate CI Jobs. -- propagate a default gateway via BGP - -### Management Servers - -The second bastion hosts are the management servers. They are the main bootstrapping components of the Out-Of-Band-Network. They also act as jump hosts for all components in a partition. Once they are installed and deployed, we are able to bootstrap all the other components. To bootstrap the management servers, we generate an ISO image which will automatically install an OS and an ansible user with ssh keys. It is preconfigured with a preseed file to allow an unattended OS installation for our needs. This is why we need remote access to the IPMI interface of the management servers: The generated ISO is attached via the virtual media function of the BMC. After that, all we have to do is boot from that virtual CD-ROM and wait for the installation to finish. Deployment jobs (Gitlab-CI) in a partition are delegated to the appropriate management servers, therefore we need a CI runner active on each management server. - -After the CI runner has been installed, you can trigger your Playbooks from the the CI. The Ansible-Playbooks have to make sure that these functionalities are present on the management servers: - -- Prometheus and exporters -- CI runner -- metal-bmc -- image-cache -- simple webserver to provide images -- [Onie Boot](https://opencomputeproject.github.io/onie/) and ZTP -- DHCP addresses for ipmi interfaces of the workers -- DHCP addresses for switches - -### Management Spines - -:::tip -If you are using SONiC switches, you should make use of Zero Touch Provisioning and Onie Boot -::: - -The purpose of these switches is to connect the management interfaces of all switches to the management servers. The management spine's own management interface is connected to the management firewall for the bootstrapping of the management spine itself. The management firewall will provide a DHCP address and DHCP options to start SONiC's [Zero Touch Provisioning](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md); the images for all switches are downloaded from the management server (nginx container). -Each management leaf is connected to both management spines to provide redundant connectivity to both management servers. BGP is used as a routing protocol such that, when a link goes down, an alternate path is used. -In the picture above you can see that there are also switch management interfaces connected to the management spine. This has to be done so that we can bootstrap these switches; the management spine relays the DHCP requests from these switches to the management servers so that they are able to Onie Boot and get their ZTP scripts. - -### Management Leaves - -All workers have to be connected with their IPMI/BMC interface to the management leaves to get DHCP addresses from the management server. The management leaves are relaying those DHCP requests to the management server which will answer the requests and provide IPs from a given range. The management interfaces of the management leaves also have to be reachable from the management server, and need to get their IP address via DHCP for the bootstrapping process. - -In the example setup, these interfaces are connected to an end-of-row-switch which aggregates them and connects them to the management spines with a fiber-optics connection. If you can reach the management spines from the management leaves with copper cables, you do not need the end of row switch. After the initial bootstrapping, the management interfaces of the management leaves continue to be used for access to the switches' command line, and for subsequent OS updates. (update=reset+bootrap+deployment) - -## Partition Deployment - -## Gardener with metal-stack - -If you want to deploy metal-stack as a cloud provider for [Gardener](https://gardener.cloud/), you should follow the regular Gardener installation instructions and setup a Gardener cluster first. It's perfectly fine to setup the Gardener cluster in the same cluster that you use for hosting metal-stack. - -You can find installation instructions for Gardener on the Gardener website beneath [docs](https://gardener.cloud/docs/). metal-stack is an out-of-tree provider and therefore you will not find example files for metal-stack resources in the Gardener repositories. The following list describes the resources and components that you need to deploy into the Gardener cluster in order to make Gardener work with metal-stack: - -:::warning -The following list assumes you have Gardener installed in a Kubernetes cluster and that you have a basic understanding of how Gardener works. If you need further help with the following steps, you can also come and ask in our Slack channel. -::: - -1. Deploy the [validator](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/validator-metal) from the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) repository to your cluster via Helm -1. Add a [cloud profile](https://github.com/gardener/gardener/blob/v1.3.3/example/30-cloudprofile.yaml) called `metal` containing all your machine images, machine types and regions (region names can be chosen freely, the zone names need to match your partition names) together with our metal-stack-specific provider config as defined [here](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/pkg/apis/metal/v1alpha1/types_cloudprofile.go) -1. Register the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) controller by deploying the [controller-registration](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/example/controller-registration.yaml) into your Gardener cluster, parametrize the embedded chart in the controller registration's values section if necessary ([this](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/provider-metal) is the corresponding values file) -1. metal-stack does not provide an own backup storage infrastructure for now. If you want to enable ETCD backups (which you should do because metal-stack also does not have persistent storage out of the box, which makes these backups even more valuable), you should deploy an extension-provider of another cloud provider and configure it to only reconcile the backup buckets (you can reference this backup infrastructure used for the metal shoot in the shoot spec) -1. Register the [os-extension-provider-metal](https://github.com/metal-stack/os-metal-extension) controller by deploying the [controller-registration](https://github.com/metal-stack/os-metal-extension/blob/v0.4.1/example/controller-registration.yaml) into your Gardener cluster, this controller can transform the operating system configuration from Gardener into Ignition user data -1. You need to use the Gardener's [networking-calico](https://github.com/gardener/gardener-extension-networking-calico) controller for setting up shoot CNI, you will have to put specific provider configuration into the shoot spec to make it work with metal-stack: - ```yaml - networking: - type: calico - # we can peer with the frr within 10.244.0.0/16, which we do with the metallb - # the networks for the shoot need to be disjunct with the networks of the seed, otherwise the VPN connection will not work properly - # the seeds are typically deployed with podCIDR 10.244.128.0/18 and serviceCIDR 10.244.192.0/18 - # the shoots are typically deployed with podCIDR 10.244.0.0/18 and serviceCIDR 10.244.64.0/18 - pods: 10.244.0.0/18 - services: 10.244.64.0/18 - providerConfig: - apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 - kind: NetworkConfig - backend: vxlan - ipv4: - pool: vxlan - mode: Always - autoDetectionMethod: interface=lo - typha: - enabled: false - ``` -1. For your seed cluster you will need to provide the provider secret for metal-stack containing the key `metalAPIHMac`, which is the API HMAC to grant editor access to the metal-api -1. Checkout our current provider configuration for [infrastructure](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_infrastructure.go) and [control-plane](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_controlplane.go) before deploying your shoot - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. -::: diff --git a/docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx b/docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx new file mode 100644 index 00000000..f4378c3e --- /dev/null +++ b/docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx @@ -0,0 +1,74 @@ +--- +slug: /deployment/partition-bootstrap +title: Deployment - Bootstrapping a Partition +sidebar_position: 4 +--- + +# Bootstrapping a Partition + +## Out-Of-Band-Network + +To be able to deploy and maintain a metal-stack partition, you need to bootstrap the Out-Of-Band-Network first. +Some considerations must be made to fulfill the requirements of our infrastructure, a partition is designed to be: + +- secure +- fully routable (BGP) +- scalable +- resilient +- deployable via CI/CD jobs +- accessible from the internet from specific IPs + +In order to accomplish this task remotely and in a nearly automatic manner, you have to bootstrap the components in this order: + +1. management firewalls +2. management servers +3. management spines +4. management leaves +5. leaves, spines and exits + +This document assumes that all cabling is done. Here is a quick overview of the architecture: + +![Out-of-Band-Network](mgmt_net_layer3.png) + +## Management Firewalls + +As you can see, the management firewalls are the first bastion hosts in a partition to provide access to our infrastructure. There are two of them in each partition to guarantee high availability and load balancing. The very first configuration of these routers has to be done manually to solve the chicken and egg problem that you need the management firewalls in place to deploy the partition. Manually means that we generate a configuration template with ansible that we deploy with copy/paste, and the load, through the machine console. Once the management server has been deployed, we are able to deploy this configuration via CI runner and ansible. For this you need the user and the ssh-key, which is deployed with the configuration file mentioned above. +The Edgerouters has to fulfill some requirements including: + +- provide and restrict access to the Out-Of-Band-Network from the internet with a firewall ruleset +- provide destination NAT to the management server and its IPMI interface +- provide Onie Boot and ztp via DHCP options for the management spine +- provide DHCP management addresses for management spine, management server and ipmi interface of the management server +- Hairpin-NAT for the management server to access itself via its public IP, needed by the CI runner to delegate CI Jobs. +- propagate a default gateway via BGP + +## Management Servers + +The second bastion hosts are the management servers. They are the main bootstrapping components of the Out-Of-Band-Network. They also act as jump hosts for all components in a partition. Once they are installed and deployed, we are able to bootstrap all the other components. To bootstrap the management servers, we generate an ISO image which will automatically install an OS and an ansible user with ssh keys. It is preconfigured with a preseed file to allow an unattended OS installation for our needs. This is why we need remote access to the IPMI interface of the management servers: The generated ISO is attached via the virtual media function of the BMC. After that, all we have to do is boot from that virtual CD-ROM and wait for the installation to finish. Deployment jobs (Gitlab-CI) in a partition are delegated to the appropriate management servers, therefore we need a CI runner active on each management server. + +After the CI runner has been installed, you can trigger your Playbooks from the the CI. The Ansible-Playbooks have to make sure that these functionalities are present on the management servers: + +- Prometheus and exporters +- CI runner +- metal-bmc +- image-cache +- simple webserver to provide images +- [Onie Boot](https://opencomputeproject.github.io/onie/) and ZTP +- DHCP addresses for ipmi interfaces of the workers +- DHCP addresses for switches + +## Management Spines + +:::tip +If you are using SONiC switches, you should make use of Zero Touch Provisioning and Onie Boot +::: + +The purpose of these switches is to connect the management interfaces of all switches to the management servers. The management spine's own management interface is connected to the management firewall for the bootstrapping of the management spine itself. The management firewall will provide a DHCP address and DHCP options to start SONiC's [Zero Touch Provisioning](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md); the images for all switches are downloaded from the management server (nginx container). +Each management leaf is connected to both management spines to provide redundant connectivity to both management servers. BGP is used as a routing protocol such that, when a link goes down, an alternate path is used. +In the picture above you can see that there are also switch management interfaces connected to the management spine. This has to be done so that we can bootstrap these switches; the management spine relays the DHCP requests from these switches to the management servers so that they are able to Onie Boot and get their ZTP scripts. + +## Management Leaves + +All workers have to be connected with their IPMI/BMC interface to the management leaves to get DHCP addresses from the management server. The management leaves are relaying those DHCP requests to the management server which will answer the requests and provide IPs from a given range. The management interfaces of the management leaves also have to be reachable from the management server, and need to get their IP address via DHCP for the bootstrapping process. + +In the example setup, these interfaces are connected to an end-of-row-switch which aggregates them and connects them to the management spines with a fiber-optics connection. If you can reach the management spines from the management leaves with copper cables, you do not need the end of row switch. After the initial bootstrapping, the management interfaces of the management leaves continue to be used for access to the switches' command line, and for subsequent OS updates. (update=reset+bootrap+deployment) diff --git a/docs/04-For Operators/03-deployment/05_partition.mdx b/docs/04-For Operators/03-deployment/05_partition.mdx new file mode 100644 index 00000000..d0fa7e0c --- /dev/null +++ b/docs/04-For Operators/03-deployment/05_partition.mdx @@ -0,0 +1,50 @@ +--- +slug: /deployment/partition +title: Deployment - Partition +sidebar_position: 5 +--- + +# Partition Deployment + +## Gardener with metal-stack + +If you want to deploy metal-stack as a cloud provider for [Gardener](https://gardener.cloud/), you should follow the regular Gardener installation instructions and setup a Gardener cluster first. It's perfectly fine to setup the Gardener cluster in the same cluster that you use for hosting metal-stack. + +You can find installation instructions for Gardener on the Gardener website beneath [docs](https://gardener.cloud/docs/). metal-stack is an out-of-tree provider and therefore you will not find example files for metal-stack resources in the Gardener repositories. The following list describes the resources and components that you need to deploy into the Gardener cluster in order to make Gardener work with metal-stack: + +:::warning +The following list assumes you have Gardener installed in a Kubernetes cluster and that you have a basic understanding of how Gardener works. If you need further help with the following steps, you can also come and ask in our Slack channel. +::: + +1. Deploy the [validator](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/validator-metal) from the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) repository to your cluster via Helm +1. Add a [cloud profile](https://github.com/gardener/gardener/blob/v1.3.3/example/30-cloudprofile.yaml) called `metal` containing all your machine images, machine types and regions (region names can be chosen freely, the zone names need to match your partition names) together with our metal-stack-specific provider config as defined [here](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/pkg/apis/metal/v1alpha1/types_cloudprofile.go) +1. Register the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) controller by deploying the [controller-registration](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/example/controller-registration.yaml) into your Gardener cluster, parametrize the embedded chart in the controller registration's values section if necessary ([this](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/provider-metal) is the corresponding values file) +1. metal-stack does not provide an own backup storage infrastructure for now. If you want to enable ETCD backups (which you should do because metal-stack also does not have persistent storage out of the box, which makes these backups even more valuable), you should deploy an extension-provider of another cloud provider and configure it to only reconcile the backup buckets (you can reference this backup infrastructure used for the metal shoot in the shoot spec) +1. Register the [os-extension-provider-metal](https://github.com/metal-stack/os-metal-extension) controller by deploying the [controller-registration](https://github.com/metal-stack/os-metal-extension/blob/v0.4.1/example/controller-registration.yaml) into your Gardener cluster, this controller can transform the operating system configuration from Gardener into Ignition user data +1. You need to use the Gardener's [networking-calico](https://github.com/gardener/gardener-extension-networking-calico) controller for setting up shoot CNI, you will have to put specific provider configuration into the shoot spec to make it work with metal-stack: + ```yaml + networking: + type: calico + # we can peer with the frr within 10.244.0.0/16, which we do with the metallb + # the networks for the shoot need to be disjunct with the networks of the seed, otherwise the VPN connection will not work properly + # the seeds are typically deployed with podCIDR 10.244.128.0/18 and serviceCIDR 10.244.192.0/18 + # the shoots are typically deployed with podCIDR 10.244.0.0/18 and serviceCIDR 10.244.64.0/18 + pods: 10.244.0.0/18 + services: 10.244.64.0/18 + providerConfig: + apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 + kind: NetworkConfig + backend: vxlan + ipv4: + pool: vxlan + mode: Always + autoDetectionMethod: interface=lo + typha: + enabled: false + ``` +1. For your seed cluster you will need to provide the provider secret for metal-stack containing the key `metalAPIHMac`, which is the API HMAC to grant editor access to the metal-api +1. Checkout our current provider configuration for [infrastructure](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_infrastructure.go) and [control-plane](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_controlplane.go) before deploying your shoot + +:::tip +We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. +::: From 8f30ec094d2998c5f2f9f2c51786269d67b0cc61 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 02:01:35 +0200 Subject: [PATCH 02/24] fix links --- .../03-deployment/{01_guide.mdx => 01_guide.md} | 2 +- .../{02_initial-cluster.mdx => 02_initial-cluster.md} | 0 docs/04-For Operators/03-deployment/03_control-plane.mdx | 4 ++-- .../{04_partition-bootstrap.mdx => 04_partition-bootstrap.md} | 0 .../03-deployment/{05_partition.mdx => 05_partition.md} | 0 5 files changed, 3 insertions(+), 3 deletions(-) rename docs/04-For Operators/03-deployment/{01_guide.mdx => 01_guide.md} (90%) rename docs/04-For Operators/03-deployment/{02_initial-cluster.mdx => 02_initial-cluster.md} (100%) rename docs/04-For Operators/03-deployment/{04_partition-bootstrap.mdx => 04_partition-bootstrap.md} (100%) rename docs/04-For Operators/03-deployment/{05_partition.mdx => 05_partition.md} (100%) diff --git a/docs/04-For Operators/03-deployment/01_guide.mdx b/docs/04-For Operators/03-deployment/01_guide.md similarity index 90% rename from docs/04-For Operators/03-deployment/01_guide.mdx rename to docs/04-For Operators/03-deployment/01_guide.md index c9695c2a..5af8b907 100644 --- a/docs/04-For Operators/03-deployment/01_guide.mdx +++ b/docs/04-For Operators/03-deployment/01_guide.md @@ -6,7 +6,7 @@ sidebar_position: 1 # Deployment Guide -We are bootstrapping the [metal control plane](../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. +We are bootstrapping the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). diff --git a/docs/04-For Operators/03-deployment/02_initial-cluster.mdx b/docs/04-For Operators/03-deployment/02_initial-cluster.md similarity index 100% rename from docs/04-For Operators/03-deployment/02_initial-cluster.mdx rename to docs/04-For Operators/03-deployment/02_initial-cluster.md diff --git a/docs/04-For Operators/03-deployment/03_control-plane.mdx b/docs/04-For Operators/03-deployment/03_control-plane.mdx index 29bff614..15e69fc7 100644 --- a/docs/04-For Operators/03-deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-deployment/03_control-plane.mdx @@ -9,7 +9,7 @@ sidebar_position: 3 The metal control plane is typically deployed in a Kubernetes cluster. Therefore, this document will assume that you have a Kubernetes cluster ready for getting deployed. Even though it is theoretically possible to deploy metal-stack without Kubernetes, we strongly advise you to use the described method because we believe that Kubernetes gives you a lot of benefits regarding the stability and maintainability of the application deployment. :::tip -For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../05-Concepts/01-architecture.mdx#target-deployment-platforms). +For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). ::: Let's start off with a fresh folder for your deployment: @@ -171,7 +171,7 @@ metal_control_plane_ingress_dns: # if you do not have a DNS en We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. -For the very basic setup you will need to create self-signed certificates for the communication between the following components (see [architecture](../05-Concepts/01-architecture.mdx) document): +For the very basic setup you will need to create self-signed certificates for the communication between the following components (see [architecture](../../05-Concepts/01-architecture.mdx) document): - [metal-api](https://github.com/metal-stack/metal-api) and [masterdata-api](https://github.com/metal-stack/masterdata-api) (in-cluster traffic communication) - [metal-api](https://github.com/metal-stack/metal-api) and [metal-hammer](https://github.com/metal-stack/metal-hammer) (partition to control plane communication) diff --git a/docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx b/docs/04-For Operators/03-deployment/04_partition-bootstrap.md similarity index 100% rename from docs/04-For Operators/03-deployment/04_partition-bootstrap.mdx rename to docs/04-For Operators/03-deployment/04_partition-bootstrap.md diff --git a/docs/04-For Operators/03-deployment/05_partition.mdx b/docs/04-For Operators/03-deployment/05_partition.md similarity index 100% rename from docs/04-For Operators/03-deployment/05_partition.mdx rename to docs/04-For Operators/03-deployment/05_partition.md From 7ee8f744e912bf37fbc9e9240b204b8baec489dc Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 02:05:20 +0200 Subject: [PATCH 03/24] move images --- .../{ => 03-deployment}/mgmt_net_layer3.drawio | 0 .../{ => 03-deployment}/mgmt_net_layer3.png | Bin 2 files changed, 0 insertions(+), 0 deletions(-) rename docs/04-For Operators/{ => 03-deployment}/mgmt_net_layer3.drawio (100%) rename docs/04-For Operators/{ => 03-deployment}/mgmt_net_layer3.png (100%) diff --git a/docs/04-For Operators/mgmt_net_layer3.drawio b/docs/04-For Operators/03-deployment/mgmt_net_layer3.drawio similarity index 100% rename from docs/04-For Operators/mgmt_net_layer3.drawio rename to docs/04-For Operators/03-deployment/mgmt_net_layer3.drawio diff --git a/docs/04-For Operators/mgmt_net_layer3.png b/docs/04-For Operators/03-deployment/mgmt_net_layer3.png similarity index 100% rename from docs/04-For Operators/mgmt_net_layer3.png rename to docs/04-For Operators/03-deployment/mgmt_net_layer3.png From ada4e1aab5f99bb9fb512c5ddd0826734c9cb653 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 02:06:37 +0200 Subject: [PATCH 04/24] fix link --- docs/04-For Operators/03-deployment/03_control-plane.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/04-For Operators/03-deployment/03_control-plane.mdx b/docs/04-For Operators/03-deployment/03_control-plane.mdx index 15e69fc7..8d536cb4 100644 --- a/docs/04-For Operators/03-deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-deployment/03_control-plane.mdx @@ -414,7 +414,7 @@ docker run --rm -it \ ``` :::tip -If you are having issues regarding the deployment take a look at the [troubleshoot document](./06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! +If you are having issues regarding the deployment take a look at the [troubleshoot document](../06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: ## Providing Images From c1413697fbe3745fe6237cbc4321b8c638c03e19 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 02:12:24 +0200 Subject: [PATCH 05/24] fix link --- docs/05-Concepts/04-Kubernetes/01-gardener.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/05-Concepts/04-Kubernetes/01-gardener.md b/docs/05-Concepts/04-Kubernetes/01-gardener.md index 45a33837..394af2ba 100644 --- a/docs/05-Concepts/04-Kubernetes/01-gardener.md +++ b/docs/05-Concepts/04-Kubernetes/01-gardener.md @@ -59,7 +59,7 @@ Every Kubernetes cluster that is fully provisioned and managed by Gardener is ca During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. This section briefly describes the controllers implemented by metal-stack to allow the creation of a Kubernetes cluster on metal-stack infrastructure. -If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-deployment-guide.mdx#gardener-with-metal-stack). +If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-deployment/05_partition.md#gardener-with-metal-stack). ### gardener-extension-provider-metal From 1fb7d8393b29ac8a6152aa9214ffe62b4a7b18f0 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 02:15:16 +0200 Subject: [PATCH 06/24] sidebar titles --- .../{03-deployment => 03-Deployment}/01_guide.md | 0 .../02_initial-cluster.md | 2 +- .../03_control-plane.mdx | 2 +- .../04_partition-bootstrap.md | 2 +- .../05_partition.md | 2 +- .../mgmt_net_layer3.drawio | 0 .../mgmt_net_layer3.png | Bin 7 files changed, 4 insertions(+), 4 deletions(-) rename docs/04-For Operators/{03-deployment => 03-Deployment}/01_guide.md (100%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/02_initial-cluster.md (70%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/03_control-plane.mdx (99%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/04_partition-bootstrap.md (99%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/05_partition.md (99%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/mgmt_net_layer3.drawio (100%) rename docs/04-For Operators/{03-deployment => 03-Deployment}/mgmt_net_layer3.png (100%) diff --git a/docs/04-For Operators/03-deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md similarity index 100% rename from docs/04-For Operators/03-deployment/01_guide.md rename to docs/04-For Operators/03-Deployment/01_guide.md diff --git a/docs/04-For Operators/03-deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md similarity index 70% rename from docs/04-For Operators/03-deployment/02_initial-cluster.md rename to docs/04-For Operators/03-Deployment/02_initial-cluster.md index 6a7768d3..f25fcfa0 100644 --- a/docs/04-For Operators/03-deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -1,6 +1,6 @@ --- slug: /deployment/initial-cluster -title: Deployment - Initial Cluster +title: Initial Cluster sidebar_position: 2 --- diff --git a/docs/04-For Operators/03-deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx similarity index 99% rename from docs/04-For Operators/03-deployment/03_control-plane.mdx rename to docs/04-For Operators/03-Deployment/03_control-plane.mdx index 8d536cb4..2777db63 100644 --- a/docs/04-For Operators/03-deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -1,6 +1,6 @@ --- slug: /deployment/control-plane -title: Deployment - Metal Control Plane +title: Metal Control Plane sidebar_position: 3 --- diff --git a/docs/04-For Operators/03-deployment/04_partition-bootstrap.md b/docs/04-For Operators/03-Deployment/04_partition-bootstrap.md similarity index 99% rename from docs/04-For Operators/03-deployment/04_partition-bootstrap.md rename to docs/04-For Operators/03-Deployment/04_partition-bootstrap.md index f4378c3e..00f8a00e 100644 --- a/docs/04-For Operators/03-deployment/04_partition-bootstrap.md +++ b/docs/04-For Operators/03-Deployment/04_partition-bootstrap.md @@ -1,6 +1,6 @@ --- slug: /deployment/partition-bootstrap -title: Deployment - Bootstrapping a Partition +title: Bootstrapping a Partition sidebar_position: 4 --- diff --git a/docs/04-For Operators/03-deployment/05_partition.md b/docs/04-For Operators/03-Deployment/05_partition.md similarity index 99% rename from docs/04-For Operators/03-deployment/05_partition.md rename to docs/04-For Operators/03-Deployment/05_partition.md index d0fa7e0c..2dc9585b 100644 --- a/docs/04-For Operators/03-deployment/05_partition.md +++ b/docs/04-For Operators/03-Deployment/05_partition.md @@ -1,6 +1,6 @@ --- slug: /deployment/partition -title: Deployment - Partition +title: Partition sidebar_position: 5 --- diff --git a/docs/04-For Operators/03-deployment/mgmt_net_layer3.drawio b/docs/04-For Operators/03-Deployment/mgmt_net_layer3.drawio similarity index 100% rename from docs/04-For Operators/03-deployment/mgmt_net_layer3.drawio rename to docs/04-For Operators/03-Deployment/mgmt_net_layer3.drawio diff --git a/docs/04-For Operators/03-deployment/mgmt_net_layer3.png b/docs/04-For Operators/03-Deployment/mgmt_net_layer3.png similarity index 100% rename from docs/04-For Operators/03-deployment/mgmt_net_layer3.png rename to docs/04-For Operators/03-Deployment/mgmt_net_layer3.png From 8a4954c7989f4eb28cd2005412ecb455e4bd4bf4 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 03:35:01 +0200 Subject: [PATCH 07/24] remove outdated gardener docs and cleanup kubernetes concepts --- .../03-Deployment/01_guide.md | 2 +- .../03-Deployment/02_initial-cluster.md | 18 ++++++- ...partition-bootstrap.md => 04_partition.md} | 2 +- .../03-Deployment/05_gardener.md | 29 ++++++++++ .../03-Deployment/05_partition.md | 50 ----------------- docs/05-Concepts/04-Kubernetes/01-gardener.md | 54 +------------------ 6 files changed, 49 insertions(+), 106 deletions(-) rename docs/04-For Operators/03-Deployment/{04_partition-bootstrap.md => 04_partition.md} (99%) create mode 100644 docs/04-For Operators/03-Deployment/05_gardener.md delete mode 100644 docs/04-For Operators/03-Deployment/05_partition.md diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 5af8b907..94ba964f 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -1,6 +1,6 @@ --- slug: /deployment-guide -title: Deployment Guide +title: Guide sidebar_position: 1 --- diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index f25fcfa0..5e6387c6 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -6,4 +6,20 @@ sidebar_position: 2 # Initial Cluster -TODO +Before creating the `Garden cluster`, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure, where both the metal-stack control plane and Gardener are deployed. + +## Suggestions for the Initial Cluster + +The initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or a self-hosted solution. Some common options: + +- **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or Terraform. +- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. + - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 + +## Next Steps + +Once the initial cluster is in place, the deployment continues with: + +1. **[Metal Control Plane Deploymen](./03_control-plane.mdx)** — Deploy the metal-stack control plane into the initial cluster. +2. **[Partition Deployment](./04_partition.md)** — Set up a partition +3. **[Gardener Setup](./05_gardener.md)** — Configure Gardener to use metal-stack as a provider. diff --git a/docs/04-For Operators/03-Deployment/04_partition-bootstrap.md b/docs/04-For Operators/03-Deployment/04_partition.md similarity index 99% rename from docs/04-For Operators/03-Deployment/04_partition-bootstrap.md rename to docs/04-For Operators/03-Deployment/04_partition.md index 00f8a00e..45f7d23f 100644 --- a/docs/04-For Operators/03-Deployment/04_partition-bootstrap.md +++ b/docs/04-For Operators/03-Deployment/04_partition.md @@ -1,6 +1,6 @@ --- slug: /deployment/partition-bootstrap -title: Bootstrapping a Partition +title: Partition sidebar_position: 4 --- diff --git a/docs/04-For Operators/03-Deployment/05_gardener.md b/docs/04-For Operators/03-Deployment/05_gardener.md new file mode 100644 index 00000000..c8bb3022 --- /dev/null +++ b/docs/04-For Operators/03-Deployment/05_gardener.md @@ -0,0 +1,29 @@ +--- +slug: /deployment/partition +title: Gardener +sidebar_position: 5 +--- + +# Gardener with metal-stack + +This guide sets up metal-stack as a cloud provider for [Gardener](https://gardener.cloud/). +Precondition is a metal-stack deployment on the same initial cluster as Gardener, as described in the [control-plane](./03_control-plane.mdx) deployment guide. + +Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). + +The roles deploy the following components: + +- virtual garden +- Gardener control plane components +- soil cluster +- managed seed cluster (into the metal-stack partition) + +In summary, this results in the following: + +- `Garden cluster` created in the initial cluster +- `soil cluster` created in the initial cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition +- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from + +:::tip +We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. +::: diff --git a/docs/04-For Operators/03-Deployment/05_partition.md b/docs/04-For Operators/03-Deployment/05_partition.md deleted file mode 100644 index 2dc9585b..00000000 --- a/docs/04-For Operators/03-Deployment/05_partition.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -slug: /deployment/partition -title: Partition -sidebar_position: 5 ---- - -# Partition Deployment - -## Gardener with metal-stack - -If you want to deploy metal-stack as a cloud provider for [Gardener](https://gardener.cloud/), you should follow the regular Gardener installation instructions and setup a Gardener cluster first. It's perfectly fine to setup the Gardener cluster in the same cluster that you use for hosting metal-stack. - -You can find installation instructions for Gardener on the Gardener website beneath [docs](https://gardener.cloud/docs/). metal-stack is an out-of-tree provider and therefore you will not find example files for metal-stack resources in the Gardener repositories. The following list describes the resources and components that you need to deploy into the Gardener cluster in order to make Gardener work with metal-stack: - -:::warning -The following list assumes you have Gardener installed in a Kubernetes cluster and that you have a basic understanding of how Gardener works. If you need further help with the following steps, you can also come and ask in our Slack channel. -::: - -1. Deploy the [validator](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/validator-metal) from the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) repository to your cluster via Helm -1. Add a [cloud profile](https://github.com/gardener/gardener/blob/v1.3.3/example/30-cloudprofile.yaml) called `metal` containing all your machine images, machine types and regions (region names can be chosen freely, the zone names need to match your partition names) together with our metal-stack-specific provider config as defined [here](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/pkg/apis/metal/v1alpha1/types_cloudprofile.go) -1. Register the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) controller by deploying the [controller-registration](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/example/controller-registration.yaml) into your Gardener cluster, parametrize the embedded chart in the controller registration's values section if necessary ([this](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/provider-metal) is the corresponding values file) -1. metal-stack does not provide an own backup storage infrastructure for now. If you want to enable ETCD backups (which you should do because metal-stack also does not have persistent storage out of the box, which makes these backups even more valuable), you should deploy an extension-provider of another cloud provider and configure it to only reconcile the backup buckets (you can reference this backup infrastructure used for the metal shoot in the shoot spec) -1. Register the [os-extension-provider-metal](https://github.com/metal-stack/os-metal-extension) controller by deploying the [controller-registration](https://github.com/metal-stack/os-metal-extension/blob/v0.4.1/example/controller-registration.yaml) into your Gardener cluster, this controller can transform the operating system configuration from Gardener into Ignition user data -1. You need to use the Gardener's [networking-calico](https://github.com/gardener/gardener-extension-networking-calico) controller for setting up shoot CNI, you will have to put specific provider configuration into the shoot spec to make it work with metal-stack: - ```yaml - networking: - type: calico - # we can peer with the frr within 10.244.0.0/16, which we do with the metallb - # the networks for the shoot need to be disjunct with the networks of the seed, otherwise the VPN connection will not work properly - # the seeds are typically deployed with podCIDR 10.244.128.0/18 and serviceCIDR 10.244.192.0/18 - # the shoots are typically deployed with podCIDR 10.244.0.0/18 and serviceCIDR 10.244.64.0/18 - pods: 10.244.0.0/18 - services: 10.244.64.0/18 - providerConfig: - apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 - kind: NetworkConfig - backend: vxlan - ipv4: - pool: vxlan - mode: Always - autoDetectionMethod: interface=lo - typha: - enabled: false - ``` -1. For your seed cluster you will need to provide the provider secret for metal-stack containing the key `metalAPIHMac`, which is the API HMAC to grant editor access to the metal-api -1. Checkout our current provider configuration for [infrastructure](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_infrastructure.go) and [control-plane](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_controlplane.go) before deploying your shoot - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. -::: diff --git a/docs/05-Concepts/04-Kubernetes/01-gardener.md b/docs/05-Concepts/04-Kubernetes/01-gardener.md index 394af2ba..879ee24d 100644 --- a/docs/05-Concepts/04-Kubernetes/01-gardener.md +++ b/docs/05-Concepts/04-Kubernetes/01-gardener.md @@ -8,7 +8,7 @@ sidebar_position: 1 [Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. -The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). Where initial clusters come from is the subject of a larger debate, with suggestions made in a later section of this document. +The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). For suggestions on how to set up the initial cluster, see the [Initial Cluster](../../04-For%20Operators/03-Deployment/02_initial-cluster.md) section in the Deployment Guide. Gardener's architecture is designed for multi-tenant environments, with a strong distinction between the operator and the end users. In Gardener, Kubernetes control planes for different tenants may reside in the same operator cluster. This approach makes it very suitable for being used with bare metal because it allows taking full advantage of the server resources. Another implication is that end users do not have access to their control plane components, such as the kube-apiserver or the ETCD. These are managed by the operator and in case of metal-stack even physically divided from the end user's workload. @@ -76,55 +76,3 @@ Due to the reason metal-stack initially used ignition to provision operating sys ### machine-controller-manager-provider-metal Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. - -## Initial Cluster Setup - -Before creating the `garden cluster`, a base K8s cluster needs to be in place. -Some suggestions for the initial K8s cluster are: - -- GCP/GKE -- metalstack.cloud - -### Initial Cluster on GCP - -- A GCP account needs to be in place. -- The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authenticating against GCP. -- The Ansible [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) can be used for creating a GKE cluster. - -Suggestions for default values are: - -- `gcp_machine_type`: e2-standard-8 -- `gcp_autoscaling_min_nodes`: 1 -- `gcp_autoscaling_max_nodes`: 3 - -### Initial Cluster on metalstack.cloud - -- A Kubernetes cluster can be created on [metalstack.cloud](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster) via UI, CLI or Terraform. - -## metal-stack Setup - -> **Attention:** Bootstrapping a metal-stack partition is out of scope and need to be done before focusing on the relationship between metal-stack and Gardener. This guide assumes a metal-stack partition (servers, switches, network, ...) is already in place. - -Start by deploying: - -- `ingress-nginx-controller` -- `cert-manager` - -This guide assumes, that metal-stack gets deployed on the same initial cluster as Gardener. On the initial cluster, the metal-stack control plane need to be deployed. This can be done as described in the metal-stack [documentation](https://docs.metal-stack.io/stable/installation/deployment/#Metal-Control-Plane-Deployment). - -### Garden Cluster Setup - -After setting up the initial K8s cluster and metal-stack, Gardener can be deployed with the [Gardener Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener). - -This deploys the following components: - -- virtual garden -- Gardener control plane components -- soil cluster -- managed seed cluster (into the metal-stack partition) - -In summary, this results in the following: - -- `Garden cluster` created in the initial cluster -- `soil cluster` created in the initial cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition -- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from From e715a4f60d70c9adbba9af1eed533031587d77a5 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 03:38:40 +0200 Subject: [PATCH 08/24] move gpu workers to deployment guide --- .../03-Deployment}/06-gpu-workers.md | 2 +- static/_redirects | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) rename docs/{05-Concepts/04-Kubernetes => 04-For Operators/03-Deployment}/06-gpu-workers.md (98%) diff --git a/docs/05-Concepts/04-Kubernetes/06-gpu-workers.md b/docs/04-For Operators/03-Deployment/06-gpu-workers.md similarity index 98% rename from docs/05-Concepts/04-Kubernetes/06-gpu-workers.md rename to docs/04-For Operators/03-Deployment/06-gpu-workers.md index c7ee3453..3a6938ce 100644 --- a/docs/05-Concepts/04-Kubernetes/06-gpu-workers.md +++ b/docs/04-For Operators/03-Deployment/06-gpu-workers.md @@ -1,5 +1,5 @@ --- -slug: /gpu-workers +slug: /deployment/gpu-workers title: GPU Workers sidebar_position: 6 --- diff --git a/static/_redirects b/static/_redirects index 2b5a590f..c728910f 100644 --- a/static/_redirects +++ b/static/_redirects @@ -13,6 +13,7 @@ https://docs.metal-stack.io/* https://metal-stack.io/:splat 301! # redirect renamed sites /docs/deployment /docs/deployment-guide +/docs/gpu-workers /docs/deployment/gpu-workers /docs/:version/deployment /docs/:version/deployment-guide /docs/isolated-kubernetes /docs/isolated-kubernetes-clusters /docs/:version/isolated-kubernetes /docs/:version/isolated-kubernetes-clusters From 2afc027cddbe63ce8a6739576781f891f99bb40a Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 03:54:32 +0200 Subject: [PATCH 09/24] add tip about autonomous control plane --- docs/04-For Operators/03-Deployment/02_initial-cluster.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 5e6387c6..5bea151a 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -16,6 +16,10 @@ The initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, - **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 +:::tip +For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). +::: + ## Next Steps Once the initial cluster is in place, the deployment continues with: From bdb6362a73033295e191b1a732ff5b6ec028a090 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 03:56:22 +0200 Subject: [PATCH 10/24] control-plane title --- docs/04-For Operators/03-Deployment/03_control-plane.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 2777db63..ec6638a5 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -1,6 +1,6 @@ --- slug: /deployment/control-plane -title: Metal Control Plane +title: Control Plane sidebar_position: 3 --- From 92c378203de376ff8772bc4bdffb48a86fc1ac24 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 17:10:25 +0200 Subject: [PATCH 11/24] improve initial cluster deployment guide --- .../03-Deployment/02_initial-cluster.md | 46 +++++++++++++++++-- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 5bea151a..26469091 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -6,24 +6,60 @@ sidebar_position: 2 # Initial Cluster -Before creating the `Garden cluster`, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure, where both the metal-stack control plane and Gardener are deployed. +Before creating the `Garden cluster`, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure for the **metal-stack control plane**. + +## Deployment Options + +There are three supported approaches for hosting the initial cluster: + +### Option 1: Shared Initial Cluster + +It is possible to use a **single initial cluster** for both metal-stack and Gardener. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. + +### Option 2: Dedicated Clusters + +We recommend using **dedicated (initial) clusters** for metal-stack and Gardener — one cluster for the metal-stack control plane and a separate cluster for Gardener. This approach provides: + +- **Clearer operational boundaries** — Separation of platform administrators and Gardener operators aligns with the role model described in the [Kubernetes Cluster Lifecycle Management](/Kubernetes%20Cluster%20Lifecycle%20Management) documentation +- **Better isolation** — Physical separation of control planes follows best practices for critical infrastructure, where operator-managed control plane components are inaccessible to end-users +- **Simplified failure boundaries** — Outages of the Gardener cluster only affect cluster provisioning, not the metal-stack infrastructure or existing workloads + +### Option 3: Autonomous Control Plane (Best for Digital Sovereignty and Critical Infrastructure) + +For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. ## Suggestions for the Initial Cluster -The initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or a self-hosted solution. Some common options: +### For Options 1 & 2: Cloud-Hosted Clusters + +For the shared and dedicated cluster approaches, the initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or any other managed Kubernetes provider. Some common options: - **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or Terraform. - **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 :::tip -For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). +For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). ::: +### For Option 3: Autonomous Control Plane with k3s + +For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because Gardener is not yet able to create an initial cluster itself (though this may change with the implementation of [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md)). + +The k3s cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). This brings several advantages: + +- **Failure isolation** — In the event of an interruption or loss of the initial k3s cluster, the production control plane remains unaffected, and end users can continue to manage their clusters as normal. +- **Separate operational responsibility** — A dedicated operations team can take care of the Day-2 maintenance of the k3s installation, which uses different tools than the rest of the setup. +- **Minimal resource requirements** — Since the number of shoot clusters to host is static, resource requirements are minimal and stable over time. + +The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and Gardener components. + +See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. + ## Next Steps Once the initial cluster is in place, the deployment continues with: -1. **[Metal Control Plane Deploymen](./03_control-plane.mdx)** — Deploy the metal-stack control plane into the initial cluster. +1. **[Metal Control Plane Deployment](./03_control-plane.mdx)** — Deploy the metal-stack control plane into the initial cluster. 2. **[Partition Deployment](./04_partition.md)** — Set up a partition -3. **[Gardener Setup](./05_gardener.md)** — Configure Gardener to use metal-stack as a provider. +3. **[Gardener Setup](./05_gardener.md)** — Configure Gardener on its own dedicated cluster to use metal-stack as a provider. From a2164b5710caa005cc5bbe4059da5ccd1e1042d4 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Wed, 1 Jul 2026 23:11:13 +0200 Subject: [PATCH 12/24] mention other kclm solutions but focus on gardener --- .../03-Deployment/02_initial-cluster.md | 40 +++- .../03-Deployment/05_gardener.md | 29 --- .../04-For Operators/03-Deployment/05_kclm.md | 202 ++++++++++++++++++ 3 files changed, 234 insertions(+), 37 deletions(-) delete mode 100644 docs/04-For Operators/03-Deployment/05_gardener.md create mode 100644 docs/04-For Operators/03-Deployment/05_kclm.md diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 26469091..27cf9e4f 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -6,7 +6,31 @@ sidebar_position: 2 # Initial Cluster -Before creating the `Garden cluster`, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure for the **metal-stack control plane**. +Before deploying a **Kubernetes Cluster Lifecycle Management (KCLM)** solution, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure for the **metal-stack control plane**. + +## KCLM Solutions + +metal-stack supports three Kubernetes Cluster Lifecycle Management solutions. Each has different maturity levels and capabilities: + +### Gardener (Recommended) + +[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** path for Kubernetes cluster lifecycle management. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +:::tip +Gardener is the recommended solution for production environments. See the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. +::: + +### Cluster-API (Alternative) + +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. + +:::warning +Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +::: + +### Kamaji (Alternative) + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. ## Deployment Options @@ -14,15 +38,15 @@ There are three supported approaches for hosting the initial cluster: ### Option 1: Shared Initial Cluster -It is possible to use a **single initial cluster** for both metal-stack and Gardener. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. +It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. ### Option 2: Dedicated Clusters -We recommend using **dedicated (initial) clusters** for metal-stack and Gardener — one cluster for the metal-stack control plane and a separate cluster for Gardener. This approach provides: +We recommend using **dedicated (initial) clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. This approach provides: -- **Clearer operational boundaries** — Separation of platform administrators and Gardener operators aligns with the role model described in the [Kubernetes Cluster Lifecycle Management](/Kubernetes%20Cluster%20Lifecycle%20Management) documentation +- **Clearer operational boundaries** — Separation of platform administrators and KCLM operators aligns with the role model described in the [Kubernetes Cluster Lifecycle Management](/Kubernetes%20Cluster%20Lifecycle%20Management) documentation - **Better isolation** — Physical separation of control planes follows best practices for critical infrastructure, where operator-managed control plane components are inaccessible to end-users -- **Simplified failure boundaries** — Outages of the Gardener cluster only affect cluster provisioning, not the metal-stack infrastructure or existing workloads +- **Simplified failure boundaries** — Outages of the KCLM cluster only affect cluster provisioning, not the metal-stack infrastructure or existing workloads ### Option 3: Autonomous Control Plane (Best for Digital Sovereignty and Critical Infrastructure) @@ -44,7 +68,7 @@ For metal-stack it does not matter where your control plane Kubernetes cluster i ### For Option 3: Autonomous Control Plane with k3s -For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because Gardener is not yet able to create an initial cluster itself (though this may change with the implementation of [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md)). +For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md) for Gardener). The k3s cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). This brings several advantages: @@ -52,7 +76,7 @@ The k3s cluster serves as a minimal control plane whose sole purpose is to host - **Separate operational responsibility** — A dedicated operations team can take care of the Day-2 maintenance of the k3s installation, which uses different tools than the rest of the setup. - **Minimal resource requirements** — Since the number of shoot clusters to host is static, resource requirements are minimal and stable over time. -The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and Gardener components. +The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. @@ -62,4 +86,4 @@ Once the initial cluster is in place, the deployment continues with: 1. **[Metal Control Plane Deployment](./03_control-plane.mdx)** — Deploy the metal-stack control plane into the initial cluster. 2. **[Partition Deployment](./04_partition.md)** — Set up a partition -3. **[Gardener Setup](./05_gardener.md)** — Configure Gardener on its own dedicated cluster to use metal-stack as a provider. +3. **[KCLM Setup](./05_kclm.md)** — Configure your Kubernetes Cluster Lifecycle Management solution (Gardener recommended) on its own dedicated cluster to use metal-stack as a provider. diff --git a/docs/04-For Operators/03-Deployment/05_gardener.md b/docs/04-For Operators/03-Deployment/05_gardener.md deleted file mode 100644 index c8bb3022..00000000 --- a/docs/04-For Operators/03-Deployment/05_gardener.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -slug: /deployment/partition -title: Gardener -sidebar_position: 5 ---- - -# Gardener with metal-stack - -This guide sets up metal-stack as a cloud provider for [Gardener](https://gardener.cloud/). -Precondition is a metal-stack deployment on the same initial cluster as Gardener, as described in the [control-plane](./03_control-plane.mdx) deployment guide. - -Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). - -The roles deploy the following components: - -- virtual garden -- Gardener control plane components -- soil cluster -- managed seed cluster (into the metal-stack partition) - -In summary, this results in the following: - -- `Garden cluster` created in the initial cluster -- `soil cluster` created in the initial cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition -- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. -::: diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md new file mode 100644 index 00000000..891d3e80 --- /dev/null +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -0,0 +1,202 @@ +--- +slug: /deployment/kclm +title: Kubernetes Cluster Lifecycle Management (KCLM) +sidebar_position: 5 +--- + +# Kubernetes Cluster Lifecycle Management (KCLM) + +This guide sets up a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: + +## KCLM Solutions Overview + +### Gardener (Recommended) + +[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +:::tip +We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. +::: + +#### Cluster Architecture + +Gardener uses a hierarchical cluster model — often called the "kubeception" model — where Kubernetes clusters host other Kubernetes clusters: + +- **Garden cluster** — The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). It is recommended to deploy via the `gardener-operator`. +- **Virtual Garden** — A recommended deployment pattern where Gardener runs inside a virtual cluster on the Garden cluster. This provides a dedicated ETCD for Gardener resources and an independent update lifecycle from the Garden cluster itself. End users get project namespaces in the virtual garden. +- **Seed cluster** — A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. +- **Shoot cluster** — Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. This is "Kubernetes-in-Kubernetes" — the Shoot control plane is physically separated from end-user workloads. + +The cluster hierarchy looks like this: + +``` +Garden Cluster (Gardener control plane) +├── Virtual Garden (dedicated ETCD for Gardener resources) +│ ├── Seed 1 (gardenlet + extensions) +│ │ ├── Shoot A (control plane on Seed 1, workers on metal-stack) +│ │ └── Shoot B +│ └── Seed 2 (gardenlet + extensions) +│ └── Shoot C +└── Soil cluster (manually deployed, can become a Seed via ManagedSeed) +``` + +#### Operational Model + +Gardener differentiates between **end-users** (shoot owners) and **platform administrators** (seed owners), similar to how hyperscalers offer Kubernetes as a Service: + +| Role | Responsibilities | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | +| **Administrators** | Set up Seed clusters in data centers, whitelist machine types, provide Kubernetes versions and OS images, define lifecycle policies, fleet-wide GitOps-driven operations | + +End-users access the Virtual Garden through a feature-rich Kubernetes API with OIDC-based authorization. Platform administrators manage the entire platform through GitOps-driven processes with approval workflows. Gardener clearly defines responsibility boundaries — administrators manage the control plane lifecycle, CNI, CSI drivers, and other platform components, while end-users focus on their workloads. + +#### Deployment Summary + +Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). + +The roles deploy the following components: + +- virtual garden +- Gardener control plane components +- soil cluster +- managed seed cluster (into the metal-stack partition) + +In summary, this results in the following: + +- `Garden cluster` created in the Gardener cluster (dedicated cluster) +- `soil cluster` created in the Gardener cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition +- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from + +:::tip +We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard helps you manage Shoots, Seeds, and Projects through a web UI. +::: + +#### Core Controllers + +The Gardener platform consists of the following core controllers, all deployed via the Ansible roles: + +| Component | Responsibility | +| ------------------------------------------- | ---------------------------------------------------------------------------------------- | +| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | +| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | +| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | +| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | +| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | +| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | +| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | +| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | +| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM | + +#### Gardener Extensions + +Gardener's extensibility model allows provider-specific reconcilers to be deployed during cluster provisioning. The `gardener-extensions` [Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions) deploys the following extensions into the Gardener runtime cluster: + +| Extension | Purpose | +| -------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `gardener-extension-provider-metal` | IaaS integration — reconciles Infrastructure, ControlPlane, and Worker resources via the metal-stack API | +| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` into cloud-init/ignition userdata | +| `gardener-extension-networking-calico` | Calico CNI extension | +| `gardener-extension-networking-cilium` | Cilium CNI extension (alternative to Calico) | +| `gardener-extension-dns-powerdns` | DNS management via PowerDNS | +| `shoot-dns-service` | DNS service for Shoot clusters | +| `gardener-extension-backup-s3` | etcd backup to S3-compatible object storage | +| `gardener-extension-audit` | Audit logging webhook | +| `gardener-extension-acl` | Access control list management | +| `shoot-cert-service` | Certificate management with Let's Encrypt (supports Shoot-level issuers) | +| `gardener-extension-csi-driver-lvm` | LVM-based CSI driver for local storage | +| `gardener-extension-ontap` | NetApp ONTAP CSI driver | + +Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_extension_provider_metal_enabled`). Key configuration variables for the metal provider include: + +- `gardener_extension_provider_metal_etcd_storage_class_name` — Storage class for Shoot etcds +- `gardener_extension_provider_metal_etcd_backup_schedule` — etcd backup schedule +- `gardener_extension_provider_metal_machine_images` — Machine images (typically matches CloudProfile) +- `gardener_extension_provider_metal_admission_default_pods_cidr` — Default pod CIDR for Shoots +- `gardener_extension_provider_metal_admission_default_services_cidr` — Default services CIDR for Shoots + +For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). + +#### External Dependencies + +The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: + +- **DNS** — For cluster domain resolution (PowerDNS or external) +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) +- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests +- **MetalLB** — For exposing Kubernetes API servers of Shoot clusters (layer 2 load balancing) + +#### Operational Features + +Gardener provides several day-2 operational capabilities out of the box: + +- **Shoot migration** — Control planes can be migrated across Seed clusters for geographic relocation +- **VPN** — Secure connection between control plane components on Seeds and worker nodes +- **Hibernation** — Scale down infrastructure by deleting worker nodes and scaling down control plane components; can be triggered manually or by schedule +- **Maintenance time windows** — End-user configurable windows for automatic updates (Kubernetes versions, machine images) +- **Auto-updates** — Shoot owners can individually enable/disable auto-updates for Kubernetes versions and OS images +- **Certificate rotation** — Automatic rotation where possible; multi-phase rotation for CA changes to avoid workload disruption +- **Fleet management** — Proven at scale (280+ clusters by metal-stack, 10,000+ clusters in production Gardener installations) + +#### Related Components + +- [Cloud Controller Manager](../../05-Concepts/04-Kubernetes/03-cloud-controller-manager.md) — metal-ccm provides load balancer configuration (MetalLB) and node properties +- [Firewall Controller Manager](../../05-Concepts/04-Kubernetes/04-firewall-controller-manager.md) — firewall rule management via CRDs (nftables, Suricata intrusion detection) + +For more details on Gardener terminology, architecture, and operational model, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). + +### Cluster-API (Alternative) + +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. + +The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. + +:::warning +Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +::: + +#### CAPMS CRDs + +CAPMS implements the CAPI infrastructure provider contract for bare metal via metal-stack. The following CRDs are provided: + +| CRD | Purpose | +| ------------------------------ | ---------------------------------------------------------------------------------- | +| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | +| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | +| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | +| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | +| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | + +#### Architecture + +- **Bootstrap**: Kubeadm (CABPK) generates bootstrap data for joining nodes +- **Add-ons**: ClusterResourceSet + CAAPH for installing CNIs, CCM, and other components +- **CNI**: Calico (default) +- **API exposure**: KubeVIP for exposing the Kubernetes API server (vs. MetalLB in Gardener) +- **External dependencies**: DNS, NTP, ACME, S3-compatible storage, Git-Hosting with CI/CD + +For more details on Cluster-API concepts and the CAPMS provider, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). + +### Kamaji (Alternative) + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, cutting down on operational overhead and costs. It supports multi-tenancy, high availability, and integrates seamlessly with Cluster API. + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. + +#### Kamaji with metal-stack + +Kamaji can be used as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup allows Kamaji to manage **tenant clusters** on metal-stack infrastructure: + +- Kamaji runs as pods in a management cluster (e.g., a `kind` cluster) +- CAPMS provisions worker nodes and firewall resources on metal-stack via the metal-stack API +- Tenant cluster control planes run as pods in the management cluster +- Worker nodes join via CABPK and kubeadm +- `metal-ccm` needs to be deployed into tenant clusters for load balancer and node management + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/tree/main/capi-lab) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. + +:::tip +If you are interested in using Kamaji with metal-stack, please reach out to the metal-stack team to discuss your requirements and help shape future integration efforts. +::: From 89fbfdd6746057b611ff9240711d219812e95eb6 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 09:38:44 +0200 Subject: [PATCH 13/24] review --- .../03-Deployment/01_guide.md | 2 +- .../03-Deployment/03_control-plane.mdx | 26 ++++++++++++------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 94ba964f..6dcb0260 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -12,7 +12,7 @@ In order to build up your deployment, we recommend to make use of the same Ansib In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. -This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe you the way of how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. +This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. :::warning Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index ec6638a5..b9f061b1 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -419,20 +419,26 @@ If you are having issues regarding the deployment take a look at the [troublesho ## Providing Images -After the deployment has finished (hopefully without any issues!), you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: +After the deployment has finished, you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: ``` metal_api_images: -- id: firewall-ubuntu-2.0.20201004 - name: Firewall 2 Ubuntu 20201004 - description: Firewall 2 Ubuntu 20201004 - url: http://images.metal-stack.io/metal-os/master/firewall/2.0-ubuntu/20201004/img.tar.lz4 +- id: firewall-ubuntu-3.0 + name: Firewall 3 Ubuntu + description: Firewall 3 Ubuntu Latest Release + url: https://images.metal-stack.io/metal-os/stable/firewall/3.0-ubuntu/img.tar.lz4 features: - firewall -- id: ubuntu-20.04.20201004 - name: Ubuntu 20.04 20201004 - description: Ubuntu 20.04 20201004 - url: http://images.metal-stack.io/metal-os/master/ubuntu/20.04/20201004/img.tar.lz4 +- id: ubuntu-24.4 + name: Ubuntu 24.04 + description: Ubuntu 24.04 Latest Release + url: https://images.metal-stack.io/metal-os/stable/ubuntu/24.04/img.tar.lz4 + features: + - machine +- id: debian-13.0 + name: Debian 13 + description: Debian 13 Latest Release + url: https://images.metal-stack.io/metal-os/stable/debian/13/img.tar.lz4 features: - machine ``` @@ -462,6 +468,8 @@ contexts: # in order to protect against unauthorized api access # the default hmac is "change-me" hmac: change-me + # Metal-Admin, Metal-Edit or Metal-View + hmac_auth_type: THE_AUTH_TYPE_OF_YOUR_HMAC ``` Issue the following command: From 921f06c9ca4a3e78141f20e5d2714fcc158d9a6a Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 09:57:25 +0200 Subject: [PATCH 14/24] newest ubuntu images and reference --- docs/04-For Operators/03-Deployment/03_control-plane.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index b9f061b1..8b23cf9f 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -429,10 +429,10 @@ metal_api_images: url: https://images.metal-stack.io/metal-os/stable/firewall/3.0-ubuntu/img.tar.lz4 features: - firewall -- id: ubuntu-24.4 - name: Ubuntu 24.04 - description: Ubuntu 24.04 Latest Release - url: https://images.metal-stack.io/metal-os/stable/ubuntu/24.04/img.tar.lz4 +- id: ubuntu-26.4 + name: Ubuntu 26.04 + description: Ubuntu 26.04 Latest Release + url: https://images.metal-stack.io/metal-os/stable/ubuntu/26.04/img.tar.lz4 features: - machine - id: debian-13.0 @@ -442,6 +442,7 @@ metal_api_images: features: - machine ``` +See the [metal-images reference](https://metal-stack.io/docs/references/metal-images/#supported-images) for currently supported images. Then, re-run the deployment to apply your changes. Our playbooks are idempotent. From 1f6dbc5aa61cac0810844ba0b75db61aa68e66ff Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 10:28:04 +0200 Subject: [PATCH 15/24] mention gateway api alternative --- .../03-Deployment/03_control-plane.mdx | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 8b23cf9f..7ff90639 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -158,6 +158,47 @@ This is how your `roles/ingress-controller/tasks/main.yaml` could look like: The [ansible-common](https://github.com/metal-stack/ansible-common) repository contains very general roles and modules that you can also use when extending your deployment further. ::: +## Alternative: Gateway API with Envoy Gateway + +As an alternative to nginx-ingress, metal-stack supports the [Kubernetes Gateway API](https://kubernetes.io/docs/concepts/services-networking/gateway/) via [Envoy Gateway](https://gateway.envoyproxy.io/). +This is particularly useful when you need to expose TCP-based services (such as the metal-api gRPC endpoint or NSQ) alongside HTTP services. + +Gateway API support is added **in addition to** existing Ingress support and is disabled by default in the metal-roles, allowing operators to migrate gradually. +Both can coexist during the transition period since some external dependencies (such as Thanos and MinIO) still rely on Ingress resources. + +A working reference implementation using Envoy Gateway on top of [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind) can be found in the [mini-lab repository](https://github.com/metal-stack/mini-lab). +It deploys Envoy Gateway via Helm and creates a `GatewayClass`, `Gateway`, and `EnvoyProxy` resource with HTTP, HTTPS, and TCP listeners. + +When using the Gateway API, define a separate DNS domain for Gateway API routes using `metal_control_plane_gateway_dns` and enable the respective route types on the metal roles. For example, in `group_vars/control-plane/metal.yml`: + +```yaml +--- +metal_deploy_ingress: false + +metal_api_httproute_enabled: true +metal_api_httproute_parent_refs: + - name: metal-control-plane + namespace: "{{ metal_control_plane_namespace }}" + sectionName: http + +metal_api_tcproute_enabled: true +metal_api_tcproute_parent_refs: + - name: metal-control-plane + namespace: "{{ metal_control_plane_namespace }}" + sectionName: metal-api-grpc + +nsq_tcproute_enabled: true +nsq_tcproute_parent_refs: + - name: metal-control-plane + sectionName: nsq +``` + +The same `*_httproute_enabled` / `*_tcproute_enabled` variables are available for other services such as `metal_apiserver`, `headscale`, and `zitadel`. See the [metal-roles/control-plane documentation](https://github.com/metal-stack/metal-roles/tree/master/control-plane) for the full list of supported variables. + +:::note +When using the Gateway API approach, you typically set `metal_deploy_ingress: false` to avoid deploying the nginx-ingress controller, since the Gateway API handles all traffic routing. +::: + ## Deployment Parametrization Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: @@ -442,6 +483,7 @@ metal_api_images: features: - machine ``` + See the [metal-images reference](https://metal-stack.io/docs/references/metal-images/#supported-images) for currently supported images. Then, re-run the deployment to apply your changes. Our playbooks are idempotent. From d3e16882f0f69b92ad14aa209040f4af84d902a4 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 14:07:14 +0200 Subject: [PATCH 16/24] seperate concepts from deployment --- .../03-Deployment/01_guide.md | 17 +++ .../03-Deployment/02_initial-cluster.md | 16 ++- .../03-Deployment/03_control-plane.mdx | 48 +------- .../04-For Operators/03-Deployment/05_kclm.md | 105 +++--------------- 4 files changed, 45 insertions(+), 141 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 6dcb0260..a1366e49 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -6,6 +6,23 @@ sidebar_position: 1 # Deployment Guide +metal-stack is a Metal-as-a-Service (MaaS) platform that turns on-premises bare-metal servers into an elastic, self-managed cloud infrastructure. At its core, metal-stack consists of two components: + +- **Control Plane** — The central management layer (APIs, databases, scheduling) +- **Partitions** — The data center infrastructure (servers, leaf switches, BMCs) + +These two components alone provide a fully functional **Bare-Metal as a Service (MaaS)** platform. You can allocate machines, manage networks, configure firewalls, and operate servers via REST/gRPC APIs and the `metalctl` CLI — without any Kubernetes cluster lifecycle management. + +## Kubernetes Cluster Lifecycle Management + +If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with KCLM solutions like [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) or [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. + +:::tip +You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. +::: + +## Deployment Approaches + We are bootstrapping the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 27cf9e4f..6fca88ee 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -6,7 +6,13 @@ sidebar_position: 2 # Initial Cluster -Before deploying a **Kubernetes Cluster Lifecycle Management (KCLM)** solution, a base Kubernetes cluster needs to be in place. This initial cluster serves as the bootstrap infrastructure for the **metal-stack control plane**. +An initial Kubernetes cluster is always required for metal-stack deployments using this guide, as the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) is deployed on Kubernetes. + +The initial cluster(s) serve as the bootstrap infrastructure for the **metal-stack control plane**. + +The number and placement of initial clusters depend on whether you use a KCLM solution, as well as your availability and autonomy requirements. + +If you only need **Bare-Metal as a Service** (allocating machines, managing networks, configuring firewalls via API) without KCLM, you need at least **one cluster** for the [Control Plane](./03_control-plane.mdx). ## KCLM Solutions @@ -79,11 +85,3 @@ The k3s cluster serves as a minimal control plane whose sole purpose is to host The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. - -## Next Steps - -Once the initial cluster is in place, the deployment continues with: - -1. **[Metal Control Plane Deployment](./03_control-plane.mdx)** — Deploy the metal-stack control plane into the initial cluster. -2. **[Partition Deployment](./04_partition.md)** — Set up a partition -3. **[KCLM Setup](./05_kclm.md)** — Configure your Kubernetes Cluster Lifecycle Management solution (Gardener recommended) on its own dedicated cluster to use metal-stack as a provider. diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 7ff90639..fd7bd4be 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -6,7 +6,12 @@ sidebar_position: 3 # Metal Control Plane Deployment -The metal control plane is typically deployed in a Kubernetes cluster. Therefore, this document will assume that you have a Kubernetes cluster ready for getting deployed. Even though it is theoretically possible to deploy metal-stack without Kubernetes, we strongly advise you to use the described method because we believe that Kubernetes gives you a lot of benefits regarding the stability and maintainability of the application deployment. +The metal control plane is the **minimum requirement** for running metal-stack as a Bare-Metal as a Service (MaaS) platform. + +This guide assumes a **Kubernetes-based deployment** because our Ansible roles are designed around Kubernetes — it is significantly more comfortable to deploy and operate metal-stack on Kubernetes than on any other platform. +While it is theoretically possible to deploy metal-stack without Kubernetes (see [Target Deployment Platforms](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) for the reasoning behind this design decision), doing so would require you to come up with your own deployment mechanism. + +The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Initial Cluster](./02_initial-cluster.md). :::tip For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). @@ -158,47 +163,6 @@ This is how your `roles/ingress-controller/tasks/main.yaml` could look like: The [ansible-common](https://github.com/metal-stack/ansible-common) repository contains very general roles and modules that you can also use when extending your deployment further. ::: -## Alternative: Gateway API with Envoy Gateway - -As an alternative to nginx-ingress, metal-stack supports the [Kubernetes Gateway API](https://kubernetes.io/docs/concepts/services-networking/gateway/) via [Envoy Gateway](https://gateway.envoyproxy.io/). -This is particularly useful when you need to expose TCP-based services (such as the metal-api gRPC endpoint or NSQ) alongside HTTP services. - -Gateway API support is added **in addition to** existing Ingress support and is disabled by default in the metal-roles, allowing operators to migrate gradually. -Both can coexist during the transition period since some external dependencies (such as Thanos and MinIO) still rely on Ingress resources. - -A working reference implementation using Envoy Gateway on top of [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind) can be found in the [mini-lab repository](https://github.com/metal-stack/mini-lab). -It deploys Envoy Gateway via Helm and creates a `GatewayClass`, `Gateway`, and `EnvoyProxy` resource with HTTP, HTTPS, and TCP listeners. - -When using the Gateway API, define a separate DNS domain for Gateway API routes using `metal_control_plane_gateway_dns` and enable the respective route types on the metal roles. For example, in `group_vars/control-plane/metal.yml`: - -```yaml ---- -metal_deploy_ingress: false - -metal_api_httproute_enabled: true -metal_api_httproute_parent_refs: - - name: metal-control-plane - namespace: "{{ metal_control_plane_namespace }}" - sectionName: http - -metal_api_tcproute_enabled: true -metal_api_tcproute_parent_refs: - - name: metal-control-plane - namespace: "{{ metal_control_plane_namespace }}" - sectionName: metal-api-grpc - -nsq_tcproute_enabled: true -nsq_tcproute_parent_refs: - - name: metal-control-plane - sectionName: nsq -``` - -The same `*_httproute_enabled` / `*_tcproute_enabled` variables are available for other services such as `metal_apiserver`, `headscale`, and `zitadel`. See the [metal-roles/control-plane documentation](https://github.com/metal-stack/metal-roles/tree/master/control-plane) for the full list of supported variables. - -:::note -When using the Gateway API approach, you typically set `metal_deploy_ingress: false` to avoid deploying the nginx-ingress controller, since the Gateway API handles all traffic routing. -::: - ## Deployment Parametrization Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md index 891d3e80..6cefe841 100644 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -6,7 +6,7 @@ sidebar_position: 5 # Kubernetes Cluster Lifecycle Management (KCLM) -This guide sets up a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: +This guide covers the deployment of a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: ## KCLM Solutions Overview @@ -15,52 +15,27 @@ This guide sets up a Kubernetes Cluster Lifecycle Management (KCLM) solution to [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. :::tip -We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. +We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Initial Cluster](./02_initial-cluster.md) documentation. ::: -#### Cluster Architecture - -Gardener uses a hierarchical cluster model — often called the "kubeception" model — where Kubernetes clusters host other Kubernetes clusters: - -- **Garden cluster** — The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). It is recommended to deploy via the `gardener-operator`. -- **Virtual Garden** — A recommended deployment pattern where Gardener runs inside a virtual cluster on the Garden cluster. This provides a dedicated ETCD for Gardener resources and an independent update lifecycle from the Garden cluster itself. End users get project namespaces in the virtual garden. -- **Seed cluster** — A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. -- **Shoot cluster** — Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. This is "Kubernetes-in-Kubernetes" — the Shoot control plane is physically separated from end-user workloads. - -The cluster hierarchy looks like this: - -``` -Garden Cluster (Gardener control plane) -├── Virtual Garden (dedicated ETCD for Gardener resources) -│ ├── Seed 1 (gardenlet + extensions) -│ │ ├── Shoot A (control plane on Seed 1, workers on metal-stack) -│ │ └── Shoot B -│ └── Seed 2 (gardenlet + extensions) -│ └── Shoot C -└── Soil cluster (manually deployed, can become a Seed via ManagedSeed) -``` - -#### Operational Model - -Gardener differentiates between **end-users** (shoot owners) and **platform administrators** (seed owners), similar to how hyperscalers offer Kubernetes as a Service: - -| Role | Responsibilities | -| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | -| **Administrators** | Set up Seed clusters in data centers, whitelist machine types, provide Kubernetes versions and OS images, define lifecycle policies, fleet-wide GitOps-driven operations | - -End-users access the Virtual Garden through a feature-rich Kubernetes API with OIDC-based authorization. Platform administrators manage the entire platform through GitOps-driven processes with approval workflows. Gardener clearly defines responsibility boundaries — administrators manage the control plane lifecycle, CNI, CSI drivers, and other platform components, while end-users focus on their workloads. +For more details on Gardener terminology, architecture, operational model, failure domains, and operational features, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). #### Deployment Summary Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). -The roles deploy the following components: +The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: -- virtual garden -- Gardener control plane components -- soil cluster -- managed seed cluster (into the metal-stack partition) +- **DNS** — For cluster domain resolution +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) +- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests + +The following dependencies are introduced: + +- CNI: Calico or Cilium +- MetalLB for exposing the Kubernetes API servers of the clusters In summary, this results in the following: @@ -117,36 +92,6 @@ Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_exte For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). -#### External Dependencies - -The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: - -- **DNS** — For cluster domain resolution (PowerDNS or external) -- **NTP** — Time synchronization across all nodes -- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) -- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) -- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests -- **MetalLB** — For exposing Kubernetes API servers of Shoot clusters (layer 2 load balancing) - -#### Operational Features - -Gardener provides several day-2 operational capabilities out of the box: - -- **Shoot migration** — Control planes can be migrated across Seed clusters for geographic relocation -- **VPN** — Secure connection between control plane components on Seeds and worker nodes -- **Hibernation** — Scale down infrastructure by deleting worker nodes and scaling down control plane components; can be triggered manually or by schedule -- **Maintenance time windows** — End-user configurable windows for automatic updates (Kubernetes versions, machine images) -- **Auto-updates** — Shoot owners can individually enable/disable auto-updates for Kubernetes versions and OS images -- **Certificate rotation** — Automatic rotation where possible; multi-phase rotation for CA changes to avoid workload disruption -- **Fleet management** — Proven at scale (280+ clusters by metal-stack, 10,000+ clusters in production Gardener installations) - -#### Related Components - -- [Cloud Controller Manager](../../05-Concepts/04-Kubernetes/03-cloud-controller-manager.md) — metal-ccm provides load balancer configuration (MetalLB) and node properties -- [Firewall Controller Manager](../../05-Concepts/04-Kubernetes/04-firewall-controller-manager.md) — firewall rule management via CRDs (nftables, Suricata intrusion detection) - -For more details on Gardener terminology, architecture, and operational model, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). - ### Cluster-API (Alternative) [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. @@ -157,27 +102,7 @@ The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cl Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. ::: -#### CAPMS CRDs - -CAPMS implements the CAPI infrastructure provider contract for bare metal via metal-stack. The following CRDs are provided: - -| CRD | Purpose | -| ------------------------------ | ---------------------------------------------------------------------------------- | -| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | -| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | -| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | -| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | -| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | - -#### Architecture - -- **Bootstrap**: Kubeadm (CABPK) generates bootstrap data for joining nodes -- **Add-ons**: ClusterResourceSet + CAAPH for installing CNIs, CCM, and other components -- **CNI**: Calico (default) -- **API exposure**: KubeVIP for exposing the Kubernetes API server (vs. MetalLB in Gardener) -- **External dependencies**: DNS, NTP, ACME, S3-compatible storage, Git-Hosting with CI/CD - -For more details on Cluster-API concepts and the CAPMS provider, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). +For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). ### Kamaji (Alternative) From 97bc6384413a868bea2042cac19498e5d1d55ca8 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 14:13:41 +0200 Subject: [PATCH 17/24] kamaji warning --- docs/04-For Operators/03-Deployment/02_initial-cluster.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 6fca88ee..dfa7f6a1 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -36,7 +36,13 @@ Cluster-API with metal-stack is in early development and not advised for product ### Kamaji (Alternative) -Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. +[Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. +Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane-providers. +However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. + +:::warning +Kamaji with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +::: ## Deployment Options From 0cd8afd9529ebee1a9b7d1a89e8078019a2ae4fe Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:23:07 +0200 Subject: [PATCH 18/24] depth --- .../03-Deployment/02_initial-cluster.md | 25 ++--- .../04-For Operators/03-Deployment/05_kclm.md | 101 +++++++++++++++--- 2 files changed, 96 insertions(+), 30 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index dfa7f6a1..7cd92b32 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -26,7 +26,7 @@ metal-stack supports three Kubernetes Cluster Lifecycle Management solutions. Ea Gardener is the recommended solution for production environments. See the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. ::: -### Cluster-API (Alternative) +### Cluster-API [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. @@ -34,7 +34,7 @@ Gardener is the recommended solution for production environments. See the [Garde Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. ::: -### Kamaji (Alternative) +#### Kamaji [Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane-providers. @@ -54,15 +54,18 @@ It is possible to use a **single initial cluster** for both metal-stack and the ### Option 2: Dedicated Clusters -We recommend using **dedicated (initial) clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. This approach provides: +We recommend using **dedicated (initial) clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. -- **Clearer operational boundaries** — Separation of platform administrators and KCLM operators aligns with the role model described in the [Kubernetes Cluster Lifecycle Management](/Kubernetes%20Cluster%20Lifecycle%20Management) documentation -- **Better isolation** — Physical separation of control planes follows best practices for critical infrastructure, where operator-managed control plane components are inaccessible to end-users -- **Simplified failure boundaries** — Outages of the KCLM cluster only affect cluster provisioning, not the metal-stack infrastructure or existing workloads +This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. -### Option 3: Autonomous Control Plane (Best for Digital Sovereignty and Critical Infrastructure) +### Option 3: Autonomous Control Plane -For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. +For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. + +The autonomous control-plane cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). +This brings several advantages like failure isolation, separate operational responsibility, minimal resource requirements and full control and ownership. + +The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. ## Suggestions for the Initial Cluster @@ -82,12 +85,6 @@ For metal-stack it does not matter where your control plane Kubernetes cluster i For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md) for Gardener). -The k3s cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). This brings several advantages: - -- **Failure isolation** — In the event of an interruption or loss of the initial k3s cluster, the production control plane remains unaffected, and end users can continue to manage their clusters as normal. -- **Separate operational responsibility** — A dedicated operations team can take care of the Day-2 maintenance of the k3s installation, which uses different tools than the rest of the setup. -- **Minimal resource requirements** — Since the number of shoot clusters to host is static, resource requirements are minimal and stable over time. - The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md index 6cefe841..6937f48d 100644 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -30,18 +30,18 @@ The following data center infrastructure dependencies are treated as given and m - **NTP** — Time synchronization across all nodes - **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) - **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) -- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests +- **Git-Hosting with CI/CD** — You must set up your own Git repository and CI/CD pipeline to manage cluster deployments (see [Fleet Management and GitOps](#fleet-management-and-gitops) below). The following dependencies are introduced: - CNI: Calico or Cilium - MetalLB for exposing the Kubernetes API servers of the clusters -In summary, this results in the following: +In summary, this results in the following cluster hierarchy: -- `Garden cluster` created in the Gardener cluster (dedicated cluster) -- `soil cluster` created in the Gardener cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition -- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from +- **Garden cluster** — The Gardener control plane (Gardener API server, controller manager, scheduler, admission controller) deployed on a dedicated cluster. +- **Seed** — A cluster running the `gardenlet` agent, connected to the Gardener control plane. Seeds are deployed inside the metal-stack partition and orchestrate cluster provisioning within that site. +- **Shoot** — Every fully provisioned and managed Kubernetes cluster. Shoot control planes run as pods in the Seed namespace, while worker nodes are provisioned as bare-metal machines on metal-stack infrastructure. :::tip We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard helps you manage Shoots, Seeds, and Projects through a web UI. @@ -92,6 +92,30 @@ Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_exte For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). +#### Fleet Management and GitOps + +You must set up your own Git repository and CI/CD pipeline to manage cluster deployments. This gives you peer review, audit trails, and rollback capabilities. + +**What you need to build:** + +1. **Git repository** — Store the following as YAML manifests: + - `cloudprofiles/` — CloudProfile definitions (whitelisted regions, machine types, OS images, Kubernetes versions) + - `seeds/` — Seed configurations per data center + - `projects//shoots/` — Per-project Shoot manifests + - `extensions/` — Helm charts for Gardener extensions +2. **CI/CD pipeline** — Deploy manifests from Git to the Gardener API (Virtual Garden). This pipeline is your primary interface for fleet-wide changes. +3. **Branching strategy** — Use separate branches or environments (staging → production) to validate changes before rolling them out fleet-wide. + +**Operational capabilities provided by Gardener:** + +Once your GitOps pipeline is in place, Gardener provides the following day-2 operational features: + +- **CloudProfile validation** — Administrators define allowed regions, machine types, operating systems, and Kubernetes versions. Shoot specs are validated against the CloudProfile before being stored in the Virtual Garden's ETCD. +- **Multi-stage environments** — End-users can label clusters as `evaluation` or `development` to test upcoming Kubernetes versions and auto-upgrades before rolling them out to `production` clusters. +- **Maintenance time windows** — Configurable per Shoot; all day-2 operations (Kubernetes patch updates, machine image updates) are carried out within these windows. +- **Emergency patching** — Administrators can apply fleet-wide changes via image vector overwrites in the Gardener deployment Git repository. Changes must be validated in a dedicated staging environment first. +- **Accidental deletion protection** — Shoot deletion is guarded by specific annotations. ETCD backup retention timeouts are configurable, allowing cluster restoration after accidental deletion. + ### Cluster-API (Alternative) [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. @@ -99,29 +123,74 @@ For the full variable reference, see the [gardener-extensions README](https://gi The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. :::warning -Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). +#### Deployment + +Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The full deployment guide is available in the [CAPMS README](https://metal-stack.io/docs/next/references/cluster-api-provider-metal-stack). + +**Deployment flow** + +1. **Prepare management cluster** — A Kubernetes cluster to host CAPI controllers and cluster state (e.g., a `kind` cluster) +2. **Configure `clusterctl`** — Add the metal-stack provider URL to `~/.config/cluster-api/clusterctl.yaml` +3. **Set environment variables** — `METAL_API_URL`, `METAL_API_HMAC`, `METAL_PROJECT_ID`, `METAL_PARTITION`, control plane/worker/firewall machine images and sizes, cluster name, namespace, and Kubernetes version +4. **Install CAPMS** — Build and push the CAPMS image, then run `make install` and `make deploy` (or apply the released `infrastructure-components.yaml` directly) +5. **Initialize clusterctl** — Run `clusterctl init --infrastructure metal-stack` on the management cluster +6. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +7. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the workload cluster via `ClusterResourceSet` and CAAPH (Cluster API Add-on Provider for Helm) +8. **Retrieve kubeconfig** — Use `clusterctl get kubeconfig` to access the provisioned cluster + +**Network integration** + +Network integration for Cluster-API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. + +For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. + +**Air-gapped environments** + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. + +**Fleet management and GitOps** + +You must set up your own Git repository and GitOps operator to manage cluster deployments. + +**What you need to build:** + +1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. +2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. + +**Platform capabilities:** + +- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss. + ### Kamaji (Alternative) +:::warning +Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + [Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, cutting down on operational overhead and costs. It supports multi-tenancy, high availability, and integrates seamlessly with Cluster API. -Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. #### Kamaji with metal-stack -Kamaji can be used as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup allows Kamaji to manage **tenant clusters** on metal-stack infrastructure: +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure: -- Kamaji runs as pods in a management cluster (e.g., a `kind` cluster) -- CAPMS provisions worker nodes and firewall resources on metal-stack via the metal-stack API -- Tenant cluster control planes run as pods in the management cluster -- Worker nodes join via CABPK and kubeadm -- `metal-ccm` needs to be deployed into tenant clusters for load balancer and node management +1. **Deploy Kamaji** into a management cluster (e.g., a `kind` cluster) using the [Kamaji on kind](https://kamaji.clastix.io/getting-started/kamaji-kind/) tutorial +2. **Install CAPMS** as the infrastructure provider into the same management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP in the management cluster's network for the tenant API server +4. **Create a tenant cluster** — Registers the VIP in MetalLB, applies the cluster template via `clusterctl`. Kamaji creates the control plane pods in the management cluster +5. **Provision infrastructure** — CAPMS provisions firewall and worker machines on metal-stack via the metal-stack API +6. **Join workers** — Machines join via CABPK and kubeadm +7. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/tree/main/capi-lab) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. -:::tip -If you are interested in using Kamaji with metal-stack, please reach out to the metal-stack team to discuss your requirements and help shape future integration efforts. -::: +**Fleet management and GitOps** + +Since Kamaji with metal-stack uses Cluster-API under the hood (Kamaji as `ControlPlaneProvider`, CAPMS as `InfrastructureProvider`), fleet management follows the same pattern as Cluster-API described above. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. From efd95f85ffe59ed4bde5d229f551a6ad6c640da7 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:28:11 +0200 Subject: [PATCH 19/24] no ansible not easy --- docs/04-For Operators/03-Deployment/01_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index a1366e49..865842e0 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -36,7 +36,7 @@ Probably you need to learn writing Ansible playbooks if you want to be able to d ::: :::info -If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. However, you will probably be able to re-use some of our contents from our [metal-roles](https://github.com/metal-stack/metal-roles) repository, e.g. the Helm chart for deploying the metal control plane. +If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configurations), so building an equivalent from scratch will require significant effort. ::: :::tip From dbff18fd59462694cc765c2e658b5c6653818433 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:35:44 +0200 Subject: [PATCH 20/24] wording --- docs/04-For Operators/03-Deployment/01_guide.md | 6 +++--- docs/04-For Operators/03-Deployment/02_initial-cluster.md | 2 +- docs/04-For Operators/03-Deployment/05_kclm.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 865842e0..be7d7e7f 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -25,11 +25,11 @@ You can use metal-stack as a pure MaaS platform without any KCLM integration. Th We are bootstrapping the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. -In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). +We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository. -In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. +To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) on GitHub Container Registry. This Docker image bundles all required tools and libraries, keeping your deployment environment lean and easy to maintain. -This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. +The remainder of this guide assumes you are using our Ansible roles and the deployment base image. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) installed. Every deployment environment is unique, so this guide describes how we set up metal-stack in our own infrastructure. You will need to adapt the steps to match your specific requirements. :::warning Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 7cd92b32..abc8fe14 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -28,7 +28,7 @@ Gardener is the recommended solution for production environments. See the [Garde ### Cluster-API -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. :::warning Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md index 6937f48d..b12b2a70 100644 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -118,7 +118,7 @@ Once your GitOps pipeline is in place, Gardener provides the following day-2 ope ### Cluster-API (Alternative) -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under heavy development** and not yet production-ready. +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. From 93f04c8b1d6afd303a0f20530557d7d15b7492d8 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:40:29 +0200 Subject: [PATCH 21/24] reference concepts --- docs/04-For Operators/03-Deployment/02_initial-cluster.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index abc8fe14..54b20465 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -16,7 +16,8 @@ If you only need **Bare-Metal as a Service** (allocating machines, managing netw ## KCLM Solutions -metal-stack supports three Kubernetes Cluster Lifecycle Management solutions. Each has different maturity levels and capabilities: +metal-stack supports three Kubernetes Cluster Lifecycle Management solutions. Each has different maturity levels and capabilities. +See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed view on the solutions. ### Gardener (Recommended) From 258acf80a3bc29c4e7d5b960e3640d2f2851434b Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:46:36 +0200 Subject: [PATCH 22/24] capi development status --- docs/04-For Operators/03-Deployment/02_initial-cluster.md | 4 ++-- docs/04-For Operators/03-Deployment/05_kclm.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_initial-cluster.md index 54b20465..077951bc 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_initial-cluster.md @@ -32,7 +32,7 @@ Gardener is the recommended solution for production environments. See the [Garde [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. :::warning -Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. ::: #### Kamaji @@ -42,7 +42,7 @@ Kamaji therefore uses ClusterAPI to support different infrastructure- and contro However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. :::warning -Kamaji with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. +Kamaji with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. ::: ## Deployment Options diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md index b12b2a70..74d27d1b 100644 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -123,7 +123,7 @@ Once your GitOps pipeline is in place, Gardener provides the following day-2 ope The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. :::warning -Cluster-API with metal-stack is in early development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. +Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). From f629c66f007dd7c83d6b575200286cd44e7515f5 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Thu, 2 Jul 2026 16:48:47 +0200 Subject: [PATCH 23/24] remove redundant tip --- docs/04-For Operators/03-Deployment/03_control-plane.mdx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index fd7bd4be..50f16425 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -13,10 +13,6 @@ While it is theoretically possible to deploy metal-stack without Kubernetes (see The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Initial Cluster](./02_initial-cluster.md). -:::tip -For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). -::: - Let's start off with a fresh folder for your deployment: ```bash From 1d2d66b66f1305c2324a5c12f2b0b020d7df7791 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Tue, 7 Jul 2026 14:31:02 +0200 Subject: [PATCH 24/24] bootstrap infra and kclm depth --- ...ster.md => 02_bootstrap-infrastructure.md} | 33 +++++++------ .../03-Deployment/03_control-plane.mdx | 2 +- .../04-For Operators/03-Deployment/05_kclm.md | 48 ++++++++++--------- 3 files changed, 44 insertions(+), 39 deletions(-) rename docs/04-For Operators/03-Deployment/{02_initial-cluster.md => 02_bootstrap-infrastructure.md} (75%) diff --git a/docs/04-For Operators/03-Deployment/02_initial-cluster.md b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md similarity index 75% rename from docs/04-For Operators/03-Deployment/02_initial-cluster.md rename to docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md index 077951bc..9548c37f 100644 --- a/docs/04-For Operators/03-Deployment/02_initial-cluster.md +++ b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md @@ -1,23 +1,30 @@ --- -slug: /deployment/initial-cluster -title: Initial Cluster +slug: /deployment/bootstrap-infrastructure +title: Bootstrap Infrastructure sidebar_position: 2 --- -# Initial Cluster +# Bootstrap Infrastructure -An initial Kubernetes cluster is always required for metal-stack deployments using this guide, as the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) is deployed on Kubernetes. +Every metal-stack deployment starts with one or more **initial clusters** — Kubernetes clusters that host the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane). Collectively, these form the **bootstrap infrastructure** for your metal-stack platform. -The initial cluster(s) serve as the bootstrap infrastructure for the **metal-stack control plane**. +The initial cluster(s) serve two purposes: -The number and placement of initial clusters depend on whether you use a KCLM solution, as well as your availability and autonomy requirements. +1. **Host the metal-stack control plane** — the core platform components for bare-metal management. +2. **Enable Kubernetes Cluster Lifecycle Management (KCLM)** — if you need to provision and manage downstream Kubernetes clusters on your bare-metal machines. -If you only need **Bare-Metal as a Service** (allocating machines, managing networks, configuring firewalls via API) without KCLM, you need at least **one cluster** for the [Control Plane](./03_control-plane.mdx). +The number and placement of initial clusters depends on your KCLM choice, availability requirements, and autonomy needs. + +If you only need **Bare-Metal as a Service** (allocating machines, managing networks, configuring firewalls via API) without KCLM, you need at least **one initial cluster** for the [Control Plane](./03_control-plane.mdx). + +:::tip +Your control plane Kubernetes cluster can run anywhere — on a hyperscaler, in your own data center, or on [metalstack.cloud](https://metalstack.cloud). A managed cluster from a hyperscaler removes the operational burden of running Kubernetes yourself and can even strengthen fail-safe operation. Learn more about the [rationale for this approach](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) and find concrete hosting suggestions below. +::: ## KCLM Solutions -metal-stack supports three Kubernetes Cluster Lifecycle Management solutions. Each has different maturity levels and capabilities. -See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed view on the solutions. +metal-stack supports three Kubernetes Cluster Lifecycle Management solutions, each with different maturity levels and capabilities. +See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed comparison. ### Gardener (Recommended) @@ -53,9 +60,9 @@ There are three supported approaches for hosting the initial cluster: It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. -### Option 2: Dedicated Clusters +### Option 2: Dedicated Initial Clusters -We recommend using **dedicated (initial) clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. +We recommend using **dedicated initial clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. @@ -78,10 +85,6 @@ For the shared and dedicated cluster approaches, the initial cluster can be host - **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 -:::tip -For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. If you are interested, you can find a reasoning behind this deployment decision [here](../../05-Concepts/01-architecture.mdx#target-deployment-platforms). -::: - ### For Option 3: Autonomous Control Plane with k3s For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md) for Gardener). diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 50f16425..594742e5 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -11,7 +11,7 @@ The metal control plane is the **minimum requirement** for running metal-stack a This guide assumes a **Kubernetes-based deployment** because our Ansible roles are designed around Kubernetes — it is significantly more comfortable to deploy and operate metal-stack on Kubernetes than on any other platform. While it is theoretically possible to deploy metal-stack without Kubernetes (see [Target Deployment Platforms](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) for the reasoning behind this design decision), doing so would require you to come up with your own deployment mechanism. -The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Initial Cluster](./02_initial-cluster.md). +The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). Let's start off with a fresh folder for your deployment: diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md index 74d27d1b..bc247bce 100644 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -15,7 +15,7 @@ This guide covers the deployment of a Kubernetes Cluster Lifecycle Management (K [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. :::tip -We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Initial Cluster](./02_initial-cluster.md) documentation. +We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md) documentation. ::: For more details on Gardener terminology, architecture, operational model, failure domains, and operational features, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). @@ -128,20 +128,20 @@ Cluster-API with metal-stack is in development and not advised for production us For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). +Unlike Gardener, which provides a complete Kubernetes-as-a-Service platform with integrated day-2 operations (DNS, backup, certificate rotation, audit), Cluster-API is a declarative cluster management framework. Operators must assemble their own day-2 tooling — CNI, CCM, DNS, backup, and certificate management — and manage them through GitOps workflows. + #### Deployment -Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The full deployment guide is available in the [CAPMS README](https://metal-stack.io/docs/next/references/cluster-api-provider-metal-stack). +Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. **Deployment flow** -1. **Prepare management cluster** — A Kubernetes cluster to host CAPI controllers and cluster state (e.g., a `kind` cluster) -2. **Configure `clusterctl`** — Add the metal-stack provider URL to `~/.config/cluster-api/clusterctl.yaml` -3. **Set environment variables** — `METAL_API_URL`, `METAL_API_HMAC`, `METAL_PROJECT_ID`, `METAL_PARTITION`, control plane/worker/firewall machine images and sizes, cluster name, namespace, and Kubernetes version -4. **Install CAPMS** — Build and push the CAPMS image, then run `make install` and `make deploy` (or apply the released `infrastructure-components.yaml` directly) -5. **Initialize clusterctl** — Run `clusterctl init --infrastructure metal-stack` on the management cluster -6. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -7. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the workload cluster via `ClusterResourceSet` and CAAPH (Cluster API Add-on Provider for Helm) -8. **Retrieve kubeconfig** — Use `clusterctl get kubeconfig` to access the provisioned cluster +1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state +2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster +3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) +4. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH +6. **Retrieve kubeconfig** — Access the provisioned cluster **Network integration** @@ -169,28 +169,30 @@ You must set up your own Git repository and GitOps operator to manage cluster de ### Kamaji (Alternative) +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. + :::warning Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: -[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, cutting down on operational overhead and costs. It supports multi-tenancy, high availability, and integrates seamlessly with Cluster API. +#### Kamaji with metal-stack -Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. -#### Kamaji with metal-stack +Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. -Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure: +**Deployment** -1. **Deploy Kamaji** into a management cluster (e.g., a `kind` cluster) using the [Kamaji on kind](https://kamaji.clastix.io/getting-started/kamaji-kind/) tutorial -2. **Install CAPMS** as the infrastructure provider into the same management cluster -3. **Create a control plane VIP** — MetalLB assigns a virtual IP in the management cluster's network for the tenant API server -4. **Create a tenant cluster** — Registers the VIP in MetalLB, applies the cluster template via `clusterctl`. Kamaji creates the control plane pods in the management cluster -5. **Provision infrastructure** — CAPMS provisions firewall and worker machines on metal-stack via the metal-stack API -6. **Join workers** — Machines join via CABPK and kubeadm -7. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster +1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers +2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server +4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster -A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/tree/main/capi-lab) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. **Fleet management and GitOps** -Since Kamaji with metal-stack uses Cluster-API under the hood (Kamaji as `ControlPlaneProvider`, CAPMS as `InfrastructureProvider`), fleet management follows the same pattern as Cluster-API described above. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. +Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster-API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline.