diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md new file mode 100644 index 00000000..be7d7e7f --- /dev/null +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -0,0 +1,44 @@ +--- +slug: /deployment-guide +title: Guide +sidebar_position: 1 +--- + +# Deployment Guide + +metal-stack is a Metal-as-a-Service (MaaS) platform that turns on-premises bare-metal servers into an elastic, self-managed cloud infrastructure. At its core, metal-stack consists of two components: + +- **Control Plane** — The central management layer (APIs, databases, scheduling) +- **Partitions** — The data center infrastructure (servers, leaf switches, BMCs) + +These two components alone provide a fully functional **Bare-Metal as a Service (MaaS)** platform. You can allocate machines, manage networks, configure firewalls, and operate servers via REST/gRPC APIs and the `metalctl` CLI — without any Kubernetes cluster lifecycle management. + +## Kubernetes Cluster Lifecycle Management + +If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with KCLM solutions like [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) or [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. + +:::tip +You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. +::: + +## Deployment Approaches + +We are bootstrapping the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. + +We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository. + +To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) on GitHub Container Registry. This Docker image bundles all required tools and libraries, keeping your deployment environment lean and easy to maintain. + +The remainder of this guide assumes you are using our Ansible roles and the deployment base image. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) installed. Every deployment environment is unique, so this guide describes how we set up metal-stack in our own infrastructure. You will need to adapt the steps to match your specific requirements. + +:::warning +Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). +::: + +:::info +If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configurations), so building an equivalent from scratch will require significant effort. +::: + +:::tip +You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this document. +::: diff --git a/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md new file mode 100644 index 00000000..9548c37f --- /dev/null +++ b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md @@ -0,0 +1,94 @@ +--- +slug: /deployment/bootstrap-infrastructure +title: Bootstrap Infrastructure +sidebar_position: 2 +--- + +# Bootstrap Infrastructure + +Every metal-stack deployment starts with one or more **initial clusters** — Kubernetes clusters that host the [metal control plane](../../05-Concepts/01-architecture.mdx#metal-control-plane). Collectively, these form the **bootstrap infrastructure** for your metal-stack platform. + +The initial cluster(s) serve two purposes: + +1. **Host the metal-stack control plane** — the core platform components for bare-metal management. +2. **Enable Kubernetes Cluster Lifecycle Management (KCLM)** — if you need to provision and manage downstream Kubernetes clusters on your bare-metal machines. + +The number and placement of initial clusters depends on your KCLM choice, availability requirements, and autonomy needs. + +If you only need **Bare-Metal as a Service** (allocating machines, managing networks, configuring firewalls via API) without KCLM, you need at least **one initial cluster** for the [Control Plane](./03_control-plane.mdx). + +:::tip +Your control plane Kubernetes cluster can run anywhere — on a hyperscaler, in your own data center, or on [metalstack.cloud](https://metalstack.cloud). A managed cluster from a hyperscaler removes the operational burden of running Kubernetes yourself and can even strengthen fail-safe operation. Learn more about the [rationale for this approach](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) and find concrete hosting suggestions below. +::: + +## KCLM Solutions + +metal-stack supports three Kubernetes Cluster Lifecycle Management solutions, each with different maturity levels and capabilities. +See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed comparison. + +### Gardener (Recommended) + +[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** path for Kubernetes cluster lifecycle management. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +:::tip +Gardener is the recommended solution for production environments. See the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. +::: + +### Cluster-API + +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. + +:::warning +Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. +::: + +#### Kamaji + +[Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. +Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane-providers. +However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. + +:::warning +Kamaji with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. +::: + +## Deployment Options + +There are three supported approaches for hosting the initial cluster: + +### Option 1: Shared Initial Cluster + +It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. + +### Option 2: Dedicated Initial Clusters + +We recommend using **dedicated initial clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. + +This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. + +### Option 3: Autonomous Control Plane + +For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. + +The autonomous control-plane cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). +This brings several advantages like failure isolation, separate operational responsibility, minimal resource requirements and full control and ownership. + +The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. + +## Suggestions for the Initial Cluster + +### For Options 1 & 2: Cloud-Hosted Clusters + +For the shared and dedicated cluster approaches, the initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or any other managed Kubernetes provider. Some common options: + +- **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or Terraform. +- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. + - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 + +### For Option 3: Autonomous Control Plane with k3s + +For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md) for Gardener). + +The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. + +See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. diff --git a/docs/04-For Operators/03-deployment-guide.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx similarity index 53% rename from docs/04-For Operators/03-deployment-guide.mdx rename to docs/04-For Operators/03-Deployment/03_control-plane.mdx index ce58e0e0..594742e5 100644 --- a/docs/04-For Operators/03-deployment-guide.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -1,38 +1,17 @@ --- -slug: /deployment-guide -title: Deployment Guide +slug: /deployment/control-plane +title: Control Plane sidebar_position: 3 --- -# Deployment Guide +# Metal Control Plane Deployment -We are bootstrapping the [metal control plane](../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. +The metal control plane is the **minimum requirement** for running metal-stack as a Bare-Metal as a Service (MaaS) platform. -In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). +This guide assumes a **Kubernetes-based deployment** because our Ansible roles are designed around Kubernetes — it is significantly more comfortable to deploy and operate metal-stack on Kubernetes than on any other platform. +While it is theoretically possible to deploy metal-stack without Kubernetes (see [Target Deployment Platforms](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) for the reasoning behind this design decision), doing so would require you to come up with your own deployment mechanism. -In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. - -This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe you the way of how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. - -:::warning -Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). -::: - -:::info -If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. However, you will probably be able to re-use some of our contents from our [metal-roles](https://github.com/metal-stack/metal-roles) repository, e.g. the Helm chart for deploying the metal control plane. -::: - -:::tip -You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this document. -::: - -## Metal Control Plane Deployment - -The metal control plane is typically deployed in a Kubernetes cluster. Therefore, this document will assume that you have a Kubernetes cluster ready for getting deployed. Even though it is theoretically possible to deploy metal-stack without Kubernetes, we strongly advise you to use the described method because we believe that Kubernetes gives you a lot of benefits regarding the stability and maintainability of the application deployment. - -:::tip -For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../05-Concepts/01-architecture.mdx#target-deployment-platforms). -::: +The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). Let's start off with a fresh folder for your deployment: @@ -73,7 +52,7 @@ At the end of this section we are gonna end up with the following files and fold └── main.yaml ``` -### Releases and Ansible Role Dependencies +## Releases and Ansible Role Dependencies As metal-stack consists of many microservices all having individual versions, we have come up with a [releases](https://github.com/metal-stack/releases) repository. It contains a YAML file (we often call it release vector) describing the fitting versions of all components for every release of metal-stack. Ansible role dependencies are also part of a metal-stack release. Both the metal-stack release vector and the metal-stack ansible-roles are shipped as OCI artifacts following a specific format that's described [here](/community/oci-artifacts). These artifacts are signed with the CI token of the metal-stack Github organization and can be verified using [cosign](https://github.com/sigstore/cosign). @@ -109,7 +88,7 @@ The public key for the validation is attached to each metal-stack release. Optional further parametrization for this module can be found in its [module documentation](https://github.com/metal-stack/ansible-common/blob/a5977fc618cd34a6cebda262602b8cbc0632500d/library/metal_stack_release_vector.py). -### Inventory +## Inventory Then, there will be an inventory for the control plane deployment in `inventories/control-plane.yaml` that adds `localhost` to the `control-plane` host group: @@ -125,7 +104,7 @@ We do this since we are deploying to Kubernetes and do not need to SSH-connect t Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases. -### Control Plane Playbook +## Control Plane Playbook Next, we will define the actual deployment playbook in a file called `deploy_metal_control_plane.yaml`. You can start with the following lines: @@ -159,7 +138,7 @@ Basically, this playbook does the following: - Deploying the postgres database for the masterdata-api (wrapped in a backup-restore-sidecar) - Applying the metal control plane helm chart -### Setup an ingress-controller +## Setup an ingress-controller As a next step we write the `ingress-controller` role for deploying an ingress-controller into the cluster. [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) is what we use. If you want to use another ingress-controller, you need to parametrize the metal roles carefully. When you just use ingress-nginx, make sure to also deploy it to the default namespace ingress-nginx. @@ -180,7 +159,7 @@ This is how your `roles/ingress-controller/tasks/main.yaml` could look like: The [ansible-common](https://github.com/metal-stack/ansible-common) repository contains very general roles and modules that you can also use when extending your deployment further. ::: -### Deployment Parametrization +## Deployment Parametrization Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: @@ -189,11 +168,11 @@ Now you can parametrize the referenced roles to fit your environment. The role p metal_control_plane_ingress_dns: # if you do not have a DNS entry, you could also start with .nip.io ``` -### Providing Certificates +## Providing Certificates We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. -For the very basic setup you will need to create self-signed certificates for the communication between the following components (see [architecture](../05-Concepts/01-architecture.mdx) document): +For the very basic setup you will need to create self-signed certificates for the communication between the following components (see [architecture](../../05-Concepts/01-architecture.mdx) document): - [metal-api](https://github.com/metal-stack/metal-api) and [masterdata-api](https://github.com/metal-stack/masterdata-api) (in-cluster traffic communication) - [metal-api](https://github.com/metal-stack/metal-api) and [metal-hammer](https://github.com/metal-stack/metal-hammer) (partition to control plane communication) @@ -404,7 +383,7 @@ metal_api_grpc_certs_ca_cert: "{{ lookup('file', 'certs/ca.pem') }}" For the actual communication between the metal-api and the user clients (REST API, runs over the ingress-controller you deployed before), you can simply deploy a tool like [cert-manager](https://github.com/cert-manager/cert-manager) into your Kubernetes cluster, which will automatically provide your ingress domains with Let's Encrypt certificates. ::: -### Running the Deployment +## Running the Deployment Finally, it should be possible to run the deployment through a Docker container. Make sure to have the [Kubeconfig file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) of your cluster and set the path in the following command accordingly: @@ -436,36 +415,44 @@ docker run --rm -it \ ``` :::tip -If you are having issues regarding the deployment take a look at the [troubleshoot document](./06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! +If you are having issues regarding the deployment take a look at the [troubleshoot document](../06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: -### Providing Images +## Providing Images -After the deployment has finished (hopefully without any issues!), you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: +After the deployment has finished, you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: ``` metal_api_images: -- id: firewall-ubuntu-2.0.20201004 - name: Firewall 2 Ubuntu 20201004 - description: Firewall 2 Ubuntu 20201004 - url: http://images.metal-stack.io/metal-os/master/firewall/2.0-ubuntu/20201004/img.tar.lz4 +- id: firewall-ubuntu-3.0 + name: Firewall 3 Ubuntu + description: Firewall 3 Ubuntu Latest Release + url: https://images.metal-stack.io/metal-os/stable/firewall/3.0-ubuntu/img.tar.lz4 features: - firewall -- id: ubuntu-20.04.20201004 - name: Ubuntu 20.04 20201004 - description: Ubuntu 20.04 20201004 - url: http://images.metal-stack.io/metal-os/master/ubuntu/20.04/20201004/img.tar.lz4 +- id: ubuntu-26.4 + name: Ubuntu 26.04 + description: Ubuntu 26.04 Latest Release + url: https://images.metal-stack.io/metal-os/stable/ubuntu/26.04/img.tar.lz4 + features: + - machine +- id: debian-13.0 + name: Debian 13 + description: Debian 13 Latest Release + url: https://images.metal-stack.io/metal-os/stable/debian/13/img.tar.lz4 features: - machine ``` +See the [metal-images reference](https://metal-stack.io/docs/references/metal-images/#supported-images) for currently supported images. + Then, re-run the deployment to apply your changes. Our playbooks are idempotent. :::info Image versions should be regularly checked for updates. ::: -### Setting up metalctl +## Setting up metalctl You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). The configuration for `metalctl` should look like this: @@ -484,6 +471,8 @@ contexts: # in order to protect against unauthorized api access # the default hmac is "change-me" hmac: change-me + # Metal-Admin, Metal-Edit or Metal-View + hmac_auth_type: THE_AUTH_TYPE_OF_YOUR_HMAC ``` Issue the following command: @@ -496,7 +485,7 @@ ubuntu-19.10.20200331 Ubuntu 19.10 20200331 Ubuntu 19.10 202 The basic principles of how the metal control plane can be deployed should now be clear. It is now up to you to move the deployment execution into your CI and add things like certificates for the ingress-controller and NSQ. -### Setting Up the backup-restore-sidecar +## Setting Up the backup-restore-sidecar The backup-restore-sidecar can come in very handy when you want to add another layer of security to the metal-stack databases in your Kubernetes cluster. The sidecar takes backups of the metal databases in small time intervals and stores them in a blobstore of a cloud provider. For each database that will be backed up, a lifecycle rule is established. The backup mechanism is deactivated by default and must be activated by the operator. This way your metal-stack setup can even survive the deletion of your Kubernetes control plane cluster (including all volumes getting lost). After re-deploying metal-stack to another Kubernetes clusters, the databases come up with the latest backup data in a matter of seconds. @@ -504,7 +493,7 @@ Encryption can be enabled for the backups by providing an AES-256 encryption key Checkout the [role documentation](https://github.com/metal-stack/metal-roles/tree/master/control-plane) of the individual databases to find out how to configure the sidecar properly. You can also try out the mechanism from the [backup-restore-sidecar](https://github.com/metal-stack/backup-restore-sidecar) repository. -### Auth +## Auth metal-stack currently supports two authentication methods: @@ -532,117 +521,3 @@ metal_masterdata_api_tenants: namespace_max_length: 20 description: ``` - -## Bootstrapping a Partition - -### Out-Of-Band-Network - -To be able to deploy and maintain a metal-stack partition, you need to bootstrap the Out-Of-Band-Network first. -Some considerations must be made to fulfill the requirements of our infrastructure, a partition is designed to be: - -- secure -- fully routable (BGP) -- scalable -- resilient -- deployable via CI/CD jobs -- accessible from the internet from specific IPs - -In order to accomplish this task remotely and in a nearly automatic manner, you have to bootstrap the components in this order: - -1. management firewalls -2. management servers -3. management spines -4. management leaves -5. leaves, spines and exits - -This document assumes that all cabling is done. Here is a quick overview of the architecture: - -![Out-of-Band-Network](mgmt_net_layer3.png) - -### Management Firewalls - -As you can see, the management firewalls are the first bastion hosts in a partition to provide access to our infrastructure. There are two of them in each partition to guarantee high availability and load balancing. The very first configuration of these routers has to be done manually to solve the chicken and egg problem that you need the management firewalls in place to deploy the partition. Manually means that we generate a configuration template with ansible that we deploy with copy/paste, and the load, through the machine console. Once the management server has been deployed, we are able to deploy this configuration via CI runner and ansible. For this you need the user and the ssh-key, which is deployed with the configuration file mentioned above. -The Edgerouters has to fulfill some requirements including: - -- provide and restrict access to the Out-Of-Band-Network from the internet with a firewall ruleset -- provide destination NAT to the management server and its IPMI interface -- provide Onie Boot and ztp via DHCP options for the management spine -- provide DHCP management addresses for management spine, management server and ipmi interface of the management server -- Hairpin-NAT for the management server to access itself via its public IP, needed by the CI runner to delegate CI Jobs. -- propagate a default gateway via BGP - -### Management Servers - -The second bastion hosts are the management servers. They are the main bootstrapping components of the Out-Of-Band-Network. They also act as jump hosts for all components in a partition. Once they are installed and deployed, we are able to bootstrap all the other components. To bootstrap the management servers, we generate an ISO image which will automatically install an OS and an ansible user with ssh keys. It is preconfigured with a preseed file to allow an unattended OS installation for our needs. This is why we need remote access to the IPMI interface of the management servers: The generated ISO is attached via the virtual media function of the BMC. After that, all we have to do is boot from that virtual CD-ROM and wait for the installation to finish. Deployment jobs (Gitlab-CI) in a partition are delegated to the appropriate management servers, therefore we need a CI runner active on each management server. - -After the CI runner has been installed, you can trigger your Playbooks from the the CI. The Ansible-Playbooks have to make sure that these functionalities are present on the management servers: - -- Prometheus and exporters -- CI runner -- metal-bmc -- image-cache -- simple webserver to provide images -- [Onie Boot](https://opencomputeproject.github.io/onie/) and ZTP -- DHCP addresses for ipmi interfaces of the workers -- DHCP addresses for switches - -### Management Spines - -:::tip -If you are using SONiC switches, you should make use of Zero Touch Provisioning and Onie Boot -::: - -The purpose of these switches is to connect the management interfaces of all switches to the management servers. The management spine's own management interface is connected to the management firewall for the bootstrapping of the management spine itself. The management firewall will provide a DHCP address and DHCP options to start SONiC's [Zero Touch Provisioning](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md); the images for all switches are downloaded from the management server (nginx container). -Each management leaf is connected to both management spines to provide redundant connectivity to both management servers. BGP is used as a routing protocol such that, when a link goes down, an alternate path is used. -In the picture above you can see that there are also switch management interfaces connected to the management spine. This has to be done so that we can bootstrap these switches; the management spine relays the DHCP requests from these switches to the management servers so that they are able to Onie Boot and get their ZTP scripts. - -### Management Leaves - -All workers have to be connected with their IPMI/BMC interface to the management leaves to get DHCP addresses from the management server. The management leaves are relaying those DHCP requests to the management server which will answer the requests and provide IPs from a given range. The management interfaces of the management leaves also have to be reachable from the management server, and need to get their IP address via DHCP for the bootstrapping process. - -In the example setup, these interfaces are connected to an end-of-row-switch which aggregates them and connects them to the management spines with a fiber-optics connection. If you can reach the management spines from the management leaves with copper cables, you do not need the end of row switch. After the initial bootstrapping, the management interfaces of the management leaves continue to be used for access to the switches' command line, and for subsequent OS updates. (update=reset+bootrap+deployment) - -## Partition Deployment - -## Gardener with metal-stack - -If you want to deploy metal-stack as a cloud provider for [Gardener](https://gardener.cloud/), you should follow the regular Gardener installation instructions and setup a Gardener cluster first. It's perfectly fine to setup the Gardener cluster in the same cluster that you use for hosting metal-stack. - -You can find installation instructions for Gardener on the Gardener website beneath [docs](https://gardener.cloud/docs/). metal-stack is an out-of-tree provider and therefore you will not find example files for metal-stack resources in the Gardener repositories. The following list describes the resources and components that you need to deploy into the Gardener cluster in order to make Gardener work with metal-stack: - -:::warning -The following list assumes you have Gardener installed in a Kubernetes cluster and that you have a basic understanding of how Gardener works. If you need further help with the following steps, you can also come and ask in our Slack channel. -::: - -1. Deploy the [validator](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/validator-metal) from the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) repository to your cluster via Helm -1. Add a [cloud profile](https://github.com/gardener/gardener/blob/v1.3.3/example/30-cloudprofile.yaml) called `metal` containing all your machine images, machine types and regions (region names can be chosen freely, the zone names need to match your partition names) together with our metal-stack-specific provider config as defined [here](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/pkg/apis/metal/v1alpha1/types_cloudprofile.go) -1. Register the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) controller by deploying the [controller-registration](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/example/controller-registration.yaml) into your Gardener cluster, parametrize the embedded chart in the controller registration's values section if necessary ([this](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/provider-metal) is the corresponding values file) -1. metal-stack does not provide an own backup storage infrastructure for now. If you want to enable ETCD backups (which you should do because metal-stack also does not have persistent storage out of the box, which makes these backups even more valuable), you should deploy an extension-provider of another cloud provider and configure it to only reconcile the backup buckets (you can reference this backup infrastructure used for the metal shoot in the shoot spec) -1. Register the [os-extension-provider-metal](https://github.com/metal-stack/os-metal-extension) controller by deploying the [controller-registration](https://github.com/metal-stack/os-metal-extension/blob/v0.4.1/example/controller-registration.yaml) into your Gardener cluster, this controller can transform the operating system configuration from Gardener into Ignition user data -1. You need to use the Gardener's [networking-calico](https://github.com/gardener/gardener-extension-networking-calico) controller for setting up shoot CNI, you will have to put specific provider configuration into the shoot spec to make it work with metal-stack: - ```yaml - networking: - type: calico - # we can peer with the frr within 10.244.0.0/16, which we do with the metallb - # the networks for the shoot need to be disjunct with the networks of the seed, otherwise the VPN connection will not work properly - # the seeds are typically deployed with podCIDR 10.244.128.0/18 and serviceCIDR 10.244.192.0/18 - # the shoots are typically deployed with podCIDR 10.244.0.0/18 and serviceCIDR 10.244.64.0/18 - pods: 10.244.0.0/18 - services: 10.244.64.0/18 - providerConfig: - apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 - kind: NetworkConfig - backend: vxlan - ipv4: - pool: vxlan - mode: Always - autoDetectionMethod: interface=lo - typha: - enabled: false - ``` -1. For your seed cluster you will need to provide the provider secret for metal-stack containing the key `metalAPIHMac`, which is the API HMAC to grant editor access to the metal-api -1. Checkout our current provider configuration for [infrastructure](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_infrastructure.go) and [control-plane](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_controlplane.go) before deploying your shoot - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. -::: diff --git a/docs/04-For Operators/03-Deployment/04_partition.md b/docs/04-For Operators/03-Deployment/04_partition.md new file mode 100644 index 00000000..45f7d23f --- /dev/null +++ b/docs/04-For Operators/03-Deployment/04_partition.md @@ -0,0 +1,74 @@ +--- +slug: /deployment/partition-bootstrap +title: Partition +sidebar_position: 4 +--- + +# Bootstrapping a Partition + +## Out-Of-Band-Network + +To be able to deploy and maintain a metal-stack partition, you need to bootstrap the Out-Of-Band-Network first. +Some considerations must be made to fulfill the requirements of our infrastructure, a partition is designed to be: + +- secure +- fully routable (BGP) +- scalable +- resilient +- deployable via CI/CD jobs +- accessible from the internet from specific IPs + +In order to accomplish this task remotely and in a nearly automatic manner, you have to bootstrap the components in this order: + +1. management firewalls +2. management servers +3. management spines +4. management leaves +5. leaves, spines and exits + +This document assumes that all cabling is done. Here is a quick overview of the architecture: + +![Out-of-Band-Network](mgmt_net_layer3.png) + +## Management Firewalls + +As you can see, the management firewalls are the first bastion hosts in a partition to provide access to our infrastructure. There are two of them in each partition to guarantee high availability and load balancing. The very first configuration of these routers has to be done manually to solve the chicken and egg problem that you need the management firewalls in place to deploy the partition. Manually means that we generate a configuration template with ansible that we deploy with copy/paste, and the load, through the machine console. Once the management server has been deployed, we are able to deploy this configuration via CI runner and ansible. For this you need the user and the ssh-key, which is deployed with the configuration file mentioned above. +The Edgerouters has to fulfill some requirements including: + +- provide and restrict access to the Out-Of-Band-Network from the internet with a firewall ruleset +- provide destination NAT to the management server and its IPMI interface +- provide Onie Boot and ztp via DHCP options for the management spine +- provide DHCP management addresses for management spine, management server and ipmi interface of the management server +- Hairpin-NAT for the management server to access itself via its public IP, needed by the CI runner to delegate CI Jobs. +- propagate a default gateway via BGP + +## Management Servers + +The second bastion hosts are the management servers. They are the main bootstrapping components of the Out-Of-Band-Network. They also act as jump hosts for all components in a partition. Once they are installed and deployed, we are able to bootstrap all the other components. To bootstrap the management servers, we generate an ISO image which will automatically install an OS and an ansible user with ssh keys. It is preconfigured with a preseed file to allow an unattended OS installation for our needs. This is why we need remote access to the IPMI interface of the management servers: The generated ISO is attached via the virtual media function of the BMC. After that, all we have to do is boot from that virtual CD-ROM and wait for the installation to finish. Deployment jobs (Gitlab-CI) in a partition are delegated to the appropriate management servers, therefore we need a CI runner active on each management server. + +After the CI runner has been installed, you can trigger your Playbooks from the the CI. The Ansible-Playbooks have to make sure that these functionalities are present on the management servers: + +- Prometheus and exporters +- CI runner +- metal-bmc +- image-cache +- simple webserver to provide images +- [Onie Boot](https://opencomputeproject.github.io/onie/) and ZTP +- DHCP addresses for ipmi interfaces of the workers +- DHCP addresses for switches + +## Management Spines + +:::tip +If you are using SONiC switches, you should make use of Zero Touch Provisioning and Onie Boot +::: + +The purpose of these switches is to connect the management interfaces of all switches to the management servers. The management spine's own management interface is connected to the management firewall for the bootstrapping of the management spine itself. The management firewall will provide a DHCP address and DHCP options to start SONiC's [Zero Touch Provisioning](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md); the images for all switches are downloaded from the management server (nginx container). +Each management leaf is connected to both management spines to provide redundant connectivity to both management servers. BGP is used as a routing protocol such that, when a link goes down, an alternate path is used. +In the picture above you can see that there are also switch management interfaces connected to the management spine. This has to be done so that we can bootstrap these switches; the management spine relays the DHCP requests from these switches to the management servers so that they are able to Onie Boot and get their ZTP scripts. + +## Management Leaves + +All workers have to be connected with their IPMI/BMC interface to the management leaves to get DHCP addresses from the management server. The management leaves are relaying those DHCP requests to the management server which will answer the requests and provide IPs from a given range. The management interfaces of the management leaves also have to be reachable from the management server, and need to get their IP address via DHCP for the bootstrapping process. + +In the example setup, these interfaces are connected to an end-of-row-switch which aggregates them and connects them to the management spines with a fiber-optics connection. If you can reach the management spines from the management leaves with copper cables, you do not need the end of row switch. After the initial bootstrapping, the management interfaces of the management leaves continue to be used for access to the switches' command line, and for subsequent OS updates. (update=reset+bootrap+deployment) diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md new file mode 100644 index 00000000..bc247bce --- /dev/null +++ b/docs/04-For Operators/03-Deployment/05_kclm.md @@ -0,0 +1,198 @@ +--- +slug: /deployment/kclm +title: Kubernetes Cluster Lifecycle Management (KCLM) +sidebar_position: 5 +--- + +# Kubernetes Cluster Lifecycle Management (KCLM) + +This guide covers the deployment of a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: + +## KCLM Solutions Overview + +### Gardener (Recommended) + +[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +:::tip +We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md) documentation. +::: + +For more details on Gardener terminology, architecture, operational model, failure domains, and operational features, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). + +#### Deployment Summary + +Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). + +The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: + +- **DNS** — For cluster domain resolution +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) +- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) +- **Git-Hosting with CI/CD** — You must set up your own Git repository and CI/CD pipeline to manage cluster deployments (see [Fleet Management and GitOps](#fleet-management-and-gitops) below). + +The following dependencies are introduced: + +- CNI: Calico or Cilium +- MetalLB for exposing the Kubernetes API servers of the clusters + +In summary, this results in the following cluster hierarchy: + +- **Garden cluster** — The Gardener control plane (Gardener API server, controller manager, scheduler, admission controller) deployed on a dedicated cluster. +- **Seed** — A cluster running the `gardenlet` agent, connected to the Gardener control plane. Seeds are deployed inside the metal-stack partition and orchestrate cluster provisioning within that site. +- **Shoot** — Every fully provisioned and managed Kubernetes cluster. Shoot control planes run as pods in the Seed namespace, while worker nodes are provisioned as bare-metal machines on metal-stack infrastructure. + +:::tip +We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard helps you manage Shoots, Seeds, and Projects through a web UI. +::: + +#### Core Controllers + +The Gardener platform consists of the following core controllers, all deployed via the Ansible roles: + +| Component | Responsibility | +| ------------------------------------------- | ---------------------------------------------------------------------------------------- | +| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | +| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | +| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | +| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | +| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | +| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | +| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | +| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | +| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM | + +#### Gardener Extensions + +Gardener's extensibility model allows provider-specific reconcilers to be deployed during cluster provisioning. The `gardener-extensions` [Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions) deploys the following extensions into the Gardener runtime cluster: + +| Extension | Purpose | +| -------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `gardener-extension-provider-metal` | IaaS integration — reconciles Infrastructure, ControlPlane, and Worker resources via the metal-stack API | +| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` into cloud-init/ignition userdata | +| `gardener-extension-networking-calico` | Calico CNI extension | +| `gardener-extension-networking-cilium` | Cilium CNI extension (alternative to Calico) | +| `gardener-extension-dns-powerdns` | DNS management via PowerDNS | +| `shoot-dns-service` | DNS service for Shoot clusters | +| `gardener-extension-backup-s3` | etcd backup to S3-compatible object storage | +| `gardener-extension-audit` | Audit logging webhook | +| `gardener-extension-acl` | Access control list management | +| `shoot-cert-service` | Certificate management with Let's Encrypt (supports Shoot-level issuers) | +| `gardener-extension-csi-driver-lvm` | LVM-based CSI driver for local storage | +| `gardener-extension-ontap` | NetApp ONTAP CSI driver | + +Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_extension_provider_metal_enabled`). Key configuration variables for the metal provider include: + +- `gardener_extension_provider_metal_etcd_storage_class_name` — Storage class for Shoot etcds +- `gardener_extension_provider_metal_etcd_backup_schedule` — etcd backup schedule +- `gardener_extension_provider_metal_machine_images` — Machine images (typically matches CloudProfile) +- `gardener_extension_provider_metal_admission_default_pods_cidr` — Default pod CIDR for Shoots +- `gardener_extension_provider_metal_admission_default_services_cidr` — Default services CIDR for Shoots + +For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). + +#### Fleet Management and GitOps + +You must set up your own Git repository and CI/CD pipeline to manage cluster deployments. This gives you peer review, audit trails, and rollback capabilities. + +**What you need to build:** + +1. **Git repository** — Store the following as YAML manifests: + - `cloudprofiles/` — CloudProfile definitions (whitelisted regions, machine types, OS images, Kubernetes versions) + - `seeds/` — Seed configurations per data center + - `projects//shoots/` — Per-project Shoot manifests + - `extensions/` — Helm charts for Gardener extensions +2. **CI/CD pipeline** — Deploy manifests from Git to the Gardener API (Virtual Garden). This pipeline is your primary interface for fleet-wide changes. +3. **Branching strategy** — Use separate branches or environments (staging → production) to validate changes before rolling them out fleet-wide. + +**Operational capabilities provided by Gardener:** + +Once your GitOps pipeline is in place, Gardener provides the following day-2 operational features: + +- **CloudProfile validation** — Administrators define allowed regions, machine types, operating systems, and Kubernetes versions. Shoot specs are validated against the CloudProfile before being stored in the Virtual Garden's ETCD. +- **Multi-stage environments** — End-users can label clusters as `evaluation` or `development` to test upcoming Kubernetes versions and auto-upgrades before rolling them out to `production` clusters. +- **Maintenance time windows** — Configurable per Shoot; all day-2 operations (Kubernetes patch updates, machine image updates) are carried out within these windows. +- **Emergency patching** — Administrators can apply fleet-wide changes via image vector overwrites in the Gardener deployment Git repository. Changes must be validated in a dedicated staging environment first. +- **Accidental deletion protection** — Shoot deletion is guarded by specific annotations. ETCD backup retention timeouts are configurable, allowing cluster restoration after accidental deletion. + +### Cluster-API (Alternative) + +[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. + +The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. + +:::warning +Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept doc](../../05-Concepts/04-Kubernetes/02-cluster-api.md). + +Unlike Gardener, which provides a complete Kubernetes-as-a-Service platform with integrated day-2 operations (DNS, backup, certificate rotation, audit), Cluster-API is a declarative cluster management framework. Operators must assemble their own day-2 tooling — CNI, CCM, DNS, backup, and certificate management — and manage them through GitOps workflows. + +#### Deployment + +Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. + +**Deployment flow** + +1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state +2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster +3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) +4. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH +6. **Retrieve kubeconfig** — Access the provisioned cluster + +**Network integration** + +Network integration for Cluster-API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. + +For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. + +**Air-gapped environments** + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. + +**Fleet management and GitOps** + +You must set up your own Git repository and GitOps operator to manage cluster deployments. + +**What you need to build:** + +1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. +2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. + +**Platform capabilities:** + +- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss. + +### Kamaji (Alternative) + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. + +:::warning +Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +#### Kamaji with metal-stack + +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. + +Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. + +**Deployment** + +1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers +2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server +4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. + +**Fleet management and GitOps** + +Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster-API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. diff --git a/docs/05-Concepts/04-Kubernetes/06-gpu-workers.md b/docs/04-For Operators/03-Deployment/06-gpu-workers.md similarity index 98% rename from docs/05-Concepts/04-Kubernetes/06-gpu-workers.md rename to docs/04-For Operators/03-Deployment/06-gpu-workers.md index c7ee3453..3a6938ce 100644 --- a/docs/05-Concepts/04-Kubernetes/06-gpu-workers.md +++ b/docs/04-For Operators/03-Deployment/06-gpu-workers.md @@ -1,5 +1,5 @@ --- -slug: /gpu-workers +slug: /deployment/gpu-workers title: GPU Workers sidebar_position: 6 --- diff --git a/docs/04-For Operators/mgmt_net_layer3.drawio b/docs/04-For Operators/03-Deployment/mgmt_net_layer3.drawio similarity index 100% rename from docs/04-For Operators/mgmt_net_layer3.drawio rename to docs/04-For Operators/03-Deployment/mgmt_net_layer3.drawio diff --git a/docs/04-For Operators/mgmt_net_layer3.png b/docs/04-For Operators/03-Deployment/mgmt_net_layer3.png similarity index 100% rename from docs/04-For Operators/mgmt_net_layer3.png rename to docs/04-For Operators/03-Deployment/mgmt_net_layer3.png diff --git a/docs/05-Concepts/04-Kubernetes/01-gardener.md b/docs/05-Concepts/04-Kubernetes/01-gardener.md index 45a33837..879ee24d 100644 --- a/docs/05-Concepts/04-Kubernetes/01-gardener.md +++ b/docs/05-Concepts/04-Kubernetes/01-gardener.md @@ -8,7 +8,7 @@ sidebar_position: 1 [Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. -The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). Where initial clusters come from is the subject of a larger debate, with suggestions made in a later section of this document. +The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). For suggestions on how to set up the initial cluster, see the [Initial Cluster](../../04-For%20Operators/03-Deployment/02_initial-cluster.md) section in the Deployment Guide. Gardener's architecture is designed for multi-tenant environments, with a strong distinction between the operator and the end users. In Gardener, Kubernetes control planes for different tenants may reside in the same operator cluster. This approach makes it very suitable for being used with bare metal because it allows taking full advantage of the server resources. Another implication is that end users do not have access to their control plane components, such as the kube-apiserver or the ETCD. These are managed by the operator and in case of metal-stack even physically divided from the end user's workload. @@ -59,7 +59,7 @@ Every Kubernetes cluster that is fully provisioned and managed by Gardener is ca During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. This section briefly describes the controllers implemented by metal-stack to allow the creation of a Kubernetes cluster on metal-stack infrastructure. -If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-deployment-guide.mdx#gardener-with-metal-stack). +If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-deployment/05_partition.md#gardener-with-metal-stack). ### gardener-extension-provider-metal @@ -76,55 +76,3 @@ Due to the reason metal-stack initially used ignition to provision operating sys ### machine-controller-manager-provider-metal Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. - -## Initial Cluster Setup - -Before creating the `garden cluster`, a base K8s cluster needs to be in place. -Some suggestions for the initial K8s cluster are: - -- GCP/GKE -- metalstack.cloud - -### Initial Cluster on GCP - -- A GCP account needs to be in place. -- The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authenticating against GCP. -- The Ansible [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) can be used for creating a GKE cluster. - -Suggestions for default values are: - -- `gcp_machine_type`: e2-standard-8 -- `gcp_autoscaling_min_nodes`: 1 -- `gcp_autoscaling_max_nodes`: 3 - -### Initial Cluster on metalstack.cloud - -- A Kubernetes cluster can be created on [metalstack.cloud](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster) via UI, CLI or Terraform. - -## metal-stack Setup - -> **Attention:** Bootstrapping a metal-stack partition is out of scope and need to be done before focusing on the relationship between metal-stack and Gardener. This guide assumes a metal-stack partition (servers, switches, network, ...) is already in place. - -Start by deploying: - -- `ingress-nginx-controller` -- `cert-manager` - -This guide assumes, that metal-stack gets deployed on the same initial cluster as Gardener. On the initial cluster, the metal-stack control plane need to be deployed. This can be done as described in the metal-stack [documentation](https://docs.metal-stack.io/stable/installation/deployment/#Metal-Control-Plane-Deployment). - -### Garden Cluster Setup - -After setting up the initial K8s cluster and metal-stack, Gardener can be deployed with the [Gardener Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener). - -This deploys the following components: - -- virtual garden -- Gardener control plane components -- soil cluster -- managed seed cluster (into the metal-stack partition) - -In summary, this results in the following: - -- `Garden cluster` created in the initial cluster -- `soil cluster` created in the initial cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition -- `shooted seed` inside the metal-stack partition, where all `shoots` are derived from diff --git a/static/_redirects b/static/_redirects index 2b5a590f..c728910f 100644 --- a/static/_redirects +++ b/static/_redirects @@ -13,6 +13,7 @@ https://docs.metal-stack.io/* https://metal-stack.io/:splat 301! # redirect renamed sites /docs/deployment /docs/deployment-guide +/docs/gpu-workers /docs/deployment/gpu-workers /docs/:version/deployment /docs/:version/deployment-guide /docs/isolated-kubernetes /docs/isolated-kubernetes-clusters /docs/:version/isolated-kubernetes /docs/:version/isolated-kubernetes-clusters