From 99c1186a2599f0a637556faffc3351372032edd9 Mon Sep 17 00:00:00 2001 From: MeoK <12069138+chinameok@users.noreply.github.com> Date: Thu, 9 Jul 2026 07:53:04 +0000 Subject: [PATCH 1/5] docs(solutions): change kubelet maxPods on immutable HCS nodes (S2) --- ...maxPods_on_immutable_HCS_workload_nodes.md | 206 ++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md diff --git a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md new file mode 100644 index 000000000..5e41c5873 --- /dev/null +++ b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md @@ -0,0 +1,206 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x,4.2.x,4.3.x +--- + +# Changing the kubelet maxPods setting on immutable infrastructure nodes + +## Issue + +You need to change the kubelet `maxPods` setting on immutable-infrastructure worker nodes — for example, to raise pod density on a cluster provisioned on Huawei Cloud Stack (HCS). On immutable nodes you cannot simply edit a file on the host and expect the change to persist: node files are reconciled by the platform's Machine Configuration component, and an out-of-band edit is treated as configuration drift. The change also has to reach two different node populations: + +- **Nodes that already exist**, which must take the new value **without being rebooted**. +- **Nodes added later** by scale-out, or replaced during an upgrade's rolling update, which must come up already carrying the new value. + +This article covers both: Machine Configuration for the existing nodes, and the cluster's provider template for future nodes. + +## Root Cause + +`maxPods` is a field of the upstream `KubeletConfiguration` (`kubelet.config.k8s.io/v1beta1`). On immutable nodes the kubelet configuration is owned by Machine Configuration, which renders and reconciles node files through the machine configuration daemon; the daemon marks a node `Degraded` if a managed file is edited out of band. Two independent delivery paths are needed because the two node populations are created differently: + +- Existing nodes are **not** reprovisioned, so their running configuration must be changed in place. +- Future nodes are created from the provider's **bootstrap configuration**, so they inherit whatever that configuration specifies at first boot. + +The kubelet reads its configuration from several layered sources, and the effective value follows a fixed precedence: **command-line flag > `--config-dir` drop-in > base configuration file > built-in default**. This precedence is what lets the two delivery paths coexist without fighting (see *Keeping the two paths consistent*, below). + +On ACP 4.1–4.3, Machine Configuration does not yet provide a dedicated custom resource for kubelet configuration. The supported interim method is a `MachineConfig` that writes a kubelet configuration drop-in, described below. A future release introduces a dedicated kubelet-configuration resource; when you upgrade to it, migrate the objects created here to that resource. + +> This is an advanced, node-level procedure. A mistake in the kubelet service override can leave a node `NotReady`. Apply it to a non-production pool first, and engage Alauda support if you are unsure. + +## Resolution + +### Part 1 — Change existing nodes with Machine Configuration (no reboot) + +`maxPods` takes effect after a kubelet restart; it does not require a node reboot, and running pods are not evicted. Deliver it with three objects, applied in order. + +**Prerequisite — confirm the node's kubelet service.** The switch in Step 2 overrides the kubelet `ExecStart`. Read the actual `ExecStart` from a target node first and reuse it verbatim, only appending the `--config-dir` flag: + +```bash +systemctl cat kubelet +``` + +**Step 1 — Node disruption policy.** By default a file change does not restart any service, so a drop-in would be written to disk but not applied. Add a policy that reloads systemd and restarts the kubelet when either file changes. Apply this first and confirm it appears in the `cluster` resource's `status.nodeDisruptionPolicyStatus` **before** creating the objects in Steps 2 and 3. + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfiguration +metadata: + name: cluster # the singleton in the cpaas-system namespace +spec: + nodeDisruptionPolicy: + files: + - path: /etc/systemd/system/kubelet.service.d/25-config-dir.conf + actions: + - type: DaemonReload + - type: Restart + restart: + serviceName: kubelet.service + - path: /etc/kubernetes/kubelet.conf.d/30-maxpods.conf + actions: + - type: DaemonReload + - type: Restart + restart: + serviceName: kubelet.service + sshkey: + actions: + - type: None +``` + +**Step 2 — Enable the kubelet `--config-dir` (once per pool).** This lets the kubelet read drop-in files from `/etc/kubernetes/kubelet.conf.d/`. Write it as a file (not as a `systemd.units` entry) so that removing it later does not disturb the base kubelet unit. Prepare the drop-in — **adjust the `ExecStart` line to match the output of `systemctl cat kubelet` on your nodes**: + +```ini +[Service] +ExecStart= +ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS --config-dir=/etc/kubernetes/kubelet.conf.d +``` + +Base64-encode it: + +```bash +base64 -w0 25-config-dir.conf +``` + +Create the `MachineConfig` (the `contents.source` below is the encoding of the drop-in above): + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 25-worker-kubelet-config-dir + labels: + machineconfiguration.alauda.io/role: worker + machineconfiguration.alauda.io/kubelet-config: "config-dir" +spec: + config: + ignition: + version: 3.4.0 + storage: + files: + - path: /etc/systemd/system/kubelet.service.d/25-config-dir.conf + mode: 0o644 + overwrite: true + contents: + source: 'data:text/plain;base64,W1NlcnZpY2VdCkV4ZWNTdGFydD0KRXhlY1N0YXJ0PS91c3IvYmluL2t1YmVsZXQgJEtVQkVMRVRfS1VCRUNPTkZJR19BUkdTICRLVUJFTEVUX0NPTkZJR19BUkdTICRLVUJFTEVUX0tVQkVBRE1fQVJHUyAtLWNvbmZpZy1kaXI9L2V0Yy9rdWJlcm5ldGVzL2t1YmVsZXQuY29uZi5kCg==' +``` + +**Step 3 — Set maxPods.** Write a partial `KubeletConfiguration` into the `--config-dir` directory. The example sets `maxPods: 250`: + +```yaml +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +maxPods: 250 +``` + +Base64-encode it: + +```bash +base64 -w0 30-maxpods.conf +``` + +Create the `MachineConfig`: + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 30-worker-kubelet-maxpods + labels: + machineconfiguration.alauda.io/role: worker + machineconfiguration.alauda.io/kubelet-config: "setting" + annotations: + machineconfiguration.alauda.io/kubelet-fields: "maxPods" +spec: + config: + ignition: + version: 3.4.0 + storage: + files: + - path: /etc/kubernetes/kubelet.conf.d/30-maxpods.conf + mode: 0o644 + overwrite: true + contents: + source: 'data:text/plain;base64,YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24KbWF4UG9kczogMjUwCg==' +``` + +After both `MachineConfig` objects are applied, the daemon reloads systemd and restarts the kubelet on each targeted node; the new `maxPods` takes effect and the node is not rebooted. + +Keep each concern in its own object: put only kubelet settings in these objects (not chrony, sysctl, or unrelated files); name drop-in files `NN-.conf` with `NN` in the 10–49 range; and keep the labels shown above. To change another kubelet field, add another Step 3-style object with a new file number, and add its path to the disruption policy in Step 1. + +### Part 2 — Make future nodes carry the value (provider template) + +New worker nodes are created from the worker pool's `KubeadmConfigTemplate` (together with its `MachineDeployment` and `HCSMachineTemplate`). Set `maxPods` there so nodes are born with it. + +Use a kubeadm `KubeletConfiguration` **patch**, not `kubeletExtraArgs`. HCS clusters already apply kubelet patches this way (for example, for the kubelet serving certificate). Add `maxPods` to the worker pool's `KubeadmConfigTemplate`, either by extending the existing `kubeletconfiguration…+strategic.json` patch or by adding a new patch file: + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: - +spec: + template: + spec: + files: + - path: /etc/kubernetes/patches/kubeletconfiguration1+strategic.json + owner: root:root + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "maxPods": 250 + } + joinConfiguration: + patches: + directory: /etc/kubernetes/patches +``` + +Because a patch modifies the base kubelet configuration file — not a command-line flag — it stays underneath the Part 1 `--config-dir` drop-in in precedence, so the two paths agree. Applying this template does not change existing nodes; it only affects nodes created after the change. + +### Keeping the two paths consistent + +A common concern is whether re-applying the Machine Configuration drop-in to a node that the provider already configured causes a problem. It does not: + +- The two paths write **different files**. The provider sets `maxPods` in the node's base kubelet configuration; Machine Configuration writes a separate drop-in under `/etc/kubernetes/kubelet.conf.d/`. The machine configuration daemon manages only its own two files and never touches the base configuration, so there is no shared-ownership conflict, no drift, and no reboot loop. +- When a newly provisioned node joins and matches the pool, the daemon applies the drop-in as well. This re-asserts the same value and triggers a single kubelet restart on first reconcile — harmless. +- Only the effective-value precedence matters. Keep the value in Part 1 and Part 2 **identical**. If they ever differ, the `--config-dir` drop-in wins over the base configuration file — which is exactly why Part 2 uses a patch and not `kubeletExtraArgs`: `kubeletExtraArgs` becomes a command-line flag, and a flag would instead win over the drop-in and silently shadow it. Set the value through these two declarative paths only; do not edit nodes over SSH. + +### Limitations + +- **Pod network sizing.** `maxPods` cannot exceed the number of pod IP addresses available per node. With the default per-node pod CIDR (a `/24`, roughly 254 usable addresses), a node cannot usefully run more pods than that regardless of `maxPods`. Before raising `maxPods` toward or beyond that number, check the cluster's per-node pod CIDR size; enlarging it is a cluster-wide networking change that must be planned separately. +- **Interim method.** On ACP 4.1–4.3, Machine Configuration has no dedicated kubelet-configuration resource, so this drop-in approach is the supported path. A future release adds a dedicated resource; once you upgrade, migrate the Part 1 objects to it. The labels and the one-object-per-field layout above are what make that migration mechanical. + +## Diagnostic Steps + +Read the live, effective kubelet configuration from a node and confirm `maxPods`. This returns the merged value actually in use: + +```bash +kubectl get --raw "/api/v1/nodes//proxy/configz" \ + | jq '.kubeletconfig.maxPods' +``` + +Check both an existing node (changed by Part 1) and a node added after the Part 2 change; both should report the value you set (`250` in the examples). If an existing node still shows the old value, confirm that the node disruption policy from Step 1 is present in the `cluster` resource's `status.nodeDisruptionPolicyStatus`, and that the kubelet `ExecStart` in the Step 2 switch matches the node's actual service. From 86cf41ca716dd603168dda3e1a341e029e13fa4b Mon Sep 17 00:00:00 2001 From: MeoK <12069138+chinameok@users.noreply.github.com> Date: Thu, 9 Jul 2026 08:05:09 +0000 Subject: [PATCH 2/5] docs(solutions): clarify applicability is bound to Machine Configuration version, not ACP version --- ...kubelet_maxPods_on_immutable_HCS_workload_nodes.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md index 5e41c5873..21193f005 100644 --- a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md +++ b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md @@ -18,6 +18,13 @@ You need to change the kubelet `maxPods` setting on immutable-infrastructure wor This article covers both: Machine Configuration for the existing nodes, and the cluster's provider template for future nodes. +## Environment + +This procedure is delivered by the Alauda Container Platform Machine Configuration component, and its applicability is tied to that component's version — **not** to the ACP platform version: + +- **Machine Configuration version.** The procedure applies to Machine Configuration releases **before v4.1.x**. Machine Configuration v4.1.x introduces a dedicated custom resource for kubelet configuration; on any release that provides it, use that resource instead and migrate the objects created here. +- **ACP platform version.** The method itself does not depend on the ACP version. Whether it can be used on a given cluster depends only on which ACP versions the installed Machine Configuration release supports. The current Machine Configuration supports ACP v4.1, v4.2, and v4.3. + ## Root Cause `maxPods` is a field of the upstream `KubeletConfiguration` (`kubelet.config.k8s.io/v1beta1`). On immutable nodes the kubelet configuration is owned by Machine Configuration, which renders and reconciles node files through the machine configuration daemon; the daemon marks a node `Degraded` if a managed file is edited out of band. Two independent delivery paths are needed because the two node populations are created differently: @@ -27,7 +34,7 @@ This article covers both: Machine Configuration for the existing nodes, and the The kubelet reads its configuration from several layered sources, and the effective value follows a fixed precedence: **command-line flag > `--config-dir` drop-in > base configuration file > built-in default**. This precedence is what lets the two delivery paths coexist without fighting (see *Keeping the two paths consistent*, below). -On ACP 4.1–4.3, Machine Configuration does not yet provide a dedicated custom resource for kubelet configuration. The supported interim method is a `MachineConfig` that writes a kubelet configuration drop-in, described below. A future release introduces a dedicated kubelet-configuration resource; when you upgrade to it, migrate the objects created here to that resource. +In the Machine Configuration releases this procedure targets (see *Environment* above), there is no dedicated custom resource for kubelet configuration. The supported interim method is therefore a `MachineConfig` that writes a kubelet configuration drop-in, described below. > This is an advanced, node-level procedure. A mistake in the kubelet service override can leave a node `NotReady`. Apply it to a non-production pool first, and engage Alauda support if you are unsure. @@ -192,7 +199,7 @@ A common concern is whether re-applying the Machine Configuration drop-in to a n ### Limitations - **Pod network sizing.** `maxPods` cannot exceed the number of pod IP addresses available per node. With the default per-node pod CIDR (a `/24`, roughly 254 usable addresses), a node cannot usefully run more pods than that regardless of `maxPods`. Before raising `maxPods` toward or beyond that number, check the cluster's per-node pod CIDR size; enlarging it is a cluster-wide networking change that must be planned separately. -- **Interim method.** On ACP 4.1–4.3, Machine Configuration has no dedicated kubelet-configuration resource, so this drop-in approach is the supported path. A future release adds a dedicated resource; once you upgrade, migrate the Part 1 objects to it. The labels and the one-object-per-field layout above are what make that migration mechanical. +- **Interim method, bounded by the Machine Configuration version.** This drop-in approach is the supported path only on Machine Configuration releases **before v4.1.x**, which have no dedicated kubelet-configuration resource. Machine Configuration v4.1.x adds that resource; once the cluster runs a Machine Configuration release that provides it, migrate the Part 1 objects to it and stop using this method. The labels and the one-object-per-field layout above are what make that migration mechanical. Note this boundary is set by the Machine Configuration version, not by the ACP version. ## Diagnostic Steps From 12199dfd782b17a4b9ad7ab0bb7d6a6faabce84f Mon Sep 17 00:00:00 2001 From: MeoK <12069138+chinameok@users.noreply.github.com> Date: Thu, 9 Jul 2026 08:15:51 +0000 Subject: [PATCH 3/5] docs(solutions): mark Machine Configuration v4.1.x as planned/unreleased --- ...hanging_kubelet_maxPods_on_immutable_HCS_workload_nodes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md index 21193f005..21be1f0b6 100644 --- a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md +++ b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md @@ -22,7 +22,7 @@ This article covers both: Machine Configuration for the existing nodes, and the This procedure is delivered by the Alauda Container Platform Machine Configuration component, and its applicability is tied to that component's version — **not** to the ACP platform version: -- **Machine Configuration version.** The procedure applies to Machine Configuration releases **before v4.1.x**. Machine Configuration v4.1.x introduces a dedicated custom resource for kubelet configuration; on any release that provides it, use that resource instead and migrate the objects created here. +- **Machine Configuration version.** This procedure applies to the current Machine Configuration releases — those prior to the planned **v4.1.x** release. v4.1.x is **planned but not yet released**; it is intended to introduce a dedicated custom resource for kubelet configuration. Once that release is available and in use, configure the kubelet through that resource instead, and migrate the objects created here. - **ACP platform version.** The method itself does not depend on the ACP version. Whether it can be used on a given cluster depends only on which ACP versions the installed Machine Configuration release supports. The current Machine Configuration supports ACP v4.1, v4.2, and v4.3. ## Root Cause @@ -199,7 +199,7 @@ A common concern is whether re-applying the Machine Configuration drop-in to a n ### Limitations - **Pod network sizing.** `maxPods` cannot exceed the number of pod IP addresses available per node. With the default per-node pod CIDR (a `/24`, roughly 254 usable addresses), a node cannot usefully run more pods than that regardless of `maxPods`. Before raising `maxPods` toward or beyond that number, check the cluster's per-node pod CIDR size; enlarging it is a cluster-wide networking change that must be planned separately. -- **Interim method, bounded by the Machine Configuration version.** This drop-in approach is the supported path only on Machine Configuration releases **before v4.1.x**, which have no dedicated kubelet-configuration resource. Machine Configuration v4.1.x adds that resource; once the cluster runs a Machine Configuration release that provides it, migrate the Part 1 objects to it and stop using this method. The labels and the one-object-per-field layout above are what make that migration mechanical. Note this boundary is set by the Machine Configuration version, not by the ACP version. +- **Interim method, bounded by the Machine Configuration version.** This drop-in approach is the supported path on the current Machine Configuration releases (those prior to the planned **v4.1.x**), which have no dedicated kubelet-configuration resource. The v4.1.x release is **planned but not yet available**; it is intended to add that resource. Once the cluster runs a Machine Configuration release that provides it, migrate the Part 1 objects to it and stop using this method. The labels and the one-object-per-field layout above are what make that migration mechanical. Note this boundary is set by the Machine Configuration version, not by the ACP version. ## Diagnostic Steps From bbb5b28f53ea3d7bc240254416f164380a6071ce Mon Sep 17 00:00:00 2001 From: MeoK <12069138+chinameok@users.noreply.github.com> Date: Thu, 9 Jul 2026 10:39:51 +0000 Subject: [PATCH 4/5] =?UTF-8?q?docs(solutions):=20fix=20worker=20provider?= =?UTF-8?q?=20config=20=E2=80=94=20KubeadmConfigTemplate=20kubeletExtraArg?= =?UTF-8?q?s,=20not=20control-plane=20patch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...maxPods_on_immutable_HCS_workload_nodes.md | 29 +++++++------------ 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md index 21be1f0b6..21b1ee9e3 100644 --- a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md +++ b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md @@ -159,9 +159,9 @@ Keep each concern in its own object: put only kubelet settings in these objects ### Part 2 — Make future nodes carry the value (provider template) -New worker nodes are created from the worker pool's `KubeadmConfigTemplate` (together with its `MachineDeployment` and `HCSMachineTemplate`). Set `maxPods` there so nodes are born with it. +New worker (compute) nodes are created from the worker pool's `KubeadmConfigTemplate` (together with its `MachineDeployment` and `HCSMachineTemplate`). Set `maxPods` there so nodes are born with it. Control-plane nodes are configured through a different resource—the `KubeadmControlPlane`—and are out of scope for this workload-node procedure. -Use a kubeadm `KubeletConfiguration` **patch**, not `kubeletExtraArgs`. HCS clusters already apply kubelet patches this way (for example, for the kubelet serving certificate). Add `maxPods` to the worker pool's `KubeadmConfigTemplate`, either by extending the existing `kubeletconfiguration…+strategic.json` patch or by adding a new patch file: +For worker nodes, set the value with `joinConfiguration.nodeRegistration.kubeletExtraArgs` in the `KubeadmConfigTemplate`. Add `max-pods` alongside any existing `kubeletExtraArgs` entries (such as `volume-plugin-dir`); do not remove them: ```yaml apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 @@ -171,30 +171,21 @@ metadata: spec: template: spec: - files: - - path: /etc/kubernetes/patches/kubeletconfiguration1+strategic.json - owner: root:root - permissions: "0644" - content: | - { - "apiVersion": "kubelet.config.k8s.io/v1beta1", - "kind": "KubeletConfiguration", - "maxPods": 250 - } joinConfiguration: - patches: - directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + max-pods: "250" ``` -Because a patch modifies the base kubelet configuration file — not a command-line flag — it stays underneath the Part 1 `--config-dir` drop-in in precedence, so the two paths agree. Applying this template does not change existing nodes; it only affects nodes created after the change. +`kubeletExtraArgs` becomes a kubelet command-line flag (`--max-pods`). Applying this template does not change existing nodes; it only affects worker nodes created after the change. Because a flag takes precedence over the Part 1 `--config-dir` drop-in, the provider value is authoritative on new nodes—keep it identical to the Part 1 value (see *Keeping the two paths consistent*). ### Keeping the two paths consistent -A common concern is whether re-applying the Machine Configuration drop-in to a node that the provider already configured causes a problem. It does not: +A common concern is whether re-applying the Machine Configuration drop-in to a new node—one the provider bootstrap already configured—causes a problem. It does not: -- The two paths write **different files**. The provider sets `maxPods` in the node's base kubelet configuration; Machine Configuration writes a separate drop-in under `/etc/kubernetes/kubelet.conf.d/`. The machine configuration daemon manages only its own two files and never touches the base configuration, so there is no shared-ownership conflict, no drift, and no reboot loop. -- When a newly provisioned node joins and matches the pool, the daemon applies the drop-in as well. This re-asserts the same value and triggers a single kubelet restart on first reconcile — harmless. -- Only the effective-value precedence matters. Keep the value in Part 1 and Part 2 **identical**. If they ever differ, the `--config-dir` drop-in wins over the base configuration file — which is exactly why Part 2 uses a patch and not `kubeletExtraArgs`: `kubeletExtraArgs` becomes a command-line flag, and a flag would instead win over the drop-in and silently shadow it. Set the value through these two declarative paths only; do not edit nodes over SSH. +- The two paths do not share a file. On existing nodes, Machine Configuration writes the `maxPods` drop-in under `/etc/kubernetes/kubelet.conf.d/`. On new worker nodes, the provider sets `maxPods` as a kubelet flag (`--max-pods`) through `kubeletExtraArgs`. The machine configuration daemon manages only its own drop-in and switch files and never touches the provider's flag, so there is no shared-ownership conflict, no drift, and no reboot loop. +- A new node also matches the `MachineConfigPool`, so the daemon applies the drop-in there too. Because the flag already sets the same value, the effective `maxPods` does not change; at most this triggers one kubelet restart on first reconcile — harmless. +- Precedence decides the effective value on a new node, where both are present: the kubelet flag wins over the `--config-dir` drop-in, so the provider value is authoritative and the drop-in is shadowed. Keep the Part 1 and Part 2 values **identical** so the result is the same either way. Set the value only through these two declarative paths; do not edit nodes over SSH. ### Limitations From f65b9e6f8fc211c7f518c7c6b7c39c3e7c7f71f1 Mon Sep 17 00:00:00 2001 From: MeoK <12069138+chinameok@users.noreply.github.com> Date: Thu, 9 Jul 2026 13:19:41 +0000 Subject: [PATCH 5/5] docs(solutions): Part 1 via systemd KUBELET_EXTRA_ARGS drop-in (no --config-dir); fix precedence + limitations after prod incident --- ...maxPods_on_immutable_HCS_workload_nodes.md | 91 ++++++------------- 1 file changed, 26 insertions(+), 65 deletions(-) diff --git a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md index 21b1ee9e3..bb3fa0dae 100644 --- a/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md +++ b/docs/en/solutions/Changing_kubelet_maxPods_on_immutable_HCS_workload_nodes.md @@ -11,7 +11,7 @@ ProductsVersion: ## Issue -You need to change the kubelet `maxPods` setting on immutable-infrastructure worker nodes — for example, to raise pod density on a cluster provisioned on Huawei Cloud Stack (HCS). On immutable nodes you cannot simply edit a file on the host and expect the change to persist: node files are reconciled by the platform's Machine Configuration component, and an out-of-band edit is treated as configuration drift. The change also has to reach two different node populations: +You need to change the kubelet `maxPods` setting on immutable-infrastructure worker nodes — for example, to adjust pod density on a cluster provisioned on Huawei Cloud Stack (HCS). On immutable nodes you cannot simply edit a file on the host and expect the change to persist: node files are reconciled by the platform's Machine Configuration component, and an out-of-band edit is treated as configuration drift. The change also has to reach two different node populations: - **Nodes that already exist**, which must take the new value **without being rebooted**. - **Nodes added later** by scale-out, or replaced during an upgrade's rolling update, which must come up already carrying the new value. @@ -32,9 +32,9 @@ This procedure is delivered by the Alauda Container Platform Machine Configurati - Existing nodes are **not** reprovisioned, so their running configuration must be changed in place. - Future nodes are created from the provider's **bootstrap configuration**, so they inherit whatever that configuration specifies at first boot. -The kubelet reads its configuration from several layered sources, and the effective value follows a fixed precedence: **command-line flag > `--config-dir` drop-in > base configuration file > built-in default**. This precedence is what lets the two delivery paths coexist without fighting (see *Keeping the two paths consistent*, below). +The kubelet reads its configuration from layered sources with a fixed precedence: **command-line flag > `--config-dir` drop-in > `--config` file > built-in default**. Both delivery paths below set `maxPods` as a command-line flag; which one wins where both apply is explained in *Keeping the two paths consistent*. -In the Machine Configuration releases this procedure targets (see *Environment* above), there is no dedicated custom resource for kubelet configuration. The supported interim method is therefore a `MachineConfig` that writes a kubelet configuration drop-in, described below. +In the Machine Configuration releases this procedure targets (see *Environment* above), there is no dedicated custom resource for kubelet configuration. The supported interim method is therefore a `MachineConfig` that installs a systemd drop-in for the kubelet, described below. > This is an advanced, node-level procedure. A mistake in the kubelet service override can leave a node `NotReady`. Apply it to a non-production pool first, and engage Alauda support if you are unsure. @@ -42,15 +42,15 @@ In the Machine Configuration releases this procedure targets (see *Environment* ### Part 1 — Change existing nodes with Machine Configuration (no reboot) -`maxPods` takes effect after a kubelet restart; it does not require a node reboot, and running pods are not evicted. Deliver it with three objects, applied in order. +`maxPods` is set with a systemd drop-in that passes `--max-pods` to the kubelet through `KUBELET_EXTRA_ARGS`. It takes effect after a kubelet restart—the node is not rebooted and running pods are not evicted. This applies to **all worker nodes, including infrastructure (`infra`) nodes**, which take the same value. Deliver it with two objects, applied in order. -**Prerequisite — confirm the node's kubelet service.** The switch in Step 2 overrides the kubelet `ExecStart`. Read the actual `ExecStart` from a target node first and reuse it verbatim, only appending the `--config-dir` flag: +**Prerequisite — confirm the node's kubelet service.** The drop-in below re-declares the kubelet `ExecStart`. Read the actual `ExecStart` from a target node first and reuse it verbatim; the one requirement is that it ends with `$KUBELET_EXTRA_ARGS`, so the value set below is applied: ```bash systemctl cat kubelet ``` -**Step 1 — Node disruption policy.** By default a file change does not restart any service, so a drop-in would be written to disk but not applied. Add a policy that reloads systemd and restarts the kubelet when either file changes. Apply this first and confirm it appears in the `cluster` resource's `status.nodeDisruptionPolicyStatus` **before** creating the objects in Steps 2 and 3. +**Step 1 — Node disruption policy.** By default a file change does not restart any service, so a drop-in would be written to disk but not applied. Add a policy that reloads systemd and restarts the kubelet when the drop-in file changes. Edit the existing singleton with **`kubectl edit machineconfiguration cluster`** — do **not** `kubectl apply`, which would overwrite other policies already on the object — and confirm the entry appears in `status.nodeDisruptionPolicyStatus` **before** creating the `MachineConfig` in Step 2: ```yaml apiVersion: machineconfiguration.alauda.io/v1alpha1 @@ -60,13 +60,7 @@ metadata: spec: nodeDisruptionPolicy: files: - - path: /etc/systemd/system/kubelet.service.d/25-config-dir.conf - actions: - - type: DaemonReload - - type: Restart - restart: - serviceName: kubelet.service - - path: /etc/kubernetes/kubelet.conf.d/30-maxpods.conf + - path: /etc/systemd/system/kubelet.service.d/30-maxpods.conf actions: - type: DaemonReload - type: Restart @@ -77,49 +71,13 @@ spec: - type: None ``` -**Step 2 — Enable the kubelet `--config-dir` (once per pool).** This lets the kubelet read drop-in files from `/etc/kubernetes/kubelet.conf.d/`. Write it as a file (not as a `systemd.units` entry) so that removing it later does not disturb the base kubelet unit. Prepare the drop-in — **adjust the `ExecStart` line to match the output of `systemctl cat kubelet` on your nodes**: +**Step 2 — Set maxPods.** Prepare the systemd drop-in. `ExecStart=` is cleared and re-declared so `$KUBELET_EXTRA_ARGS` is applied last; **replace `250` with your target value, and match the `ExecStart` line to the output of `systemctl cat kubelet` on your nodes**: ```ini [Service] +Environment="KUBELET_EXTRA_ARGS=--max-pods=250" ExecStart= -ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS --config-dir=/etc/kubernetes/kubelet.conf.d -``` - -Base64-encode it: - -```bash -base64 -w0 25-config-dir.conf -``` - -Create the `MachineConfig` (the `contents.source` below is the encoding of the drop-in above): - -```yaml -apiVersion: machineconfiguration.alauda.io/v1alpha1 -kind: MachineConfig -metadata: - name: 25-worker-kubelet-config-dir - labels: - machineconfiguration.alauda.io/role: worker - machineconfiguration.alauda.io/kubelet-config: "config-dir" -spec: - config: - ignition: - version: 3.4.0 - storage: - files: - - path: /etc/systemd/system/kubelet.service.d/25-config-dir.conf - mode: 0o644 - overwrite: true - contents: - source: 'data:text/plain;base64,W1NlcnZpY2VdCkV4ZWNTdGFydD0KRXhlY1N0YXJ0PS91c3IvYmluL2t1YmVsZXQgJEtVQkVMRVRfS1VCRUNPTkZJR19BUkdTICRLVUJFTEVUX0NPTkZJR19BUkdTICRLVUJFTEVUX0tVQkVBRE1fQVJHUyAtLWNvbmZpZy1kaXI9L2V0Yy9rdWJlcm5ldGVzL2t1YmVsZXQuY29uZi5kCg==' -``` - -**Step 3 — Set maxPods.** Write a partial `KubeletConfiguration` into the `--config-dir` directory. The example sets `maxPods: 250`: - -```yaml -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -maxPods: 250 +ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS ``` Base64-encode it: @@ -128,7 +86,7 @@ Base64-encode it: base64 -w0 30-maxpods.conf ``` -Create the `MachineConfig`: +Create the `MachineConfig` (the `contents.source` below is the encoding of the drop-in above). The `role: worker` label applies it to every worker node, `infra` included: ```yaml apiVersion: machineconfiguration.alauda.io/v1alpha1 @@ -146,22 +104,24 @@ spec: version: 3.4.0 storage: files: - - path: /etc/kubernetes/kubelet.conf.d/30-maxpods.conf + - path: /etc/systemd/system/kubelet.service.d/30-maxpods.conf mode: 0o644 overwrite: true contents: - source: 'data:text/plain;base64,YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24KbWF4UG9kczogMjUwCg==' + source: 'data:text/plain;base64,W1NlcnZpY2VdCkVudmlyb25tZW50PSJLVUJFTEVUX0VYVFJBX0FSR1M9LS1tYXgtcG9kcz0yNTAiCkV4ZWNTdGFydD0KRXhlY1N0YXJ0PS91c3IvYmluL2t1YmVsZXQgJEtVQkVMRVRfS1VCRUNPTkZJR19BUkdTICRLVUJFTEVUX0NPTkZJR19BUkdTICRLVUJFTEVUX0tVQkVBRE1fQVJHUyAkS1VCRUxFVF9FWFRSQV9BUkdTCg==' ``` -After both `MachineConfig` objects are applied, the daemon reloads systemd and restarts the kubelet on each targeted node; the new `maxPods` takes effect and the node is not rebooted. +> **Do not enable `--config-dir` against a directory that does not exist.** An earlier version of this procedure pointed the kubelet at `--config-dir=/etc/kubernetes/kubelet.conf.d` and restarted it before that directory existed, which left nodes `NotReady`. The drop-in above avoids `--config-dir` entirely. If you must use it for a setting that has no command-line flag (see *Limitations*), create the directory in the same `MachineConfig`—by writing a file into it—before anything references it. + +After the `MachineConfig` is applied, the daemon reloads systemd and restarts the kubelet on each worker node; the new `maxPods` takes effect and the node is not rebooted. -Keep each concern in its own object: put only kubelet settings in these objects (not chrony, sysctl, or unrelated files); name drop-in files `NN-.conf` with `NN` in the 10–49 range; and keep the labels shown above. To change another kubelet field, add another Step 3-style object with a new file number, and add its path to the disruption policy in Step 1. +Keep only kubelet settings in this object (not chrony, sysctl, or unrelated files); name the drop-in `NN-.conf` with `NN` in the 10–49 range; and keep the labels shown above. To set additional flag-settable fields, add them to the **same** `KUBELET_EXTRA_ARGS` line, space-separated—do **not** create a second drop-in that also assigns `KUBELET_EXTRA_ARGS` (systemd keeps only the last assignment of a variable)—and list every field in the `kubelet-fields` annotation. ### Part 2 — Make future nodes carry the value (provider template) New worker (compute) nodes are created from the worker pool's `KubeadmConfigTemplate` (together with its `MachineDeployment` and `HCSMachineTemplate`). Set `maxPods` there so nodes are born with it. Control-plane nodes are configured through a different resource—the `KubeadmControlPlane`—and are out of scope for this workload-node procedure. -For worker nodes, set the value with `joinConfiguration.nodeRegistration.kubeletExtraArgs` in the `KubeadmConfigTemplate`. Add `max-pods` alongside any existing `kubeletExtraArgs` entries (such as `volume-plugin-dir`); do not remove them: +For worker nodes, set the value with `joinConfiguration.nodeRegistration.kubeletExtraArgs` in the `KubeadmConfigTemplate`. Add `max-pods` alongside any existing `kubeletExtraArgs` entries (such as `volume-plugin-dir`); do not remove them. Apply the same value to **every** worker pool's template, the `infra` pool included, so all worker nodes match: ```yaml apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 @@ -177,20 +137,21 @@ spec: max-pods: "250" ``` -`kubeletExtraArgs` becomes a kubelet command-line flag (`--max-pods`). Applying this template does not change existing nodes; it only affects worker nodes created after the change. Because a flag takes precedence over the Part 1 `--config-dir` drop-in, the provider value is authoritative on new nodes—keep it identical to the Part 1 value (see *Keeping the two paths consistent*). +`kubeletExtraArgs` becomes a kubelet command-line flag (`--max-pods`). Applying this template does not change existing nodes; it only affects worker nodes created after the change. On a node that Machine Configuration also manages, the Part 1 value overrides this one (see *Keeping the two paths consistent*), so set both to the **same** value. ### Keeping the two paths consistent -A common concern is whether re-applying the Machine Configuration drop-in to a new node—one the provider bootstrap already configured—causes a problem. It does not: +A common concern is whether re-applying the Machine Configuration drop-in to a new node—one the provider bootstrap already configured—causes a problem. It does not, but be clear about which value wins: -- The two paths do not share a file. On existing nodes, Machine Configuration writes the `maxPods` drop-in under `/etc/kubernetes/kubelet.conf.d/`. On new worker nodes, the provider sets `maxPods` as a kubelet flag (`--max-pods`) through `kubeletExtraArgs`. The machine configuration daemon manages only its own drop-in and switch files and never touches the provider's flag, so there is no shared-ownership conflict, no drift, and no reboot loop. -- A new node also matches the `MachineConfigPool`, so the daemon applies the drop-in there too. Because the flag already sets the same value, the effective `maxPods` does not change; at most this triggers one kubelet restart on first reconcile — harmless. -- Precedence decides the effective value on a new node, where both are present: the kubelet flag wins over the `--config-dir` drop-in, so the provider value is authoritative and the drop-in is shadowed. Keep the Part 1 and Part 2 values **identical** so the result is the same either way. Set the value only through these two declarative paths; do not edit nodes over SSH. +- **No shared file.** Machine Configuration writes its systemd drop-in at `/etc/systemd/system/kubelet.service.d/30-maxpods.conf`. The provider's `kubeletExtraArgs` is rendered by kubeadm into `/var/lib/kubelet/kubeadm-flags.env` (`KUBELET_KUBEADM_ARGS`). The daemon manages only its own drop-in and never touches the kubeadm file, so there is no shared-ownership conflict, no drift, and no reboot loop. +- **Both are command-line flags, and Machine Configuration wins.** The kubelet unit's `ExecStart` ends with `... $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS`. The provider value arrives in `$KUBELET_KUBEADM_ARGS`; the Machine Configuration value arrives in `$KUBELET_EXTRA_ARGS`, which is **last**. When the same flag (`--max-pods`) appears twice, the kubelet uses the last occurrence. So on any node managed by Machine Configuration, the Machine Configuration value takes effect and overrides the provider value. +- **Consequence.** Set both paths to the **same** value. If they differ, the `max-pods` in the `KubeadmConfigTemplate` will have no effect on managed nodes. Set the value only through these two declarative paths; do not edit nodes over SSH. ### Limitations - **Pod network sizing.** `maxPods` cannot exceed the number of pod IP addresses available per node. With the default per-node pod CIDR (a `/24`, roughly 254 usable addresses), a node cannot usefully run more pods than that regardless of `maxPods`. Before raising `maxPods` toward or beyond that number, check the cluster's per-node pod CIDR size; enlarging it is a cluster-wide networking change that must be planned separately. -- **Interim method, bounded by the Machine Configuration version.** This drop-in approach is the supported path on the current Machine Configuration releases (those prior to the planned **v4.1.x**), which have no dedicated kubelet-configuration resource. The v4.1.x release is **planned but not yet available**; it is intended to add that resource. Once the cluster runs a Machine Configuration release that provides it, migrate the Part 1 objects to it and stop using this method. The labels and the one-object-per-field layout above are what make that migration mechanical. Note this boundary is set by the Machine Configuration version, not by the ACP version. +- **Only settings that have a command-line flag can use this method.** `--max-pods` has a flag, so it works through `KUBELET_EXTRA_ARGS`. Settings that exist only in `KubeletConfiguration`—for example `systemReserved`, `evictionHard`, or `cpuManagerPolicy`—have no flag and must be delivered through the kubelet `--config-dir` mechanism (a config drop-in directory) instead. If you use `--config-dir`, the directory must be **created first** (by a `MachineConfig` that writes a file into it), before any unit points the kubelet at it—otherwise the kubelet fails to start and the node goes `NotReady`. +- **Interim method, bounded by the Machine Configuration version.** This drop-in approach is the supported path on the current Machine Configuration releases (those prior to the planned **v4.1.x**), which have no dedicated kubelet-configuration resource. The v4.1.x release is **planned but not yet available**; it is intended to add that resource. Once the cluster runs a Machine Configuration release that provides it, migrate the Part 1 objects to it and stop using this method. The labels above are what make that migration mechanical. This boundary is set by the Machine Configuration version, not by the ACP version. ## Diagnostic Steps @@ -201,4 +162,4 @@ kubectl get --raw "/api/v1/nodes//proxy/configz" \ | jq '.kubeletconfig.maxPods' ``` -Check both an existing node (changed by Part 1) and a node added after the Part 2 change; both should report the value you set (`250` in the examples). If an existing node still shows the old value, confirm that the node disruption policy from Step 1 is present in the `cluster` resource's `status.nodeDisruptionPolicyStatus`, and that the kubelet `ExecStart` in the Step 2 switch matches the node's actual service. +Check both an existing node (changed by Part 1) and a node added after the Part 2 change; both should report the value you set (`250` in the examples). If an existing node still shows the old value, confirm that the node disruption policy from Step 1 is present in the `cluster` resource's `status.nodeDisruptionPolicyStatus`, and that the kubelet `ExecStart` in the Step 2 drop-in matches the node's actual service.