Daemon pods are not being scheduled onto control plane nodes

[cheesesashimi]

Description

If a BootcNodePool is created for control-plane nodes, daemon pods are not being scheduled onto those nodes.

Reproduction

1. Create a BootcNodePool:

kubectl apply -f - <<EOF
apiVersion: node.bootc.dev/v1alpha1
kind: BootcNodePool
metadata:
  name: control-plane
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: ""
  image:
    ref: <pullspec>

bootcnodepool.node.bootc.dev/control-plane created
EOF

2. Verify that the control plane nodes received the bootc.dev/managed= label:

kubectl get nodes -l 'node-role.kubernetes.io/control-plane=,bootc.dev/managed='
NAME                                       STATUS   ROLES                  AGE    VERSION
ip-10-0-36-12.us-west-1.compute.internal   Ready    control-plane,master   140m   v1.36.2
ip-10-0-41-94.us-west-1.compute.internal   Ready    control-plane,master   140m   v1.36.2
ip-10-0-7-244.us-west-1.compute.internal   Ready    control-plane,master   140m   v1.36.2

3. Verify that the BootcNodes have been created:

kubectl get bootcnodes
NAME                                       AGE
ip-10-0-36-12.us-west-1.compute.internal   3s
ip-10-0-41-94.us-west-1.compute.internal   3s
ip-10-0-7-244.us-west-1.compute.internal   3s

5. Query for daemon pods on the control-plane nodes:

kubectl get pods -l 'app.kubernetes.io/name=bootc-operator,app.kubernetes.io/component=daemon' -n bootc-operator
No resources found in bootc-operator namespace.

# Check the DaemonSet object:
kubectl get daemonset/bootc-operator-daemon -n bootc-operator
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR        AGE
bootc-operator-daemon   0         0         0       0            0           bootc.dev/managed=   5m15s

Expected behavior

# Create a BootcNodePool object
kubectl apply -f - <<EOF
apiVersion: node.bootc.dev/v1alpha1
kind: BootcNodePool
metadata:
  name: control-plane
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: ""
  image:
    ref: <pullspec>

bootcnodepool.node.bootc.dev/control-plane created
EOF

# BootcNodes get created
kubectl get bootcnodes
NAME                                       AGE
ip-10-0-36-12.us-west-1.compute.internal   3s
ip-10-0-41-94.us-west-1.compute.internal   3s
ip-10-0-7-244.us-west-1.compute.internal   3s

# List of control plane nodes which should be managed
kubectl get nodes -l 'node-role.kubernetes.io/control-plane=,bootc.dev/managed='
NAME                                       STATUS   ROLES                  AGE    VERSION
ip-10-0-36-12.us-west-1.compute.internal   Ready    control-plane,master   145m   v1.36.2
ip-10-0-41-94.us-west-1.compute.internal   Ready    control-plane,master   145m   v1.36.2
ip-10-0-7-244.us-west-1.compute.internal   Ready    control-plane,master   145m   v1.36.2

# DaemonSet status:
kubectl get daemonset/bootc-operator-daemon -n bootc-operator
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR        AGE
bootc-operator-daemon   3         3         3       3            3           bootc.dev/managed=   7m9s

# Running DaemonSet pods:
kubectl get pods -l 'app.kubernetes.io/name=bootc-operator,app.kubernetes.io/component=daemon' -n bootc-operator
NAME                          READY   STATUS    RESTARTS   AGE
bootc-operator-daemon-d8bdn   1/1     Running   0          87s
bootc-operator-daemon-fjj7w   1/1     Running   0          87s
bootc-operator-daemon-s276s   1/1     Running   0          87s

# DaemonSet pods scheduled on control-plane nodes:
kubectl get pods -l 'app.kubernetes.io/name=bootc-operator,app.kubernetes.io/component=daemon' -n bootc-operator -o json | jq '.items[].spec.nodeName'
"ip-10-0-41-94.us-west-1.compute.internal"
"ip-10-0-7-244.us-west-1.compute.internal"
"ip-10-0-36-12.us-west-1.compute.internal"

Solution

In this case the most likely culprit is the tolerations section of the DaemonSet config, which excludes control plane nodes. Instead, a more permissive toleration config, such as the one used by the OpenShift Machine Config Daemon (source), should be used.