Add state machine for image switch (milestone 4c) by alicefr · Pull Request #50 · bootc-dev/bootc-operator

alicefr · 2026-06-03T09:50:25Z

Implements the daemon-side state machine that detects image mismatches between spec.desiredImage and the booted image,
stages updates via bootc switch in a background goroutine, and triggers reboot when desiredImageState == Booted.

Add Switch(ctx, image, apply) to the Executor interface
Rewrite the reconciler with async staging, cancel-on-spec-change, and single-patch-per-reconcile
Background goroutine signals completion via source.Channel, letting the reconcile loop handle all status updates through
one path
Apply operations (reboot) are never cancelled — the daemon restarts after reboot and reconciles fresh

alicefr · 2026-06-03T09:51:07Z

@jlebon I skipped the e2e test since we still need to define how to test the upgrade. The state machine is only checked by the env tests for now

jlebon · 2026-06-09T16:21:46Z

+
+	mu           sync.Mutex
+	switchImage  string
+	switchApply  bool


I think we discussed this at some point. My inclination is to not try to optimize this. I.e. let's not support an --apply special-case at all. We just do the bootc switch and then use the GenericEvent to immediately do another reconcile if desiredImageState is booted.

To clarify, --apply is not used during staging, we always stage with plain bootc switch . The --apply is only used in the reboot step: when the image is already staged and desiredImageState is Booted, we run bootc switch --apply to boot into it.

Are you suggesting we should decouple this further and trigger the reboot through a different mechanism (e.g. systemctl reboot) instead of bootc switch --apply?

Sorry, this is wrong bootc switch --apply doesn't reboot, but it is instead bootc upgrade --apply, the right command

The --apply is only used in the reboot step: when the image is already staged and desiredImageState is Booted, we run bootc switch --apply to boot into it.

OK, that makes sense and I guess it does match closer to what we'll actually do once we have --from-downloaded.

I do like the executor verb being called Reboot() though as you have now to make this clearer. So maybe then we just change it to take an image arg so it runs bootc switch --apply.

But yeah, I think the bug you're hitting is bootc-dev/bootc#2249.

alicefr · 2026-06-11T13:50:14Z

There are still some issue with this PR... investigating.

alicefr · 2026-06-11T15:14:58Z

I think we should prioritize #57 and implement at least the happy path test for the reboot

jlebon · 2026-06-12T16:30:15Z

+func (e *HostExecutor) Switch(ctx context.Context, image string) error {
+	log := logf.FromContext(ctx)
+
+	cmd := e.nsenterCmd(ctx, "bootc", "switch", image)


TODO link to bootc-dev/bootc#2137 would still be good here.

Should I rename the function Stage then? It doesn't follow bootc API but it makes the intention more explicit like the reboot

jlebon · 2026-06-12T16:37:32Z

          limits:
            cpu: 500m
-            memory: 128Mi
+            memory: 512Mi


The commit message needs a rationale.

jlebon · 2026-06-17T01:58:18Z

+	return nil
+}
+
+func (f *fakeExecutor) Upgrade(_ context.Context) error {


Shouldn't this be Reboot instead?

jlebon · 2026-06-17T02:16:06Z

 			Type:               bootcv1alpha1.NodeDegraded,
 			Status:             metav1.ConditionTrue,
 			Reason:             bootcv1alpha1.NodeReasonError,
 			Message:            fmt.Sprintf("failed to get bootc status: %v", err),


OK as is, but I wonder if it would be cleaner to instead define a new error type, e.g. degradedError which gets "caught" in the top-level Reconcile() loop and sets NodeReasonError and the message to from the degradedError object. See also invalidSpecError() in the controller code.

Actually, we can treat all the error from the reconcileBootcNode as Degraded errors and we don't need an extra type

Hmm, maybe. I think it's helpful to distinguish between "OS errors", and any other error (e.g. failed to get or patch some k8s object). I think only the former should cause a NodeDegraded condition.

The reconcileBootcNode does already return an error only if there are an OS error. Except if we we consider the bn.Status.Booted == nil not as an OS error. The other cases is when a bootc operation fails.

jlebon · 2026-06-17T02:36:22Z

+		return ctrl.Result{}, nil
+	}
+
+	_, desiredDigest, _ := strings.Cut(desiredImage, "@")


Yeah, should parse here.

jlebon · 2026-06-17T02:53:40Z

+	err := executor.Switch(ctx, image)
+
+	s.mu.Lock()
+	if ctx.Err() != nil {


I guess one TODO here is to switch to exec'ing the bootc switch async, and then a select here where we monitor the cancel channel so we can actually send a SIGINT to bootc.

jlebon · 2026-06-17T02:56:38Z

I haven't re-reviewed past the 2nd commit yet (tests).

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Rewrite the reconciler to detect image mismatches between spec.desiredImage and the booted image, stage via bootc switch in a background goroutine. Once, it finished to staged the image, the termination of the goroutine triggers once more the reconciliation loop which will detect that the system requires a reboot. The reconciliation function ensures that the bootc node transitions from Staging to Staged, and then to Rebooting. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Replace raw JSON bytes with a bootc.Status struct in the test fake. Status() serializes the struct via json.Marshal, and Switch() auto-mutates the status (staging sets Staged). Upgrade() records the call for test assertions. Add newBootcStatus() and newBootEntry() helpers to build test state without verbose JSON constants. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Add envtest cases for the daemon reconciler state machine: - TestStagingTriggered: image mismatch triggers bootc switch - TestStagingError: switch failure sets Degraded condition - TestAlreadyStaged: skip switch when image already staged - TestRebootingSet: reboot triggered when desiredImageState is Booted - TestRollback: restage when desired image changes - TestCancelInflightSwitch: spec change cancels in-flight switch Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

alicefr marked this pull request as draft June 3, 2026 09:52

alicefr force-pushed the milestone-4c branch from ef264c4 to 0456b98 Compare June 3, 2026 10:12

alicefr marked this pull request as ready for review June 3, 2026 11:50

jlebon reviewed Jun 9, 2026

View reviewed changes

alicefr force-pushed the milestone-4c branch 4 times, most recently from 7e1ff20 to 2afd447 Compare June 10, 2026 12:36

alicefr mentioned this pull request Jun 11, 2026

Detect external bootc status changes via fsnotify (milestone 4d) #59

Open

alicefr marked this pull request as draft June 11, 2026 13:49

alicefr force-pushed the milestone-4c branch 2 times, most recently from 56119dc to ba87f0d Compare June 11, 2026 14:21

alicefr marked this pull request as ready for review June 11, 2026 15:14

alicefr force-pushed the milestone-4c branch 2 times, most recently from e4557ce to 413cf0d Compare June 12, 2026 11:47

alicefr mentioned this pull request Jun 12, 2026

Set the target image reference for the bink node #57

Merged

jlebon force-pushed the milestone-4c branch from 413cf0d to abc1f6f Compare June 12, 2026 15:46

alicefr force-pushed the milestone-4c branch 2 times, most recently from 1226849 to ed7b12d Compare June 16, 2026 07:23

jlebon reviewed Jun 17, 2026

View reviewed changes

alicefr added 6 commits June 17, 2026 11:26

daemon: add Switch and Reboot methods to bootc executor

eb592b8

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

e2e: add UpdateImageDigestedPullSpec to e2e env

e1cf5bc

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

e2e: add TestUpdateReboot for full update lifecycle

ad9ea49

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

daemon: increase memory resource

e76449d

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

alicefr force-pushed the milestone-4c branch from ed7b12d to e76449d Compare June 17, 2026 14:07

Conversation

alicefr commented Jun 3, 2026

Uh oh!

alicefr commented Jun 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alicefr commented Jun 11, 2026

Uh oh!

alicefr commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jlebon commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants