feat(planner): add replace-pod task to NodeUpdate plan (closes #211)

[bdchatham]

Closes #211.

Summary

K8s native StatefulSet RollingUpdate refuses to delete pods that are not Ready, which deadlocks the rollout when seid is intentionally unready — the canonical case is halted at a chain upgrade height awaiting a binary swap. The NodeUpdate plan would loop forever in observe-image until a human ran kubectl delete pod. We hit this on 2026-05-08 during pacific-1's archive-0 v6.3.1 → v6.4.3 swap; ~9000 reconcile cycles before manual unstick.

This PR inserts a new replace-pod task between apply-service and observe-image in the NodeUpdate plan:

apply-statefulset → apply-service → replace-pod → observe-image → mark-ready

replace-pod:

• Reads the StatefulSet, identifies its Status.UpdateRevision
• Lists pods owned by the StatefulSet (via Selector.MatchLabels)
• Deletes any pod whose controller-revision-hash label != UpdateRevision
• Skips pods already terminating (deletionTimestamp set)
• StatefulSet then recreates the pods at the new revision via its standard creation path — that path doesn't gate on readiness, so the deadlock is broken

Idempotent: re-running on a converged StatefulSet (CurrentRevision == UpdateRevision) is a no-op completion. Missing StatefulSet or empty UpdateRevision returns transient wait so the executor retries on next reconcile.

Files

File	Change
internal/task/replace_pod.go (new)	Task implementation
internal/task/replace_pod_test.go (new)	5 tests (stale pod / already-converged / terminating / missing STS / empty UpdateRevision)
internal/task/task.go	Register TaskTypeReplacePod in deserializer registry
internal/planner/planner.go	Insert task in NodeUpdate plan + add params case
internal/planner/node_update_test.go	Update expected task list (4 → 5 tasks)
CLAUDE.md	Document the new task in the NodeUpdate description

Test plan

• All new replace-pod tests pass (5/5)
• Existing planner / task / controller tests pass after the plan-shape change
• CI green
• Empirical verification: next pacific-1 chain upgrade (or manual archive-0 image bump) triggers a NodeUpdate plan that completes without operator intervention. The pod recreates within ~30s of the apply-statefulset task completing, regardless of seid's readiness state at the moment of the upgrade halt.

References

• NodeUpdate plan stalls forever when existing pod is unready (chain-upgrade halt) #211 — issue with full failure analysis
• feat(ensure_pvc): accept ReadWriteOncePod on imported archive PVCs #214 — RWOP validator change (separate prerequisite for SELinux mount labeling)
• pacific-1 archive-0 v6.4.3 swap incident: 2026-05-08, ~9000 reconcile cycles before manual kubectl delete pod unstuck

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes rollout semantics for all SeiNode StatefulSets by switching to OnDelete updates and introducing controller-driven pod deletions; mistakes could cause stuck updates or unintended pod restarts. Also expands RBAC to allow pod deletions and adds cache-bypassing reads that could mask eventual consistency issues if misused.

Overview
Fixes SeiNode image rollouts that can deadlock when pods are intentionally NotReady by inserting a new replace-pod step into the NodeUpdate plan (apply-statefulset → apply-service → replace-pod → observe-image → mark-ready).

To support controller-driven rollouts, generated StatefulSets now use OnDelete update strategy, the new replace-pod task deletes old-revision pods (single-replica only) to force recreation at the new revision, and observe-image (plus task execution wiring) now uses an uncached APIReader to read freshly-applied resources. RBAC is updated to grant pod delete, and tests/docs are updated accordingly.

^{Reviewed by Cursor Bugbot for commit aac07fa. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note on rollback

If this PR is reverted in the controller, in-cluster StatefulSets will keep updateStrategy: OnDelete (server-side apply does not reset unset fields to defaults). The pre-PR controller has no replace-pod task, so image bumps would stall on halted/unready pods exactly as they did before this PR — recoverable, not wedging, but rollback does not auto-restore RollingUpdate. To fully revert, edit the StatefulSets manually: kubectl patch sts <name> -n <ns> --type=json -p '[{"op":"replace","path":"/spec/updateStrategy/type","value":"RollingUpdate"}]'.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6ed60e5. Configure here.}