Symptom
TestWorkerPoolPodTemplateUpdate intermittently fails on main with an optimistic-concurrency conflict:
Operation cannot be fulfilled on workerpools.ate.dev "test-template-update":
the object has been modified; please apply your changes to the latest version and try again
Seen on three independent runs on main. Each was verified: all are failure, all carry the conflict signature in the failed-step log, and in each the failing test is TestWorkerPoolPodTemplateUpdate itself.
Cause
The test does a bare get-then-update on the WorkerPool:
k8sClient.Get(ctx, types.NamespacedName{...}, wp)
wp.Spec.Template.NodeSelector = map[string]string{"workload": "updated"}
k8sClient.Update(ctx, wp) // no conflict handlingWorkerPoolReconciler writes the same object concurrently — r.Status().Update(ctx, wp) at cmd/atecontroller/internal/controllers/workerpool_controller.go:142 — which bumps resourceVersion. If that lands between the test's read and its write, the write is rejected and the test fails. The test also runs with t.Parallel(), so the reconcile loop is under load from the rest of the package.
The window is narrow enough that it never shows up locally and only surfaces on loaded CI runners; I did not measure its width.
The file already has a helper for exactly this, updateWorkerPoolSpec, which retries on conflict; the comment on its sibling spells the problem out:
the controller reconciles the Deployment concurrently (bumping its resourceVersion on every SSA apply), so a plain get-then-update flakes with "the object has been modified" under load.
This test simply does not use it.
Verification approach
Worth noting for whoever reviews the fix: -count=N proves nothing here. The unfixed test passes at -count=100 -race locally because the window is too narrow to hit. The conflict has to be constructed deliberately — insert a write between the test's read and its write — at which point the old code fails with the exact CI error and the helper-based version retries and lands.
Symptom
TestWorkerPoolPodTemplateUpdateintermittently fails on main with an optimistic-concurrency conflict:Seen on three independent runs on
main. Each was verified: all arefailure, all carry the conflict signature in the failed-step log, and in each the failing test isTestWorkerPoolPodTemplateUpdateitself.Cause
The test does a bare get-then-update on the WorkerPool:
WorkerPoolReconcilerwrites the same object concurrently —r.Status().Update(ctx, wp)atcmd/atecontroller/internal/controllers/workerpool_controller.go:142— which bumpsresourceVersion. If that lands between the test's read and its write, the write is rejected and the test fails. The test also runs witht.Parallel(), so the reconcile loop is under load from the rest of the package.The window is narrow enough that it never shows up locally and only surfaces on loaded CI runners; I did not measure its width.
The file already has a helper for exactly this,
updateWorkerPoolSpec, which retries on conflict; the comment on its sibling spells the problem out:This test simply does not use it.
Verification approach
Worth noting for whoever reviews the fix:
-count=Nproves nothing here. The unfixed test passes at-count=100 -racelocally because the window is too narrow to hit. The conflict has to be constructed deliberately — insert a write between the test's read and its write — at which point the old code fails with the exact CI error and the helper-based version retries and lands.