cocoonset: 409 conflict and 403 namespace-terminating surface as reconcile errors

[tonicmuroq]

Summary

cocoonset.Reconciler returns two expected apiserver races as reconcile errors. Neither leads to incorrect state, but both log ERR Reconciler error and increment controller_runtime_reconcile_errors_total, and one of them burns ~10 backoff retries per object on an operation that can never succeed.

Repo-wide, only two apierror classes are ever classified:

classification	call sites
apierrors.IsNotFound	14
apierrors.IsAlreadyExists	5
apierrors.IsConflict	0
apierrors.IsForbidden	0

Both cases below were observed on prod-cocoon on 2026-08-03 (times UTC).

Case A — 409 conflict on the finalizer Update

cocoonset/reconciler.go:77

if controllerutil.AddFinalizer(&cs, finalizerName) {
    if err := r.Update(ctx, &cs); err != nil {
        return ctrl.Result{}, fmt.Errorf("add finalizer: %w", err)
    }
    return ctrl.Result{Requeue: true}, nil
}

ERR Reconciler error controller=cocoonset controllerKind=CocoonSet CocoonSet=live-cocoonset/vm-b1b626ec
error="add finalizer: Operation cannot be fulfilled on cocoonsets.cocoonset.cocoonstack.io \"vm-b1b626ec\": the object has been modified; please apply your changes to the latest version and try again"

r.Update is issued against a cache-sourced object. When another writer (webhook defaulting, a status write) bumps resourceVersion first, the update returns 409. This is expected under informer cache staleness, not an exceptional condition — every occurrence succeeded on the following reconcile.

9 occurrences on 2026-08-03, spread across default/e2e-peer-probe (6), live-cocoonset/vm-b1b626ec (1), netconn-test/netprobe-a (1), netconn-test/netprobe-c (1).

Suggested: treat apierrors.IsConflict as a quiet requeue, or add the finalizer with a merge patch instead of Update.

Case B — 403 forbidden while the namespace terminates

cocoonset/reconciler.go:200

if err := r.Create(ctx, mainPod); err != nil {
    if apierrors.IsAlreadyExists(err) {
        // Old pod still Terminating; requeue and wait.
        return ctrl.Result{RequeueAfter: requeueWaitForMain}, nil
    }
    return ctrl.Result{}, fmt.Errorf("create main agent: %w", err)
}

ERR Reconciler error controller=cocoonset controllerKind=CocoonSet CocoonSet=netconn-test/netprobe-b
error="create main agent: pods \"netprobe-b-0\" is forbidden: unable to create new content in namespace netconn-test because it is being terminated"

30 occurrences within ~2 minutes (12:03–12:05) across netprobe-a, netprobe-b and netprobe-c after the netconn-test namespace was deleted. The counter went 9 → 39 and then stayed flat once reconcileDelete removed the finalizers.

Sequence: namespace GC deletes the child pods first. The cached CocoonSet does not yet carry a deletionTimestamp, so Reconcile passes the delete branch at cocoonset/reconciler.go:73 and reaches classified.main == nil → createMainAgent → 403. The delete-first ordering is correct; this cache-lag window cannot be closed by ordering alone.

A 403 whose cause is a terminating namespace is terminal: retrying cannot succeed while the namespace terminates, and once termination completes the CocoonSet no longer exists. The backoff retries are pure waste.

Suggested: treat apierrors.IsForbidden from the pod Create as terminal — stop requeueing and do not return an error.

Impact

No incorrect state and no user-visible effect in the observed instances. workqueue_depth{controller="cocoonset"} was 0 at every 60s sample throughout, and the only production VM involved (live-cocoonset/vm-b1b626ec) succeeded on retry. The cost is wasted reconciles plus alert noise: controller_runtime_reconcile_errors_total is what CocoonOperatorReconcileErrors alerts on, at rate > 0. The rule shape is tracked separately in simular-ai/ops#20.