First-sync apply ordering starves the ArgoCD controller of cluster-admin, deadlocking gated installs #887

Summary

On a clean install, the root Application's first sync applies the imperative ServiceAccount objects before the ClusterRoleBinding that grants the ArgoCD controller cluster-admin, so those applies fail with forbidden: cannot create resource "serviceaccounts". Ungated installs self-heal on the next automated sync (~3 min). With gitops.applicationHealthCheckEnabled: "true" (app-of-apps sync-wave gating), the first sync operation parks waiting on wave-0 health and the retry never happens — a hard deadlock on every clean install.

Evidence (OCP 4.22.17, patterns-operator v0.0.82, singleArgoCD)

The operator-provisioned controller role (verbatim rules):

['']                          | ['serviceaccounts']  | ['impersonate']     ← no create
['rbac.authorization.k8s.io'] | ['*']                | ['*']

So the failure is purely apply ordering, not a missing permission in the operator's design: ArgoCD orders wave-0 resources by kind (Namespace → ServiceAccount → ConfigMap → ClusterRole → ClusterRoleBinding → …), and the cluster-admin grant ships in the same wave as the ServiceAccounts that need it.

This is not pattern-specific

The clustergroup chart's own argocd-super-role.yaml documents the same hazard:

# This CRB is also pre-created by the ACM policy (application-policies.yaml)
# on managed clusters to ensure the ArgoCD controller has cluster-admin
# before the first sync. It's kept here for non-ACM deployments.

i.e. managed/spoke clusters are protected by ACM pre-creation; standalone installs are not. Any pattern using singleArgoCD: true + imperative jobs (wave-0 ServiceAccounts) is affected the moment gating is enabled — which the framework needs for wave ordering to work at all, since ArgoCD 1.8 dropped the Application health check (argoproj/argo-cd#3781).

Option A (preferred): the operator creates the instance's cluster-admin binding

The operator already owns the ArgoCD instance identity: it creates the ArgoCD CR and injects global.vpArgoNamespace = getClusterWideArgoNamespace() (argo.go, ParamVpArgoNamespace) into the root Application's Helm parameters. It knows the name and namespace, so it can create — idempotently, at instance creation / reconcile, before the root Application exists — the equivalent of:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <vpArgoNamespace>-cluster-admin-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: <vpArgoNamespace>-argocd-application-controller
    namespace: <vpArgoNamespace>

Notes:

• Mirrors exactly what the ACM policy already does on managed clusters.
• Use activeArgoNamespace/activeArgoName (utils.go) so the legacy openshift-gitops path is handled identically.
• Consider an opt-out CM key (e.g. gitops.createClusterAdminBinding, default true) so multi-tenant / least-privilege installs can keep the current behaviour — the chart warns "ONLY USE THIS FOR MANAGING CLUSTERS NOT FOR REGULAR USERS", and that concern applies to the operator-created binding equally.
• Fixes the class for every pattern, permanently, and lets patterns delete their own out-of-band bridges.

Option B (alternative): move the chart's binding to an earlier sync wave

If the operator prefers not to create cluster-scoped RBAC, the equivalent fix is a one-annotation change in validatedpatterns/clustergroup-chart (templates/plumbing/argocd-super-role.yaml):

metadata:
  name: {{ $.Values.global.vpArgoNamespace }}-cluster-admin-rolebinding
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Why this is sufficient: the controller can already create ClusterRoleBindings before it has cluster-admin (rbac.* is fully granted by the operator-provisioned role — empirically it created the binding mid-sync), and ClusterRoleBindings have no health status, so wave −1 completes immediately and wave 0's ServiceAccount applies then run with cluster-admin.

Trade-offs vs Option A: smaller change and no operator release, but it only fixes patterns that consume a released clustergroup chart version — and for disconnected users it additionally requires re-mirroring the chart.

Interim mitigation (pattern-side, declarative)

Patterns can work around it today with a clusterGroup.extraObjects CRB under a distinct name at sync-wave: "-1"; that is what validatedpatterns/coco-pattern#165 is doing while this is decided. It is a bridge, not a fix: it hardcodes the operator's vp-gitops constant and leaves a redundant grant next to the chart's own binding.

Related

• fix(airgap): mode-agnostic disconnected fixes and sync-wave gating coco-pattern#165 — wave gating + connected/disconnected parity; blocked on this class of first-sync ordering issue.
• patterns-operator-config ConfigMap omits gitops.applicationHealthCheckEnabled, so app-of-apps sync-wave gating is silently inert pattern-install-chart#52 / Upgrade version to 0.0.10 #53 — the companion gap where gitops.applicationHealthCheckEnabled was not rendered at all.