[IAmJSD]

Root cause

Both Azure compute paths set ADMIN_PASSWORD_SECRET and ADMIN_EMAIL, which arms the admin-password sync (buildAdminPasswordSyncCallback, internal/server/app.go). When the admin changes their password, the sync calls azsecrets.SetSecret. The only Key Vault grant either runtime identity holds is the vault-wide Key Vault Secrets User, which is read-only, so the write gets a 403. The callback logged that at WARN and carried on. The password change had already been committed to the database, so the vault copy silently drifted.

On current main (a14dc8f2) both halves still reproduce:

• AKS: workload_kv_secrets_user is Secrets User (modules/compute/azure/aks/main.tf). fix(iac/azure): wire AKS workload identity so pods can assume the workload UAI #996 has since wired workload identity (fix(iac/azure): AKS workload identity is not wired up, so the UAI is unassumable by any pod #198), so once AKS is deployed this write path is live.
• Container Apps (audit finding A13c-004 on the issue): the only grant is compute_key_vault_secrets_user (Secrets User) in environments/azure/compute.tf. The secrets module's writeable_secrets_role Officer grant is count = 0 everywhere, because container_app_identity_principal_id is never passed. All three Azure environments deploy container-apps, so this half is what is actually deployed.

Fix

I took the issue's option 2 in spirit: an explicit write grant scoped as narrowly as Azure allows.

• modules/compute/azure/container-apps (admin_password_secret_writer) and modules/compute/azure/aks (workload_admin_password_secret_writer) each add an azurerm_role_assignment of Key Vault Secrets Officer:
  • Scope: <key_vault_id>/secrets/<admin_password_secret_name>, which is that one secret, not the vault. Key Vault RBAC supports secret-level role assignments. No built-in role is set-only, so delete/backup/restore on this one secret come with it, and nothing on any other secret. The vault-wide grant stays read-only.
  • Principal: the identity the app actually authenticates as. That is the Container App UAI (AZURE_CLIENT_ID) or the AKS workload UAI (federated to the pod service account).
  • Count: admin_password_secret_name != "" && admin_email != "", the same condition that arms the Go callback. The grant exists only when something will use it.
  • Plan-time stable: the scope is built from var.key_vault_id plus the secret name rather than the secret's resource_versionless_id. scope is ForceNew, and an apply-time-only value would replace the grant (the #1802 pattern).
  • admin_password_secret_name gains a validation that it must be a bare Key Vault secret name (^[0-9A-Za-z-]{1,127}$), so a path or ID cannot move the scope off the single secret.
• Bootstrap vs runtime (CLAUDE.md): both are runtime grants in the per-cloud compute modules. The deploy SP already holds Microsoft.Authorization/roleAssignments/write (ci-cd-permissions/locals_data.tf), the same permission it uses for the existing Secrets User, Crypto User and AcrPull assignments. The bootstrap does not need re-applying, and I added no fallback.
• Fail loudly (internal/server/app.go):
  • A failed PutSecret now logs at ERROR, naming the secret and stating it no longer matches the database password, with a hint to check the runtime identity's write access. Before, this was a WARN.
  • A failed user lookup used to return silently. It now also logs an ERROR, because the callback cannot tell whether the admin's copy just went stale.
  • The password is never logged, and success still logs at INFO.
  • I did not change the HTTP behavior: the password change is already committed, so failing the request would misreport it.

Deploy pause

Azure deploys have been paused since 2026-10-20 (deploy-azure workflows disabled). The Terraform half takes effect only on the next Azure deploy after the pause lifts. The PR is otherwise merge-ready, and merging it deploys nothing. The Go logging change ships with the next image build.

Closes #199

Verification

All of the following is unit, mock and fixture evidence. Nothing was planned or applied against a real subscription, and no real credentials were used.

Go (fail-before/pass-after): new tests in internal/server/app_coverage_test.go:

• TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError (ERROR names the secret and the cause, no WARN, no false success line, password not logged)
• TestBuildAdminPasswordSyncCallback_UserLookupFailureLogsError
• TestBuildAdminPasswordSyncCallback_QuietPathsLogNoError (success, non-admin and missing user log no ERROR)

# with internal/server/app.go reverted to main:
go test ./internal/server/ -count=1 -run TestBuildAdminPasswordSyncCallback
--- FAIL: TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError    app_coverage_test.go:146: Expected 1, got 0
--- FAIL: TestBuildAdminPasswordSyncCallback_UserLookupFailureLogsError   app_coverage_test.go:171: Expected 1, got 0
# with the fix: all 9 TestBuildAdminPasswordSyncCallback_* PASS
go test ./internal/server/ -count=1          # ok
go test ./internal/mocks/ -count=1           # ok
go build ./... && go vet ./... && go vet -tags integration ./...   # ok
golangci-lint v2.10.1 run --allow-parallel-runners ./internal/server/   # 0 issues
gocyclo -over 10 internal/server/app.go      # clean

Terraform (mocked terraform test, fail-before/pass-after): new tests/admin_password_writer.tftest.hcl in both modules.

• Both directions: with sync armed, exactly one grant scoped to <vault>/secrets/admin-password with Secrets Officer. Container-apps also asserts the principal is the app's UAI and that the secret matches the container's ADMIN_PASSWORD_SECRET. AKS asserts the same for the pod spec, and that the vault-wide grant is still Secrets User at vault scope.
• With no admin email or no secret name there is no grant, and a path-shaped secret name is rejected.

terraform -chdir=terraform/modules/compute/azure/<m> init -backend=false -lockfile=readonly -input=false
terraform -chdir=terraform/modules/compute/azure/<m> test
# container-apps: Success! 4 passed, 0 failed.   aks: Success! 5 passed, 0 failed.
# with main.tf/variables.tf reverted to main: both fail ("Reference to undeclared resource")

Mutation probes, each run against both modules, were all killed: vault-wide scope, dropping the admin_email half of the count gate, removing the name validation, and Secrets User instead of Officer.

Other checks:

• terraform fmt -check -recursive terraform/ passes. terraform validate passes in both modules and in terraform/environments/azure after init -backend=false.
• bash scripts/check-azure-kv-access-policy.sh passes (207 files), and scripts/test-azure-kv-access-policy.sh gives 13/13.
• The rewritten HCL-parsing guard from our in-flight fix/issue-204 branch, run on this change merged onto that branch, passes, and its harness gives 26/26.
• bash scripts/check-azure-role-parity.sh passes.
• actionlint .github/workflows/ci.yml and pre-commit run zizmor --files .github/workflows/ci.yml pass.
• git merge-tree against open chore(iac/aws): add multi_az support and run prod RDS Multi-AZ on a non-burstable class #1086, fix(iac/aws): route private egress through a persistent fck-nat ENI #1090 and fix(iac): bound every Terraform provider constraint below the next major #1203 is clean. With fix(iac): bound every Terraform provider constraint below the next major #1203's provider-constraint bump (~> 3.117) applied on top, both module tests still init with -lockfile=readonly and pass.

CI wiring: a new "Test Azure compute modules" step in the terraform-validate job (azure leg) runs init/fmt/test for every terraform/modules/compute/azure/*/tests. Both modules' .terraform.lock.hcl gain linux_amd64 and darwin_amd64 hashes, following the #1143 precedent, so -lockfile=readonly init works on the Linux runner. They previously carried a single platform's h1:.

Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree: queue/platform @ b56afb2b = origin/main a14dc8f2 plus these heads, merged in this order:

Branch	Tested head
fix/issue-204	76f85a4
fix/issue-438	78fe011
fix/issue-838	120b07f
fix/issue-143	aef1256
fix/issue-349	cfbb551
fix/issue-80	63405f9
fix/issue-199	36b9f8f
fix/issue-41	96fc237
fix/issue-299	6af14ed
fix/issue-324	3ab6729

fix/issue-324 and fix/issue-204 both edit .github/workflows/ci.yml next to the azure-kv-access-policy job. The tree kept #324's new "Run Function URL reader self-tests" step and #204's updated job comment; whichever PR merges second takes that same resolution.

Linux, Go 1.26.9, AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers. The full suite ran on ed0c7174, which differs from b56afb2b only by one .gitallowed line on fix/issue-324:

• Passed: go build ./..., go vet ./..., go vet -tags integration ./... and the full go test -race -short -count=1 ./....
• Static checks: golangci-lint v2.10.1 found 0 issues on every changed package, with and without --build-tags integration. gocyclo -over 10 is silent on the changed non-test files.
• Integration tests, every changed package that has them, one package at a time (-tags integration -count=1, 60-minute local timeout): internal/api, internal/purchase, internal/scheduler and internal/server all pass.
• Secrets: the first run caught a bare placeholder account ID in fix/issue-324's new tftest, now allowlisted with an anchored .gitallowed line. Re-run at b56afb2b: git-secrets, using the repo's scripts/setup-git-secrets.sh patterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged. scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.
• Nothing was applied to any cloud.

Gaps and things to eyeball

• AKS principal binding is not value-asserted. Under mock providers on Terraform 1.10, the cluster's kubelet_identity block (Optional+Computed) plans as an empty list, so aks_acr cannot be evaluated. Any apply run also fails at teardown, so the AKS runs are targeted plans, where principal_id is unknown. The reference is azurerm_user_assigned_identity.workload.principal_id, the same as the read grant next to it. The container-apps test applies and asserts its principal.
• Role dataActions are from documentation and in-tree usage, not an az role definition list response. As the issue notes, I could not query Azure. "Key Vault Secrets Officer" (Microsoft.KeyVault/vaults/secrets/*) and secret-level scopes are documented by Microsoft.
• A narrower alternative would be a custom role with only secrets/setSecret/action. That needs a bootstrap-created role definition (roleDefinitions/write is bootstrap-only), and the runtime lookup would fail every Azure deploy until a human re-applies the bootstrap. Scoping the built-in role to one secret seemed the better trade. Happy to switch if the owner prefers the custom role.

Out of scope (noted, not changed)

• modules/secrets/azure still carries the unused container_app_secrets_user / writeable_secrets_role grant (count = 0 in every environment). Its comment, and known_issues/18_tf_env_azure.md, describe it as the runtime's write path. Removing or rewording it is a separate cleanup.
• The issue's option 3 (stop setting ADMIN_PASSWORD_SECRET / ADMIN_EMAIL) was not taken, because the startup path also reads ADMIN_PASSWORD_SECRET.

A maintainer, please mirror the issue's triage labels on this PR: type/bug, priority/p3, severity/low, urgency/eventually, impact/internal, effort/s, triaged.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}