Skip to content

fix(iac/azure): cleanup-function and AKS grant Key Vault access the RBAC-enabled vault ignores #1621

Closed

Description

@cristim

Reviewed commit: be11bdcb5. Note: origin/main moved to 3e9660d06 during the review; re-verify against current main before changing code, since a finding may have been fixed or moved.

Blast radius: our own hosted infrastructure. Failure is a runtime outage on the Azure deployment path, not customer exposure.

Where

  • terraform/modules/compute/azure/cleanup-function/main.tf:31-40 (azurerm_key_vault_access_policy)
  • terraform/modules/compute/azure/aks/main.tf:105-114 (azurerm_key_vault_access_policy)
  • terraform/modules/secrets/azure/main.tf:39 (enable_rbac_authorization = true on the vault these target)
  • Consumers that do it correctly: terraform/modules/compute/azure/container-apps/main.tf:309, container-apps/scheduled-tasks.tf:439-461, terraform/modules/secrets/azure/main.tf:295-301

What

The vault this project provisions sets enable_rbac_authorization = true (secrets/azure/main.tf:39). An RBAC-enabled Key Vault ignores access policies entirely: data-plane authorization comes from Azure RBAC role assignments only, and the accessPolicies array on the vault is not consulted.

The Azure cleanup function and the AKS workload identity both grant themselves access with azurerm_key_vault_access_policy. Terraform applies these resources successfully because the control plane accepts the write; the data plane does not honour it. Every other consumer in this repo already uses azurerm_role_assignment, so the correct pattern exists in-tree and simply was not applied to these two.

Failure scenario

terraform apply reports success with no warning. Then:

  • The Azure cleanup function 403s on every attempt to read db-password at runtime, so expired sessions and stuck executions are never cleaned up. Nothing alarms, because the cleanup job's failure is not itself monitored.
  • The AKS workload identity likewise cannot read any secret from the vault, so any pod depending on it fails at startup or at first secret fetch.

The whole failure mode is invisible at plan and apply time and only surfaces as runtime 403s, which is the worst possible place to discover it.

Fix direction

Replace both azurerm_key_vault_access_policy resources with:

resource "azurerm_role_assignment" "..." {
  scope                = var.key_vault_id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = <identity principal id>
}

Then add a repo guard: grep for azurerm_key_vault_access_policy in CI and fail, since the project's vault is RBAC-enabled by design and no correct use of an access policy exists here. Note that role assignments need propagation time; see the related depends_on finding.

Related

Activity

  1. cristim commented on Aug 13, 2026

    @cristim
    MemberAuthor

    Re-triaging urgency/now to urgency/this-sprint, with the reasoning, rather than working around the label quietly.

    Neither affected module is currently deployed

    Verified against the #1817 branch by enumeration, not sampling:

    • cleanup-function is instantiated nowhere. rg 'cleanup-function' --glob '*.tf' returns only the module's own files, the new guard scripts and their fixtures. No module block sources it.
    • The AKS module is gated off. terraform/environments/azure/compute.tf:129-131 declares module "compute_aks" with count = var.compute_platform == "aks" ? 1 : 0, and all three environments select Container Apps: github-dev.tfvars:18, github-staging.tfvars:18, github-prod.tfvars:18 are each compute_platform = "container-apps" (the variable default in variables.tf:60 is also container-apps).
    • The live Azure cleanup path is the Logic App in container-apps/scheduled-tasks.tf:455-461, which already used an azurerm_role_assignment and was never affected.

    What that means for this issue

    This issue's description, and #1817's original body, both framed this as a live outage: the cleanup function 403ing on every attempt to read db-password, with expired sessions and stuck executions never cleaned up and nothing alarming. That outage cannot currently occur, because nothing deploys that function.

    The defect is real and worth fixing: both modules are selectable, the grant would be silently inert the moment either is deployed, and the copy-paste hazard that produced it is exactly what the new guard ratchets against. But it is a latent defect plus a ratchet, not an incident.

    Keeping severity/high: if it does occur, the failure is a silent runtime 403 with a green terraform apply, which is the worst place to discover it. Dropping urgency/now: nothing is broken in any deployed environment today, and treating this as now-urgent displaces work that is.

    A second, separate gap found while verifying

    Fixing the grant does not make AKS pods able to read secrets. The role assignment lands on a user-assigned identity no pod can assume: the module enables neither oidc_issuer_enabled nor workload_identity_enabled, there is no azurerm_federated_identity_credential anywhere under terraform/, and the pod template carries no azure.workload.identity/use label. Only the service-account annotation exists.

    That is pre-existing and out of scope for #1817. Filed as LeanerCloud/cloud-commitments-platform#198.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions