Skip to content

Deploy creates orphaned KMS keys on every run #337

@jesseturner21

Description

@jesseturner21

Problem

Every agentcore deploy creates a new KMS key and calls SetTokenVaultCMKCommand to re-configure the account-level token-vault, even if a key already exists from a previous deploy. The old key is never deleted, disabled, or reused.

This means:

  • Each deploy accumulates an orphaned KMS key (~$1/month per key)
  • remove all + deploy (teardown) destroys the CloudFormation stack but does not delete the KMS key
  • The identityKmsKeyArn stored in deployed-state.json is lost when the target entry is cleaned up during teardown
  • Concurrent deploys to the same account race on SetTokenVaultCMKCommand, causing "conflicting request during InitializeMutation" errors

Root Cause

setupTokenVaultKms in src/cli/operations/deploy/pre-deploy-identity.ts:91-121 unconditionally creates a new KMS key via CreateKeyCommand and sets it on the token-vault. It does not:

  1. Check if a KMS key already exists in deployed state and reuse it
  2. Clean up old keys when creating a new one

The teardown flow in src/cli/operations/deploy/teardown.ts does not call ScheduleKeyDeletion or DisableKey for the stored identityKmsKeyArn.

Suggested Fix

  1. Reuse existing key: Before creating a new KMS key, check deployed-state.json for an existing identityKmsKeyArn. If present and the key is still valid (enabled), reuse it instead of creating a new one.
  2. Cleanup on teardown: When performing a teardown deploy, schedule deletion of the KMS key stored in identityKmsKeyArn using ScheduleKeyDeletionCommand.
  3. Handle concurrent deploys: The SetTokenVaultCMKCommand call should retry with backoff when it receives a mutation-in-progress error.

Impact

  • Every user accumulates ~$1/month in orphaned KMS keys per deploy
  • CI/CD pipelines and e2e test suites that deploy frequently create many orphaned keys
  • Concurrent deploys fail due to KMS mutation race condition

Relevant Files

  • src/cli/operations/deploy/pre-deploy-identity.tssetupTokenVaultKms()
  • src/cli/operations/identity/api-key-credential-provider.tssetTokenVaultKmsKey()
  • src/cli/operations/deploy/teardown.tsperformStackTeardown()
  • src/schema/schemas/deployed-state.tsidentityKmsKeyArn field

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions