Skip to content

fix(infra): retry the VPC delete until DigitalOcean releases a cluster's subnets - #30

Merged
psanders merged 1 commit into
mainfrom
fix/vpc-delete-retry
Nov 10, 2026
Merged

psanders merged 1 commit into
mainfrom
fix/vpc-delete-retry

Conversation

@psanders

Copy link
Copy Markdown
Member

Problem

Runs with a DOKS cluster ended with exit 3 at teardown:

perfkit: could not delete VPC <id> (perfkit-<runId>) after 3 attempts: Can not delete VPC with active Subnets. Run: perfkit sweep --run <runId>

DigitalOcean releases the cluster's subnets some minutes after the cluster delete returns. The VPC's members list is already empty by then, so the existing members wait passed and the delete got only 3 tries over 6 seconds.

Change

  • New createDeleteVpc (src/infra/createDeleteVpc.ts): polls GET /v2/vpcs/{id}/members until empty, then deletes; a refused delete is retried with backoff (5, 10, 20, 30, then every 60 s) until the VPC timeout (5 min, or 10 min after a cluster). The last try lands on the deadline. Failure reason: after 10 minutes and N attempts: <message>.
  • createTeardown uses it in place of the members wait plus 3-try delete. Sweep goes through teardown, so it behaves the same.
  • Unit tests with the fake client and clock (no real waits): success after members drain, success after refused deletes, give-up at the deadline, members that never drain, a failing members lookup, validation failure; plus a teardown test.

Exit code 3 at collect

createCollect returns 0 or 1 from the verdicts (EXIT_NOT_PASSED = 1) and turns the code into 3 only when teardown reports a failure (or the run failed / produced nothing). So the exit 3 here came from the VPC failure alone; verdict exit codes are unchanged.

Verify

npm test (816 passing), npm run typecheck, npm run lint (ESLint + Prettier).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WSt25UoKGj2mgDNQ4dHVrB

…r's subnets

DigitalOcean refuses to delete a VPC with "Can not delete VPC with active
Subnets" for minutes after a DOKS cluster's delete returns, even when the
VPC lists no members. Teardown gave up after 3 tries over 6 seconds, so
runs with a cluster ended with exit 3.

createDeleteVpc waits for the VPC to have no members and retries a refused
delete with backoff (5, 10, 20, 30, then every 60 seconds) until the
existing VPC timeout: 5 minutes, or 10 after a cluster. Teardown and sweep
both use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSt25UoKGj2mgDNQ4dHVrB
@psanders
psanders merged commit ea0a191 into main Nov 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant