Skip to content

fix(ci): remove broken integ teardown safety net + map stack name to commit sha#567

Merged
scottschreckengaust merged 4 commits into
aws-samples:mainfrom
ayushtr-aws:fix/566-remove-integ-teardown-safety-net
Jul 9, 2026
Merged

fix(ci): remove broken integ teardown safety net + map stack name to commit sha#567
scottschreckengaust merged 4 commits into
aws-samples:mainfrom
ayushtr-aws:fix/566-remove-integ-teardown-safety-net

Conversation

@ayushtr-aws

@ayushtr-aws ayushtr-aws commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Two coupled changes to .github/workflows/integ.yml (+ the integ smoke test), both about the integ harness's teardown/naming plumbing.

1. Remove the broken Ensure stack torn down safety net (#566)

The if: always() backstop ran raw aws cloudformation delete-stack / wait stack-delete-complete as the GitHub OIDC role, which has no cloudformation:DeleteStack or DescribeStacks permission. Both calls AccessDenied on every run; the un-guarded wait exits 255 and fails the job even when the integ test passed and integ-runner already destroyed the stack.

Evidence — run 28892156232: SUCCESS integ.task-api-smoke-TaskApiSmoke/DefaultTest 291.541s (integ-runner's own deploy → assert → destroy completed), then the safety-net step failed with AccessDenied on both CloudFormation calls.

integ-runner remains the teardown path — it forces cdk destroy on success and failure via the CDK bootstrap roles (which hold the CloudFormation permissions). Stranded-stack reclaim for cancelled/crashed runs is owned by #400 (non-blocking cleanup) and #72 (scheduled sweep).

2. Map the integ stack name to the commit SHA (int-<short-sha>-<run-number>)

Replaces the single hardcoded backgroundagent-integ stack name with a run-unique int-<short-sha>-<run-number> derived from the resolved head SHA, threaded into integ-runner via a new INTEG_STACK_NAME env var. The test falls back to backgroundagent-integ-local so the local mise //cdk:integ path is unchanged.

Why it pairs with #1: with the blocking safety net gone, a stranded stack from a cancelled/crashed run would otherwise sit under one fixed name and head-of-line block every future run (can't CREATE a name that already exists in DELETE_FAILED, and cancel-in-progress: false queues behind it). A run-unique name removes that failure mode (the run number also makes same-commit re-runs distinct); the int- prefix marks the stack ephemeral for #72's sweep. The concurrency group is now a cost/parallelism bound, not a correctness requirement.

Scope note: the naming change is a partial pull-forward of #400's first ask (ephemeral run-unique names). #400 remains open for its other two asks (offline cdk diff for the approver, non-blocking best-effort cleanup). Flagging for the reviewer since #400 isn't yet approved — happy to split this back out if preferred.

Validation

  • mise //cdk:compile — clean
  • mise //cdk:eslint — clean
  • zizmor .github/workflows/integ.yml — no findings

Area

  • tooling — root mise.toml, scripts, CI workflows
  • cdk — infrastructure, handlers, constructs (integ smoke test only)

Related

Fixes #566. Partially addresses #400 (naming). Related: #72, #236.

Changes

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

… workflow (aws-samples#566)

The if: always() backstop ran raw aws cloudformation delete-stack /
wait stack-delete-complete as the GitHub OIDC role, which has no
cloudformation:DeleteStack or DescribeStacks permission. Both calls
AccessDenied on every run; the un-guarded wait exits 255 and fails the
job even when the integ test passed and integ-runner already destroyed
the stack (e.g. run 28892156232: SUCCESS in 291s, then teardown-step
failure).

integ-runner remains the teardown path (cdk destroy on success and
failure via the CDK bootstrap roles). Stranded-stack reclaim for
cancelled/crashed runs is owned by aws-samples#400 (non-blocking cleanup) and aws-samples#72
(scheduled sweep).
@ayushtr-aws ayushtr-aws requested a review from a team as a code owner July 9, 2026 17:55
Replace the single hardcoded backgroundagent-integ stack name with a
run-unique int-<short-sha> derived from the resolved head SHA, threaded
into integ-runner via INTEG_STACK_NAME. The test falls back to a stable
backgroundagent-integ-local name so the local mise //cdk:integ path is
unchanged.

A stranded stack from a cancelled/crashed run no longer collides with or
head-of-line blocks a later run under one fixed name; the int- prefix
marks it ephemeral for the scheduled sweep (aws-samples#72). Concurrency-comment
updated: run-unique names make single-concurrency a cost bound, not a
correctness requirement.

Partial pull-forward of aws-samples#400's ephemeral-name ask, scoped to naming.
@ayushtr-aws ayushtr-aws requested a review from a team as a code owner July 9, 2026 18:08
@ayushtr-aws ayushtr-aws changed the title fix(ci): remove broken 'Ensure stack torn down' safety net from integ… fix(ci): remove broken integ teardown safety net + map stack name to commit sha Jul 9, 2026
Same-commit re-runs (re-run, or dispatch + workflow_run on one SHA) now
get distinct stack names, matching aws-samples#400's proposed int-{short-sha}-{run}
scheme.
@codecov-commenter

codecov-commenter commented Jul 9, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@092361e). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #567   +/-   ##
=======================================
  Coverage        ?   88.99%           
=======================================
  Files           ?      222           
  Lines           ?    53019           
  Branches        ?     5531           
=======================================
  Hits            ?    47182           
  Misses          ?     5837           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread .github/workflows/integ.yml Outdated

@scottschreckengaust scottschreckengaust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a unique stackName

Co-authored-by: Scott Schreckengaust <scottschreckengaust@users.noreply.github.com>

@scottschreckengaust scottschreckengaust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@scottschreckengaust scottschreckengaust added this pull request to the merge queue Jul 9, 2026
Merged via the queue into aws-samples:main with commit ed7f9f4 Jul 9, 2026
4 checks passed
ayushtr-aws added a commit to ayushtr-aws/sample-autonomous-cloud-coding-agents that referenced this pull request Jul 9, 2026
Fold in upstream aws-samples#567 (integ.yml overhaul) which conflicted with the
Phase-1 changes in the same job:
- Drop the raw-CLI "Ensure stacks torn down" safety net: the GitHub OIDC
  role has no cloudformation:DeleteStack/DescribeStacks, so it only ever
  AccessDenied and failed otherwise-passing runs (aws-samples#566/aws-samples#567). integ-runner
  still tears down via the CDK bootstrap roles; stranded-stack reclaim is
  owned by the sweeper (aws-samples#72) and non-blocking cleanup (aws-samples#400).
- Set both stack-name env vars on the run step so the two integ tests get
  distinct CloudFormation names: INTEG_STACK_NAME=int-<sha>-<run> for the
  Phase-0 smoke test (aws-samples#567's scheme) and COMMIT_HASH for the Phase-1
  lifecycle test (int-<hash>). Keep the plain `mise //cdk:integ` invocation
  (our mise.toml runs integ-runner --verbose and takes config via env, not
  the -c github:* CDK context, which no integ test source reads).
- Keep the Phase-1 gate env (INTEG_SANDBOX_REPO/INTEG_PAT_SECRET_ID) and the
  bgagent/* sandbox branch/PR cleanup step.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): remove broken 'Ensure stack torn down' safety net from integ workflow

3 participants