Skip to content

[AAASM-5675] 🐛 (examples): Gate Dependabot manifest coverage and fix the red live lane install - #439

Open
Chisanan232 wants to merge 8 commits into
mainfrom
v0.0.1/AAASM-5675/fix/dependabot_coverage_gate
Open

[AAASM-5675] 🐛 (examples): Gate Dependabot manifest coverage and fix the red live lane install#439
Chisanan232 wants to merge 8 commits into
mainfrom
v0.0.1/AAASM-5675/fix/dependabot_coverage_gate

Conversation

@Chisanan232

@Chisanan232 Chisanan232 commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changed

AAASM-5675 describes a defect pair: a Dependabot security job that fails against a directory its config cannot cover, and a repository where nothing makes that failure visible. This PR addresses the second half in the examples repo and reports measurements for the first.

The gate. scripts/check_dependabot_coverage.py asserts two properties against the real tree: each discovered dependency manifest directory (pnpm-lock.yaml, go.mod, uv.lock, plus .github/workflows/ for github-actions) maps to an entry of its ecosystem in .github/dependabot.yml, and each configured directory pattern expands to at least one manifest. The second assertion is the one that matters most — a mistyped directory: reads as coverage while delivering zero, and is indistinguishable from the original bug without expanding it against the tree. Wired into CI on pull requests and pushes to main (.github/workflows/dependabot-coverage.yml), with 17 unit tests.

Why main is red. The three live: check runs fail at .github/scripts/start-aasm.sh line 18, brew: command not found, exit 127 — the jobs run on ubuntu-latest, where Homebrew is absent. That is a repo-local scripting bug, and it was masking the rc-gated condition the lane documents. The script now installs aasm and aa-api-server from the release's Linux tarballs, checksum-verified against the release SHA256SUMS.

Stale claims corrected. Measuring the above turned up three inaccurate claims repeated across the live-lane docs; see the table below.

The brew fix, measured on a real Linux runner

verify-live.yml was dispatched against this branch — run 31789721513 — rather than asserting the fix from a reading of the script. Identical result on the python, node and go jobs:

Installing the aasm CLI and aa-api-server from ai-agent-assembly/agent-assembly@v0.0.1-rc.6 (linux-amd64)...
Verifying checksums against the release SHA256SUMS...
aasm-api-v0.0.1-rc.6-linux-amd64.tar.gz: OK
aasm-cli-v0.0.1-rc.6-linux-amd64.tar.gz: OK
Installed: aasm 0.0.1-rc.6
Starting the local gateway (aasm start --mode local)...
Waiting for the gateway REST surface at http://127.0.0.1:7700/api/v1/health ...
aa-api: generated admin API key (set AASM_API_KEY to reuse): aa_********
ERROR: aasm gateway did not become healthy within 120s.

Before: exit 127 at line 18, brew: command not found, 7 seconds in. After: the CLI installs, aasm --version answers, aa-api starts, and the job fails at the 120s health-check — the condition the script documents. That is the whole claim being made for this change; it does not make the lane pass.

This run also reproduces the continue-on-error measurement independently: run conclusion success, three job conclusions failure.

A lead, recorded rather than acted on. aa-api announces generated admin API key (set AASM_API_KEY to reuse), and assert-agent.sh / the health probe call curl -fsS with no key — so a 401 would make -f fail and the loop spin to timeout. That is a hypothesis, untested here: it was not chased because the lane is separately rc-gated on AAASM-4447/4467/4468/4469/4446, so clearing it would not turn the lane green, and guessing at an auth header in a script this PR cannot verify end-to-end would substitute speculation for the measurement above.

js-yaml — measured, already patched

The ticket's AC1 is satisfied in this repo, by earlier work rather than by this PR. Recorded here because "the absence of a red alert is not evidence of a green one" is the ticket's own standard:

Lockfile js-yaml Advisory status
node/mastra/pnpm-lock.yaml 4.3.1 Above GHSA-5p4m-2wfm-xmqj (>= 4.0.0, < 4.3.1) and GHSA-52cp-r559-cp3m (>= 4.0.0, < 4.3.0). Patched by #405
scenarios/approval-gates/node/pnpm-lock.yaml 5.2.3 Above the three 5.x ranges — GHSA-pm4m-ph32-ghv5 (<= 5.2.1), GHSA-724g-mxrg-4qvm (<= 5.2.0), GHSA-g796-fgmg-93mv (<= 5.1.0). Bumped by #387
scenarios/policy-enforcement/node/pnpm-lock.yaml 5.2.3 As above

Population: 12 pnpm lockfiles scanned, 3 resolve js-yaml, 0 at a vulnerable version. The other 9 do not reference it. gh api repos/.../dependabot/alerts returns zero open alerts across the repo; the js-yaml entry for node/mastra reads fixed. No dependency change was needed here, so this PR ships zero lockfile edits.

The gate's fail-then-pass evidence

A gate that has only ever passed is measuring nothing, so both defect classes were injected against the real tree and reverted.

Baseline — 36 manifest directories map to an entry, with zero left over., exit 0.

Defect A, an uncovered lockfile (node-extras/scratch-client/pnpm-lock.yaml, a directory no npm pattern reaches):

  npm              12/13 manifest directories covered
::error::/node-extras/scratch-client holds a npm manifest that matches zero 'npm' entries
exit 1

Reverted: exit 0.

Defect B, a mistyped directory ("/node/*" → "/nodes/*", one character):

  npm              6/12 manifest directories covered
::error::/node/custom-tool-policy holds a npm manifest that matches zero 'npm' entries
  ... (6 such lines)
::error::'/nodes/*' is configured for 'npm' but expands to zero manifest directories.
exit 1

Reverted: 36 manifest directories map to an entry, with zero left over., exit 0.

Both are also covered as unit tests, alongside fail-closed cases for an unparseable and an absent config — a coverage checker that returns 0 on a tree it failed to read converts an unmeasured state into a reported-healthy one.

The same pair, run through CI. A local exit code proves the script; it does not prove the workflow wiring. #440 was opened as a throwaway probe on a branch cut from this one, so this PR's history stays bisectable:

Commit dependabot coverage Job
18357d4 — uncovered lockfile added fail (exit 1) 94732302913
3a1595f — the revert pass 94732493446

The red run surfaced a GitHub error annotation naming the directory:

##[error]/node-extras/scratch-client holds a npm manifest that matches zero 'npm'
entries in .github/dependabot.yml. ... Add the directory to the 'npm' entry.

#440 is closed. Its branch is left in place for you to delete.

The parser was refactored after this proof was taken (see the Sonar section), so both defects were re-injected against the final code: exit 1 and exit 1, reverting to exit 0.

SonarCloud found two real defects in this PR's own code

Recorded because the first push turned the SonarCloud Code Analysis check red, and the cause was mine, not a pre-existing condition:

  • shell:S6506 × 3 — the curl -fsSL -O calls follow a redirect (-L) to objects.githubusercontent.com with no protocol constraint, for tarballs the script then executes. Fixed with --proto '=https' --proto-redir '=https' --tlsv1.2, verified against the real rc.6 asset URL.
  • python:S3776 — the config parser measured cognitive complexity 58 against a limit of 15. A parser whose correctness decides whether the gate measured anything should be reviewable in pieces, so it was split into isolate-block / split-entries / parse-entry with named helpers.

Quality gate is back to OK with new_security_rating 1.

Stale claims — population found vs corrected

Claim Measurement Sites found Corrected
"the release pipeline does not yet ship aa-api-server" (AAASM-4449) Published since v0.0.1-rc.4. components.json lists an api component for linux-amd64/arm64; tar tzf aasm-api-v0.0.1-rc.6-linux-amd64.tar.gz → aa-api-server 5 5
"continue-on-error: true" quarantines the lane It neutralises the run conclusion, not the per-job check runs. Run 31779321984 (main @ 9edecae): run conclusion success, three job conclusions failure 4 4
"ghcr.io/ai-agent-assembly/aa-gateway is not published" (.claude/CLAUDE.md, contradicting verify-live.yml's own header) Published, tagged v0.0.1-rc.6 and latest 1 1

Dependabot ecosystem coverage — the config was already correct

Worth recording, because the ticket's parent defect in agent-assembly was an uncovered directory and the same could have been true here. It is not: 0 uncovered, 0 dead patterns. npm 12/12, pip 18/18, gomod 5/5 — 35 dependency-manifest directories, plus the repository root for github-actions, which is the 36 the gate reports. .github/dependabot.yml needed no change — which is exactly why a gate is the deliverable rather than a config edit. Nothing held that property in place before this PR; a lockfile added tomorrow could have gone uncovered silently.

What this PR does not do

  • It does not make main green. After the brew fix, the live lane stays rc-gated on SDK/transport tickets outside this repo — AAASM-4447, AAASM-4467, AAASM-4468, AAASM-4469, AAASM-4446. Papering over that would violate this repo's Verification policy, so the lane is left failing at the condition it documents.
  • It does not touch branch protection. main carries 1 required review and zero required status checks, so this gate reports without blocking. That half is owner-gated and tracked on AAASM-5677.

Reviewer note — three views of this PR's head disagree, and that is the ticket

Dispatching verify-live against this branch stamped its three rc-gated job results onto this PR's head commit. The result, on 4d4d4d3, right now:

View Says
gh pr checks 439 Passed: 24, Failed: 0
gh api commits/4d4d4d3/check-runs 24 success, 3 failure, 1 skipped
The verify-live run's own conclusion success

Same commit, three answers. The three failures are live: node/go/python from run 31789721513 — my verification dispatch, not this PR's checks (verify-live has no pull_request trigger) and not a regression from this diff. Each check that this PR actually triggers is green.

This was not staged. It is the same disagreement AAASM-5675 describes on main, reproduced by accident on a PR head — and a reasonable argument that "is this branch red?" should not depend on which API a reviewer happens to call.

Related ticket

https://lightning-dust-mite.atlassian.net/browse/AAASM-5675

How to verify

# The gate, against the real tree
python scripts/check_dependabot_coverage.py     # -> exit 0
python -m unittest scripts.test_check_dependabot_coverage   # -> 17 tests OK

# Prove it can fail
mkdir -p node-extras/scratch-client && echo 'lockfileVersion: "9.0"' > node-extras/scratch-client/pnpm-lock.yaml
python scripts/check_dependabot_coverage.py     # -> exit 1, names the directory
rm -rf node-extras
python scripts/check_dependabot_coverage.py     # -> exit 0

The dependabot coverage check run on this PR exercises the same two commands in CI.

Checklist

  • PR title follows [AAASM-XXXX] <GitEmoji> (<scope>): <summary>
  • No secrets, API keys, or .env files committed
  • Example sub-projects include their own README.md with prerequisites and run instructions
  • SDK/runtime version dependencies are documented or pinned

@Chisanan232
✨ (scripts): Add Dependabot manifest-coverage checker
e51c7ff 
AAASM-5675 was filed after a Dependabot security job failed for weeks
against a directory .github/dependabot.yml did not list. Security updates
are advisory-driven and run against a manifest regardless of that file, so
an unlisted directory does not go quiet — it gets a job whose configuration
cannot satisfy it, and the vulnerable dependency stays put.

The checker asserts two things against the real tree: each discovered
manifest directory maps to an entry of its ecosystem, and each configured
directory pattern expands to at least one manifest. The second is the one
that hides — a mistyped path looks like coverage while delivering zero.

Stdlib only, matching scripts/generate_example_metadata.py, so a parse
failure cannot be caused by a missing third-party package.

Refs AAASM-5675
@Chisanan232
✅ (scripts): Add tests for the Dependabot coverage checker
8b86a81 
Each green case is paired with the specific edit that should turn it red:
an uncovered manifest, a mistyped directory pattern, a config missing its
updates block, and an absent config file.

A coverage checker that returns 0 on a tree it failed to parse is worse
than no checker — it converts an unmeasured state into a reported-healthy
one. The parser is therefore held to fail-closed behaviour by test, not by
inspection.

Refs AAASM-5675
@Chisanan232
🔧 (ci): Run the Dependabot coverage checker on PRs and main
7ad9f39 
Runs without a paths filter: the defect guarded against is a manifest
arriving somewhere unanticipated, and a paths filter would have to predict
those locations in advance — the same assumption that produced the gap. The
script is stdlib-only and finishes in under a second.

Also runs on pushes to main, because a pull-request-only gate leaves the
default branch free to drift red between merges, which is the second half
of what AAASM-5675 documented.

This job reports but does not block: main lists zero required status checks
today, and changing that is owner-gated under AAASM-5677.

Refs AAASM-5675
@Chisanan232
🐛 (ci): Install aasm from release tarballs instead of Homebrew
95e302e 
The verify-live jobs run on ubuntu-latest, where Homebrew is absent, so
`brew install ai-agent-assembly/tap/aasm` exited 127 with "brew: command
not found" before the gateway was reached. That is why the three live check
runs are red on main — not the rc-gated health-check timeout the script's
header described. A repo-local scripting bug was masking the condition the
lane exists to report.

The release pipeline publishes per-component Linux tarballs, so the tap is
not the only route: aasm-cli-<version>-linux-<arch> carries aasm and
aa-gateway, aasm-api-<version>-linux-<arch> carries aa-api-server. Both are
downloaded, checksum-verified against the release SHA256SUMS, and put on
PATH via GITHUB_PATH so later steps in the job see them.

The lane stays rc-gated on the SDK/transport tickets tracked outside this
repo (AAASM-4447/4467/4468/4469/4446); this change does not address those.
It makes the lane fail at the condition it documents rather than at a
missing package manager.

Refs AAASM-5675
@Chisanan232
📝 (ci): Correct stale rc-gate claims around the live lane
d8de20e 
Three claims repeated across the live-lane documentation were measured and
found inaccurate:

1. "the release pipeline does not yet ship aa-api-server (AAASM-4449)" —
   it has since v0.0.1-rc.4. components.json lists an `api` component for
   linux-amd64/arm64 and the tarball contains an aa-api-server binary.
   Corrected at 4 sites: verify-live.yml, .claude/CLAUDE.md, and the go-agent
   and node-agent scenario READMEs. A 5th site, start-aasm.sh, was corrected
   in the preceding commit.

2. "continue-on-error: true quarantines the lane" — it quarantines the
   workflow RUN, not the per-job check runs. Measured on run 31779321984
   (main @ 9edecae): run conclusion `success`, three job conclusions
   `failure`. That is why main presents as red while the scheduled run
   reports green. Documented as a known property, with an explicit note not
   to "fix" it by moving continue-on-error to the step level — that would
   hide the rc-gated state the Verification policy requires stay visible.

3. .claude/CLAUDE.md said the ghcr.io/ai-agent-assembly/aa-gateway image "is
   not published"; verify-live.yml's own header already said otherwise. The
   image is published, tagged v0.0.1-rc.6 and latest. CLAUDE.md now agrees
   with the workflow.

Refs AAASM-5675
@Chisanan232 Chisanan232 mentioned this pull request Aug 14, 2026
Closed
@Chisanan232
🔒 (ci): Pin the aasm download to HTTPS across redirects
ce20a42 
SonarCloud shell:S6506 flagged the three `curl -fsSL -O` calls: -L follows
the release download redirect to objects.githubusercontent.com, and without
a protocol constraint that chain could downgrade the transport for binaries
this script then executes.

--proto '=https' rejects a non-HTTPS URL outright and --proto-redir '=https'
holds it across the redirect chain. Verified against the real rc.6 asset URL.

Refs AAASM-5675
@Chisanan232
♻️ (scripts): Split the dependabot config parser into three stages
4d4d4d3 
SonarCloud python:S3776 measured the single-loop parser at cognitive
complexity 58 against a limit of 15. The parser's correctness is
load-bearing — a config it silently misreads makes the gate report coverage
it did not measure — so it should be reviewable in pieces.

Split into isolate-the-updates-block, cut-into-entry-chunks, and
read-one-chunk, with key splitting and entry validation as named helpers.
Behaviour is unchanged: the 17 tests pass, the real tree still reports 36/36,
and both injected defects still exit 1.

Refs AAASM-5675
@Chisanan232
🐛 (ci): Probe the port the gateway actually binds
04af83b 
The health check polled 7700 for the full 120s while the gateway was healthy on
7391 within ~200ms. 7700 is the default for the standalone aa-api-server binary,
which this lane downloads but never runs directly; `aasm start --mode local`
embeds the API on the CLI's own --port, default 7391.

So the first fix in this branch traded `brew: command not found` for a second
repo-local defect of the same class — and the script then printed "This is the
rc-gated failure described in verify-live.yml's header" into CI at exactly the
point where that was false.

A timeout says the probe did not succeed and nothing more. The script now prints
what it probed and points at the gateway output, instead of naming a cause it
has no way to determine. Found in independent review (AAASM-5675).
@sonarqubecloud

sonarqubecloud Bot commented Aug 14, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Chisanan232 Chisanan232

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Chisanan232