[kyle-sexton]

No related issue: a measured API-budget fix for the verdict reader, asked for in session; no issue was filed.

Summary

The pr-verify verdict reader proves "no FAIL tied to this PR" by paging the whole-repo actions/artifacts listing, about 135 pages, and accepting it only when its ids equal GitHub's live total_count. That check rarely settles under this repository's upload churn. Between 03:44 and 04:44Z on 2026-11-02, whole-listing pages made up 1,290 calls and drove a GITHUB_TOKEN breaker trip.

This PR makes every FAIL findable by exact-name lookups:

• Index artifacts. Before a FAIL uploads, its run uploads one index artifact per tie the gate counts: pr-verify-index-pr-<n>, -sha-<head>, -ref-<head-ref digest>, and -issue-<i> for each issue the FAIL's name carries.
• No FAIL without its index. The FAIL uploads only if every index uploaded. The index step first reads actions/runs/<run_id>/artifacts under the job's actions: read token. If the run already holds one of its index names from an earlier attempt, the step fails, so a re-run attempt writes no FAIL (verify:missing). That attempt then neither deletes the earlier pointer, which overwrite would, nor leaves a new FAIL that outlives a kept index.
• Re-runs of older jobs. A job without the index step writes no FAIL at all. That covers a re-run of a pr-verify.yml from before this change, which still runs main's write-verdict.mjs.
• Index mode in the reader. The reader then reads those names, one call each, plus the runs they name, plus the pass names at each FAILed head. Each read holds its own total_count.
• Legacy FAILs. FAILs uploaded before the index come from a committed legacy file, verify-legacy-fails.json, built once by the new legacy-fails subcommand.
• Turn-on condition. The reader enters index mode only when that file loads. Until a follow-up commits the file, every caller reads exactly as it does today.

The merge-gate intent is unchanged. A counted FAIL still refuses, and any read that fails refuses (verify:unreadable).

Fix

Measurements

Basis: one whole listing taken 2026-11-02 05:01:15-05:03:32Z. That was 135 pages, saved in the session's .work/measure/pages, and summarized with jq/python over that file.

1a. Why total_count runs above the listed ids

• Counts. The read returned 13,470 items and 13,462 unique ids, with 8 duplicate ids. total_count was 13,468 on the first page and 13,470 on the last. Over the read it ranged from 13,466 to 13,471.
• No expired artifacts.
  • 0 items had expired: true.
  • 0 items had an expires_at before the first page's Date.
  • So no expired-but-counted artifacts explain the gap.
• Uploads during paging. 27 artifacts were created in the 10 minutes before page 1. Uploads during the 137 s read push items down a page, which gives duplicates. Deletions shift items up, which drops them.
• The fresh top-up gap stays unexplained (UNVERIFIED). The earlier gap, where total_count stayed 1-8 above a top-up 88-110 s after a fresh listing, is not explained by this one read. The new design does not depend on it.

1b. FAIL-named artifacts

• Count and window. There are 541 FAIL-named artifacts, all unexpired, all at 90-day retention. They were created 2026-10-17T05:11Z to 2026-11-02T04:54Z and expire 2027-01-15 to 2027-01-31.
• Spread. They come from 541 distinct runs and 374 distinct PRs, with at most 6 per PR.
• Digest and grammar. 0 carry a head-ref digest, because the writer never wrote one. 0 are off-grammar.
• Issues in the name. 272 carry no issue, 264 carry one, 4 carry two and 1 carries three.

1c. Listing by name prefix: count, retention days min-max

Name prefix	Count	Retention (days)
pr-verify-*-pass-notes	3,009	6-89
pr-merge-dispatched	1,189	29-89
skill-reply-work-resolve-issue	808	6-7
pr-verify-*-fail	541	89
pr-verify-*-pass	512	6-89
mutation-range	502	6-7
pr-review-main-claude-review	443	89
pr-review-main-claude-security-review	442	89

About 50 more families hold 0-7 day artifacts. Workflow is inferred from each name's upload site, since the listing carries only workflow_run.id.

• Name filter. Live check: ?name=mutation-range&page=6 returned total_count 502 with 2 items on the page, across runs.
• Run artifacts. Live check: actions/runs/41390871184/artifacts returned total_count 1 and 1 item, carrying workflow_run.id.

Design

Legacy FAILs need a closed legacy set, not a self-ending fallback.

• Why not the fallback. A whole-listing fallback that turns itself off when the last pre-index FAIL expires would keep whole listings on the hot path until about 2027-02. That is 90 days after merge (Basis: 1b expiry). So it doesn't meet the goal.
• What the legacy file is. The set of FAILs from before the index is closed: no new FAIL joins it once this is on main and old runs have drained. Its file needs no live-count match at read time. Each entry is listing-shaped and records:
  • its run's verified origin;
  • its verdict's head branch, read once at build time.
• What legacy-fails does.
  • Cutover. It takes the cutover from --merge-sha, the merge commit that brought FAIL_INDEX to main. That commit must be on main, and its pr-verify.yml must set FAIL_INDEX: "1". The cutover is its committer date plus a 1 h margin. An explicit --cutover earlier than that refuses; a later one only reads more runs.
  • Runs. It lists every pr-verify.yml run created from --since to the cutover, the cutover's own second included, in one-day created= windows. A window is split while its count reaches GitHub's 1,000-result cap. Each window's listed run ids must equal its total_count; a past window is stable, since runs are not deleted inside retention.
  • Per run. For each run it reads actions/runs/<id>/artifacts, one page whose ids must equal its total_count.
  • Re-run lookback. Before those windows, it lists runs created in [since - 31 days, since) with the same windowed count check, and reads each one whose updated_at is at or after --since. GitHub allows a re-run up to 30 days after the initial run (Re-running workflows and jobs, read 2026-11-02), and a re-run keeps its created_at. 31 days covers that limit with a day to spare. --rerun-lookback-days sets it.
  • In-flight runs. Both before and after the listing read, it reads each running status (queued, in_progress, waiting, requested, pending) in one page that must hold its total_count, and refuses on any verifier run created at or before the cutover.
  • Listing. It then reads one whole artifact listing, through the same pacer, as a second check. It applies no count check, because the listing only adds runs. A listed FAIL whose run no window holds is read the same way and added: a re-run keeps its created_at, and a three-job chain can upload a FAIL hours after it. So the file is the union of the windowed runs and the listing.
  • Refusal on a missing FAIL. If a listed FAIL is still held by no run read, the build refuses and lists the run ids. A run that uploaded an index, or is of another origin, is exempt.
  • It keeps every unexpired FAIL-shaped artifact of a run of verifier origin that uploaded no index, with its verdict's head branch.
  • It refuses (exit 1, nothing written) when any run is not completed, any read fails or miscounts, an origin or head branch cannot be read, or a listed FAIL stays missing. A permission 403 refuses too.
  • It paces every call (default 0.5 a second), stops at 1,800 calls per owner-token reset window, and stops on the first rate-limit or secondary-limit answer: rate-limit wording, x-ratelimit-remaining: 0, 429, or retry-after. It also stops when the breaker file exists or core remaining is under 2,000, checked by a rate_limit read at start and in each window. A stop exits 3 and resumes from the checkpoint file. Every finished window and run is kept there. It prints one legacy-fails: window reset=... calls=... total=... line per window. All five limits are flags.

Tie coverage. Each rule in the module docstring, and what finds it in index mode:

Tie in the FAIL rules	Index mode reads
This PR's number, any head (verify:fail, verify:earlier-fail, the cap)	pr-verify-index-pr-<n>
A FAIL at the head under any number	pr-verify-index-sha-<head>
A FAIL at one of the PR's commits (pulls/<n>/commits)	pr-verify-index-sha-<c> per commit; past INDEX_NAMES_CAP (100) names, the whole listing
An untied FAIL recording this head branch	pr-verify-index-ref-<digest>, uploaded by the FAIL's own verifier run from verdict.head_ref
The cap: FAILs of PRs that close one of the live issues	pr-verify-index-pr-<p> for each closer from the GraphQL already read
The cap: FAILs whose name carries a live issue (Refs: PRs)	pr-verify-index-issue-<i>
A FAIL whose run cannot be read (unreadable, sibling unreadable)	Same index names; the index is found whatever the FAIL run's state, and the existing rules apply
Passes at the head; passes superseding a FAIL at another head	Exact names pr-verify-<n>-<sha>-pass and -pass-notes
Off-grammar FAIL names	The writer cannot produce one (0 exist); any FAIL-shaped name in a pointed run or in the legacy file goes through the existing rule
Legacy untied, no-digest FAILs	The legacy file, with each head_ref and origin recorded

Two deliberate narrowings, both fail-closed or justified.

1. An unreadable unrelated record no longer blocks every PR.
   • Before: an unrelated new FAIL whose record cannot be read was verify:unreadable for every PR.
   • Now: it does not block. Its head branch travels in an index name its own verifier run uploads, so its tie is known without a download.
   • Legacy FAILs keep their recorded head_ref. The build refuses when one cannot be read.
2. Pass names with the digest are not read. No writer writes one (write-verdict.mjs artifactName), and resolve-pr reads only the plain names. A pass missed this way leaves verify:missing.

Pointer safety.

• An index only points. Every FAIL it leads to is still judged by its own run's origin, so a forged index can only add FAILs.
• An index whose run holds no FAIL:
  • While the run has not completed, it refuses. The FAIL may still be uploading.
  • Once the run has completed, it is ignored. That FAIL upload failed.
• Bounded reads.
  • More than INDEX_RUNS_CAP (120) runs refuses.
  • A name past 7 pages refuses.
  • Basis: a whole listing was about 135 pages.

Consumers moved. These all read in index mode:

• _evidence
• _own_fails / fail_findings
• _answer_key (Reader.targeted, handed to the next judgment of the same head)
• babysit_merge_skip.py's key (Reader.own_verdicts)

_late_pass and refresh() need no listing there. Each judgment reads fresh, and refresh() drops a kept hand-off.

Retention (step 3): no change.

Family	Why not cut
pr-merge-dispatched	Already 30 days. The 89-day ones predate that cut and age out (.github/artifact-retention.json).
pr-review-main-*	Its row records a pending-cut prerequisite: below-CRITICAL totals must first be read from check runs.
pr-merge-rerun-*	Must stay at or above dispatched. Cutting it saves 42 artifacts.
branch-cleanup-ledger-*	The only restore record.

Verdict artifacts are untouched. The new pr-verify-index-* row keeps the FAIL's 90 days, so an index never expires before its FAIL.

Per-run call delta per caller

Before the follow-up commits verify-legacy-fails.json, every caller's delta is 0. The reader stays on the listing path.

The listing grows meanwhile:

• At 3.5 index artifacts per FAIL, from 1b's issue counts, that is about 1,900 per 541 FAILs.
• That is about 119 a day, or about 10.7k at 90 days, against 13,470 today.
• That is up to about +80% pages on every whole-listing read.

So the follow-up should land as soon as old runs drain.

After it lands, per judgment:

Reads	Count
pr	1
Closers	1 each
Commits (head included)	1 each
ref	1
Live issues	1 each
Each FAIL run named	1
Passes at the head	2
Each FAILed head	2 more

That comes to about 8-11 calls for a PR with no FAIL and about 15-20 with two FAILed heads. These are estimates (judgment on the formula), to be re-measured from the new babysit-index-reads stderr line and the existing babysit-gh-calls route counts.

Caller	Measured 03:44-04:44Z	Estimated in index mode
ci-status merge_group, job 125090915891	437 calls, 352 whole pages, verify:unreadable	about 95-105
ci-status merge_group, job 125092070741	179 calls, 134 whole pages, verify:unreadable	about 45-65
pr-merge merge job 125086673975, babysit_merge_skip.py (5 candidates)	478 calls, 402 whole pages; the step took 5.5 min, then "3 whole listings could not be checked"	about 96-116 (5 × own_verdicts at 4-8)
pr-merge merge job 125086673975, babysit_merge.py (5 candidates; budget ran out after 3)	577 calls, 402 whole pages	about 225-275
enable-auto-merge (5 runs)	53 / 92 / 128 / 0 / 0, 0 whole pages (saved listing)	+10-20 per judged PR; it no longer shares one listing

Breakeven. One whole listing costs about 135 calls, so index mode costs more only when one process judges more than about 11 PRs. Past INDEX_NAMES_CAP names, one judgment falls back to a listing.

Follow-up (not in this PR)

1. Wait for old runs to drain. After merge, wait until every pr-verify run created before the merge time has completed. The build refuses any run that has not completed.
2. Build the legacy file from runs. Use an Actions-read owner token, and, to avoid up to about 541 verdict downloads, set BABYSIT_VERIFY_HEAD_REFS to a head-ref map from a scheduled pr-merge run. Run it once, and run it again with the same arguments after any stop (exit 3):
   python3 plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py legacy-fails --repo melodic-software/claude-code-plugins --since 2026-10-17T00:00:00Z --merge-sha <this PR's squash commit on main> --checkpoint <file> --out plugins/source-control/skills/babysit-prs/scripts/verify-legacy-fails.json --breaker-file <hub breaker.txt>
--since comes from the 05:01Z listing: its oldest unexpired FAIL was created 2026-10-17T05:11Z. A run created earlier with a FAIL still unexpired is caught by the build's listing union, not by --since.
3. Commit the file. ci-status's sparse checkout already names it, and the babysit test package pins no legacy file, so the suites keep faking the listing.

Expected calls (estimate, judgment on the measured counts): 2,500 runs created since 2026-10-17 as of 05:5xZ, plus about 150 a day up to the cutover. The total is about 2,550-2,800 run and window reads, one compare per distinct FAIL run head (at most about 550), up to about 550 verdict downloads without a head-ref map (few with one), one rate_limit read per window, 3 reads for the merge commit, 15 running-status reads (5 at each of three checks), about 31-62 re-run lookback window reads plus 2 per lookback run updated since --since, and one whole listing at about 135-250 pages (more as index artifacts accumulate), plus 2 per run the listing adds. That makes about 2,800-4,250 calls in all: 2-3 reset windows at the 1,800 cap, about 1.6-2.4 h of pacing at 0.5 a second.

Verification

• node --test passed:
  • .github/scripts/pr-verify/write-verdict.test.mjs and workflow.test.mjs: 71 pass.
  • reconcile, verify-notes, workflow-gates and resolve-pr tests: 369 pass.
• pytest over plugins/source-control/skills/babysit-prs/scripts/tests: 3,973 passed, 6,523 subtests.
• New tests:
  • test_babysit_verify_index.py: index-mode allow with no listing read; FAIL at the head under another number; earlier head; head-branch tie; issue-named cap; legacy FAIL on recorded origin; expired legacy; still-uploading run; failed or short name read; the names cap falling back to the listing; the runs cap; own_verdicts; an expired legacy FAIL left out of fail_findings.
  • Per-run legacy-fails: the built file; a window or a run's artifacts that miss their count refuse with nothing written; a run not completed refuses; a rate-limit answer stops with exit 3 and the checkpoint resumes without re-reading; a rate limit inside a verdict download still stops; the window cap; the breaker and low-remaining stops before any read; pacing at 0.5 a second; a checkpoint for another range refuses; the last window reads the cutover's own second; a FAIL of a run created before --since is added through the listing; a listed FAIL that no run read holds refuses, naming the run; a cutover before the merge commit refuses; a merge commit without FAIL_INDEX refuses; a permission 403 refuses with exit 1 rather than stopping; a run created before --since and updated after it is read through the lookback windows, and one not updated since is not; a lookback window that misses its count refuses; a pre-cutover verifier run still in progress refuses, and a later one does not; a run unfinished at the first build is listed again on resume; a resume inside the same reset window keeps counting toward its cap; in index mode, a FAIL uploaded while its run completes still counts.
  • upload-fail-index's name split, run with a stand-in gh: it fails on an index name the run already holds, uploads nothing on a failed or miscounted read, and sets no overwrite.
  • The answer key's index branch.
  • The write-verdict index names and the FAIL_INDEX refusal.
• Other checks passed:
  • scripts/check-artifact-retention.sh and its test: 32 pass.
  • scripts/ci-lanes-pr-hold.test.sh: 300 pass.
  • zizmor --no-online-audits on pr-verify.yml, pr-require-checks.yml and the new action: no findings.
  • scripts/run-ruff.sh check: clean.
• Not run, and why:
  • actionlint fails only on the existing cache-mode job key, which its local version does not know.
  • The composite action has not run on GitHub (UNVERIFIED until the first FAIL after merge).

Related

• fix(source-control): re-read a short verdict listing and title verifier holds #12233 (merged 76addbc): the short incremental re-read this replaces on the hot path. Its body records why per-run reads were rejected: pr-verify runs carry main's head_sha, and the cross-PR FAIL rules need every pr-verify artifact. The index answers both, because each tie gets its own exact name.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T06:21:36.372252Z	9babebd	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9babebd465

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-security-review reviewed 44b849d13290b763af31bf549dc8733f6c9db77b: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12302, incremental (696ba56 → 44b849d)

I read every hunk of the 4 in-scope files in .claude-lane/incremental.diff, using the full diff, the PR context and the head files for context. I found no security issues in this change.

critical-findings: 0

Prior finding

My earlier IMPORTANT finding (discussion_r4439678973) is addressed. A re-run's FAIL could outlive an index that had been skipped. Now, when this run already holds an index name, the step fails (action.yml#L78-L81). A FAIL upload in all three jobs requires steps.fail-index.outcome == 'success' (pr-verify.yml#L1890-L1897, L3712-L3719, L5268-L5275). So a re-run attempt writes no FAIL that could outlive its index. That attempt leaves no verdict, and the gate reads that as missing, so it fails closed.

Paths checked (no finding)

upload-fail-index, re-run of a FAIL whose first upload failed

• Suppose attempt 1 uploaded its index, but its FAIL upload was skipped or failed. Attempt 2 now records no FAIL either.
• That FAIL was already lost when attempt 1's upload failed, so this change does not create the gap. It is also a missed verification count, not a leak, a compromise or a charge.
• Only the owner's actors can trigger re-runs. I am noting this and not counting it as a finding.

_run_artifacts second read (babysit_verify_evidence.py, merge-hold trusted path)

• This change narrows a fail-open race. Before, a FAIL that uploaded between the first artifact read and the completed-status read was ignored. Now, once the run is seen completed, the artifacts are read again, and that read decides.
• The second read goes through the same checks: the total_count match, the entry shape and the workflow_run.id match. It raises _Unreadable on any mismatch.
• A completed run cannot gain artifacts unless it is re-run. A re-run attempt cannot write a FAIL past a name it already holds (the action change above).

_window completed-status refusal (legacy-fails build)

• Any run listed in a window that has not completed now raises _Unreadable before the window is checkpointed. A resume therefore lists that window again, and a checkpoint can no longer keep a window with a run that was still uploading.
• This only adds refusals (fail-closed).

_Pacer window state file

• The window's reset time and call count are saved to <--checkpoint>.window. --checkpoint is required (L6005).
• A saved count is reused only when its reset matches and the count is an int. An unreadable file counts as empty, and the window cap check runs at open.
• This is an operator-run tool, and its output is committed only through a reviewed PR. The change only tightens rate limiting and grants nothing.

Tests

• workflow.test.mjs and test_babysit_verify_index.py change only fixtures and cases.

Spend (not counted as a finding)

The verdict reader makes one extra prepaid GitHub API read per index run that holds no FAIL. The legacy-fails build re-reads a window when it resumes after an in-progress refusal.

[github-actions]

claude-review: 3 inline finding(s) on 9babebd46520b749e2748705fd25eed283043a94; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 44b849d13290b763af31bf549dc8733f6c9db77b: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, 696ba56e → 44b849d1)

No findings in this push. I read every hunk of the 4 changed files in .claude-lane/incremental.diff. For context I also read the current _Pacer, _window and _run in babysit_verify_evidence.py, the upload-fail-index call sites with their scan and verdict-upload gates in pr-verify.yml, and the RunsGitHub and index fakes in the tests.

The round cap applied (past the cap: yes, cap 3), so this review would have posted only blocking findings. It found none, blocking or advisory.

Prior findings

• Fixed: a not-completed run stuck in the checkpoint (r4439616621, r4439455915).
  • _window now refuses before it returns when any listed run's status is not completed (L5580-L5584). The caller's save() never sees that window, so a resume lists it again.
  • The refusal also covers the halves of a split window, since each recursive call checks its own listing.
  • The new test leaves fake.running empty, so _running_check passes and the refusal it hits comes from _window itself. Removing the change would fail it.
• Fixed: the per-window cap reset on every resume (r4439455923).
  • _Pacer now keeps {reset, calls} in <checkpoint>.window and writes it after every call.
  • open_window restores the count only when the saved reset equals the live core reset, which stays fixed within one GitHub window, and stops at once if the count is already past the cap. --checkpoint is required, so the state path always exists.
  • A rollover inside one process reads a stale reset and starts again from 0, which is correct.
  • The test checks that a resume at the cap makes only the rate_limit read.

Checked in this push, and it holds up

• _run_artifacts reads again after completion.
  • A run with an index and no FAIL-shaped name is now read a second time, after its status shows completed. A FAIL uploaded between the first read and that status check is therefore counted, not taken for a failed upload.
  • Both reads go through the same total_count, _listed_entry and workflow_run.id checks.
  • The new test feeds the first read only the index, and the default second read holds the FAIL.
• upload-fail-index fails on a held name instead of skipping it.
  • A re-run attempt that judges FAIL now fails the index step. The scan and verdict upload are both gated on steps.fail-index.outcome == 'success', so that attempt writes no new FAIL, and the earlier attempt's pointer is never deleted or left pointing at a newer FAIL.
  • A FAIL verdict's name is the same across attempts of one run, so the verdict upload would have hit a name conflict anyway. That makes the stricter step a no-op for the normal case.
  • Trade-off: if attempt 1 uploaded its index but not its FAIL (for example the credential scan was not clean), a re-run of that run can no longer write the FAIL. The head then needs a fresh run. This fails closed: the reader treats a completed run with an index and no FAIL as holding none, and that head gets no verdict from that run.
  • The workflow.test.mjs case covers all three paths: a held index name fails and emits no slots, another held name (the verdict's own) uploads every index, and a fresh run uploads every index.

Verification is by reading only. I ran no tests (this lane has no shell), and any test results in the PR are the author's claim and unverified.

[github-actions]

claude-review: 2 inline finding(s) on 8342d7565f6c176a904afb9f3210c0d1f8a92bf9; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-security-review: 1 inline finding(s) on 696ba56ec612791d7387ccedf2fd16870d6cf7f5; the summary is in this lane's pull request comment.

claude-lane: claude-security-review