Skip to content

perf(pr-pipeline): read pr-verify verdicts by exact name in pr-unblock and pr-feedback-dispatch #13004

Open

Description

@kyle-sexton

Refs: #12932

Problem

In the REST window 2026-11-03T22:52:41Z to 23:52:41Z, the dispatch job of pr-feedback-dispatch.yml was the top caller. Most of its calls come from the pr-verify FAIL-cap read in front of a dispatch, and nearly all of that read is spent re-listing every artifact page. It does this because the carried artifact listing is missing.

Baseline (measured from each job's api-routes annotation):

  • 33 non-skipped dispatch jobs, 523 REST calls in total.
  • 10 heavy jobs (9 pr-unblock dispatches and 1 sweep) spent 455 of them (87%), 36 to 71 each. The 23 light jobs spent 2 to 7 each.
  • Top routes: actions/runs/{n}/artifacts 108 (.github/scripts/pr-merge/rotation.mjs:154 newestTrustedRun, which checks up to 10 runs, reached through listing-carry.mjs from verify-cap.mjs:138-153); actions/artifacts 73 (babysit_verify_evidence.py:2148 and :2178, every page); contents/* 51; actions/workflows/{x}/runs 46; graphql 45; git/ref/* 40.
  • In every heavy job, the carry costs about 15 calls (listing-carry 11, carry-run-cache 3, head-ref-map 1), finds nothing, and logs no earlier artifact listing; the verdict reader lists every page. The reader then reads 6 to 7 full actions/artifacts pages.
  • The job and run id pairs are kept on the host that ran the measurement, at .work/pr-verify-cut/dispatch-jobs.txt. That directory is not committed.

Evidence that the listing is missing: GET actions/artifacts?name=pr-merge-verify-listing returns total_count 9, and the newest is from 2026-11-03T03:22:28Z (run 41571354158). Nothing has been uploaded since. The upload step .github/workflows/pr-merge.yml:2146-2153 runs only on schedule and workflow_dispatch, and only when the listing read whole. It uses if-no-files-found: ignore with retention-days: 1, so a partial read uploads nothing and logs nothing. Once the last copies expire, every FAIL-cap read lists all artifacts.

#12932 is not this path: its commit 431f923 changes only the rerun-ci-status job and scripts/ci-rerun-ci-status.test.sh.

Done when

  1. Move the verdict readers to exact-name index reads. Likely cause, UNVERIFIED until a dispatch log confirms it: feat(source-control): commit the legacy FAILs file to turn on FAIL index mode #12558 (FAIL index mode) merged 2026-11-03T03:28:44Z, six minutes after the last listing. In index mode the merge job reads verdicts by exact name and never writes verify-listing.json, so the upload has nothing to upload. Move the pr-unblock and pr-feedback-dispatch verdict readers (verify-cap.mjs -> rotation.mjs:154, babysit_verify_evidence.py:2148 and :2178) to the same exact-name index reads, so they no longer depend on a carried listing.
  2. Only if a listing reader survives item 1: have it find the carried artifact with one actions/artifacts?name=pr-merge-verify-listing call, checked against the trusted run list it already reads, instead of checking up to 10 runs. Also log a notice when a listing is partial or missing, instead of skipping the upload silently (pr-merge.yml:2146-2153).
  3. Short-circuit wasted reads. 6 of the 10 heavy reads ended refused verify:missing (), not its FAIL cap; the round runs under its own caps, spending about 30 calls each without changing the dispatch decision. Detect that outcome before the full read.
  4. Re-measure. Run the same count over a 1-hour window after the change and compare it with the baseline above: calls per heavy job and the total for dispatch.

Notes

  • Owner lane: poteto (pr-merge area). pr-merge.yml is a workflow file, so the change is pushed from a host session, not a lane.
  • The hub counted 488 calls for this job and this measurement counted 523. The gap is likely core calls split across two reset times (one annotation reads core-by-reset=…:19,…:18, i.e. 23:49:34Z and 23:52:41Z). UNVERIFIED.

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 4, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 4, 2026
  3. github-actions commented on Nov 4, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    I understand this as a request to cut REST calls in the pr-unblock and pr-feedback-dispatch verdict reads. Those reads would use exact-name index reads, so they no longer depend on the carried pr-merge-verify-listing artifact, which has not been uploaded since 2026-11-03. It also asks for a short-circuit on reads that end verify:missing, and for a re-measurement over a 1-hour window.

    The suspected cause (#12558 index mode) is marked unverified, so a dispatch log should confirm it before the change. The re-measurement and the pr-merge.yml push need a host session, so admission routes them as host work. I found no likely duplicate among the open issues.

    Automated triage, applied by the lanes App.

  4. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 4, 2026
  5. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 4, 2026
  6. added
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Nov 4, 2026
  7. melodic-automation-lanes commented on Nov 4, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited; its body or a criterion names .github/workflows/pr-feedback-dispatch.yml, .github/workflows/pr-merge.yml, which the issue lane cannot push (patch-protected-path); a builder session takes it; it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  8. melodic-automation-lanes commented on Nov 4, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

  9. added
    status: needs-decisionAwaiting a human or maintainer judgment call.
    and removed
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    on Nov 4, 2026
  10. melodic-automation-lanes commented on Nov 4, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Make the pr-unblock and pr-feedback-dispatch FAIL-cap reads use exact-name index reads instead of the carried listing, add a pre-read short-circuit for verify:missing, and log partial or missing listings.

    Related work

    Open questions

    1. The cause is marked UNVERIFIED (#​12558 index mode stopped the listing upload). Must a dispatch log confirm it before the change, and who supplies it?
      RECOMMENDED: Yes. The owner or a host session attaches one heavy dispatch log showing the 'no earlier artifact listing' line, and the work waits for it. Basis: judgment
    2. Index mode turns on only when the legacy FAILs file loads (index_mode, _legacy). Should the dispatch readers get exact-name reads by passing that file in, or by changing the gating?
      RECOMMENDED: Pass the legacy file path in from verify-cap.mjs and the two dispatch workflows, and keep the gating as is. Basis: .base/plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:2704
    3. Item 2 applies only if a listing reader survives item 1. Is it in scope now or deferred?
      RECOMMENDED: Defer it unless a listing reader remains after item 1. If one does, add the partial-or-missing notice at pr-merge.yml:2143-2152 in this issue. Basis: .base/.github/workflows/pr-merge.yml:2143
    4. Item 3 gives no rule for detecting 'refused verify:missing' before the full read. What cheap check decides it?
      RECOMMENDED: Make one exact-name index read for the PR head SHA and skip the full read when no verdict exists, with a test showing the dispatch decision is unchanged. Basis: judgment
    5. pr-merge.yml is pushed from a host session and item 4 needs a live 1-hour measurement. Should those be split out as host checks?
      RECOMMENDED: Yes. Keep the code and test changes as the unattended part, and make the workflow push and the re-measure host checks. Basis: judgment

    Sources read

    • /home/runner/work/claude-code-plugins/claude-code-plugins/refine-data/context.json: the issue body, triage comment and candidate list
    • plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:2704: index_mode is true only once the legacy FAILs file loads
    • plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:147: FAIL_INDEX unset means an older pr-verify.yml, which writes the older listing form
    • plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:2840: _index_hits reads verdicts by exact name
    • plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:5492: finds when FAIL_INDEX reached main
    • /.github/scripts/pr-unblock/verify-cap.mjs:122: runs listing-carry.mjs first and reads the carried listing
    • /.github/workflows/pr-merge.yml:2143: the listing upload runs only on schedule or workflow_dispatch, retention 1 day, if-no-files-found ignore

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  11. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Related measurement gap. The same full listing also blocks per-PR metrics.

    • What happened: the pr-pipeline: tidy a lane's diff inside resolve-issue before its one push #8485 step 1 baseline could not count re-verifications per PR. GET actions/artifacts returned HTTP 500 on 7 of 9 calls between 02:40Z and 02:50Z on 2026-11-04, and one answer came back with an empty page 2 while reporting total_count 18,023.
    • Why the artifacts are the only source: pr-verify runs start from workflow_run on main with an empty pull_requests list, so only the pr-verify-<pr>-<sha>-<verdict> artifact names tie a run to its PR.
    • Proposed fix: a run-name: in pr-verify.yml that carries github.event.workflow_run.pull_requests[0].number and the head SHA. That expression is already used in the concurrency group at pr-verify.yml:86. Each run's display_title would then name its PR, and actions/workflows/387380935/runs?created=<range> could count runs per PR without touching the artifact listing.
    • Open caveat: the listing also returns the many skipped runs, 1,617 in one 2-day window. A reader filters those on conclusion, and whether that page cost is acceptable is UNVERIFIED.

    Posted by host-run Lane 5. This was generated by AI.

  12. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    One more full listing on a gate path: the cold merge-queue pre-check. Not covered by this issue's items so far.

    • Where: .github/workflows/pr-require-checks.yml:6529-6538, on main at the fetch of 2026-11-04T04:40Z. With no carried run cache, ci-status pages actions/artifacts?per_page=100 with --paginate only to compute cold_cost.
    • Cost: about 184 pages at today's 18,253 artifacts (listing_pages=$(((total + 99) / 100 + 1)), :6537), per cold entry.
    • Failure: any read error exits 1 with "the artifact listing could not be read ... Re-queue it." The unfiltered endpoint answered HTTP 500 on 7 of 9 calls at 02:40-02:50Z and 3 of 3 at 03:12Z, and returned empty bodies at per_page=100 at 04:35Z, while per_page=1 answered 200 at 04:40Z. Name-filtered reads kept working. So every cold entry is held while the listing is down.
    • Stale price: in index mode the verdict reader no longer lists (babysit_verify_evidence.py Reader._targeted), so the formula's listing_pages and 3 * fails terms describe a read that no longer happens.
    • Fix: drop the pre-check or re-price it from the index read. Either way, do not page the unfiltered listing.

    Whoever builds this issue can take it in the same change. Lane 5 is not building it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: needs-decisionAwaiting a human or maintainer judgment call.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    Work status

    needs-owner-decision

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions