Repository navigation
Description
Refs: #7531
Problem
The 2026-10-26 record on #7531 split #7531. The data retro stays there, gated on live merges. The static friction audit and the 2026-10-18 gaps checklist move to a child issue, because they need no live merge data. That record's first criterion is this child.
The 2026-10-26 amendment changed the deliverable. The findings are posted as a comment on this issue, plus follow-up issues, and any lasting conclusion goes into docs/conventions/pr-pipeline/README.md. No report file is committed (AGENTS.md: one-off investigations stay in the scratchpad, and only the conclusion is committed). The revert-triggered stop is a criterion on #7528's record, so the audit cites #7528 for it.
Routing: host-run. Posting findings as a comment and filing follow-up issues are tracker writes, and the issue lane can only read the tracker. Both #7531 lane runs say so (6536975267, 6537727272).
Scope, from #7531's body: CI/CD performance, the autonomous Actions lanes, and the merge path; find the friction points.
The 2026-10-18 gaps checklist (comment), from that comment:
- No owner for post-merge bookkeeping (
track done, container close-out). - Triage runs in two places (CI intake lane and local
/work-items:triage) with no rule for which decides. - Merge authority moves from the local babysit loop to
pr-mergein one switch, and the singlemerge.authorityfield with rung lowering after a bad merge (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202). Phase 9 removes the rung ladder (Q505), so check whether this is still a gap. - Post-merge failure response: revert or fix forward. The revert stop is decided; fix-forward is not.
- Post-merge smoke check against the deployed environment (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202).
- Stages outside the plan: build/package; monitor after deploy with rollback; incident response; postmortem; migrations before and after deploy. Define "release" (version cut) and add
rolloutif feature flags come into scope.
That comment's last item, the candidates to evaluate on #7531's data, stays with #7531's data retro.
Folded in from #7612 (comment): its items 2, 3, 5 and 6, the label-normalization restrictions on admission.
Done when
- A findings comment on this issue covers CI/CD performance, the autonomous Actions lanes and the merge path, and names each friction point with the
file:lineor run that shows it. - Each of the six gaps-checklist items above, and each of pr-pipeline: harden issue-lane admission after the ratification marker #7612's items 2, 3, 5 and 6, is answered in that comment as keep, drop, or a follow-up issue number.
- Every follow-up the findings recommend is a filed issue, cited by number.
- Any lasting conclusion is written into
docs/conventions/pr-pipeline/README.mdthrough a PR. No report file is committed, and no lane behavior changes in this issue. - The revert-triggered stop is cited as pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528's criterion, not re-tracked here.
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Large
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I understood this as a static audit of the unattended merge flow. It covers CI/CD performance, the autonomous Actions lanes and the merge path. It also answers the six 2026-10-18 gaps-checklist items and #7612's items 2, 3, 5 and 6. The deliverable is a findings comment, follow-up issues for each recommendation, and any lasting conclusion written into the pr-pipeline README through a PR. The done-when is exact, and the tracker writes are already routed to host-run. The audit spans several subsystems, so it may need splitting. I found no likely duplicates among the open issues.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: it carries host-run; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 2, 2026
Claim. Backlog HITL host-run worker is taking this now (read-only static friction audit; findings comment follows).
kyle-sexton commentedon Nov 2, 2026
Host run. A read-only static friction audit of the unattended path from a ready PR to merge, with run costs measured from the hub's local runs cache and a sampled read of failing
ci-statussteps.What ran
origin/maincb8429f3cf (2026-11-02T15:30Z). Everyfile:linebelow is at that commit.GET repos/.../rules/branches/main.updated_at - run_started_at) over non-skipped runs. On this public repository hosted runners do not bill, so run-h measures runner contention and, for the model lanes, subscription usage. It is not a charge (costs).pr-require-checksruns on 11-01, read throughactions/runs/{id}/jobs:pull_requestruns, of which 24 were readable (one returned HTTP 502).merge_groupruns.ci-statusannotations.Where the time goes (measured)
unlabeled)About 336 PRs merged through the queue on 11-01 (successful
merge_groupruns). On that day the merge path after a PR is ready used roughly 300 run-h. The lifecycle baseline (2026-11-01, quoted) puts the PR-side wait at ready -> first queue entry p50 0.80 h / p90 7.41 h over 7d, and queue entry -> merged at p50 0.11 h. The wait before the queue is the binding stage.Why ci-status is red (sample)
Of 24 failed
pull_requestruns:do-not-mergelabel (intended cross-lane hold)failurestatus on the headneeds-humanlabelA red ci-status from
do-not-mergeorneeds-humanis the hold doing its job, not a defect. The friction rows are the others:Gate by gate
Owner direction s10: a hold against our own actors is friction. The only HIGH/CRITICAL boundary is a token or secret leak, an account compromise, or billing. No verdict below loosens one of those.
ci-status, the one required check (rulesetrequired_status_checks,strict_required_status_checks_policy: false)pr-require-checks.yml:4003; ruleset readdo-not-mergelabelpr-require-checks.yml:4795;loop-lane/README.md:460needs-humanlabel holdsdo-not-merge(#17)pr-require-checks.yml:5445;pr-pipeline/README.md:2804pr-require-checks.yml:5551;pr-pipeline/README.md:2806pr-require-checks.yml:5122human-merge-holdlabel, lifted only by a trusted person acting directlydo-not-merge. It is an own-actor hold.pr-require-checks.yml:5686;pr-pipeline/README.md:2812ci-statusdo-not-merge, and ci-status reads the label only. That removes the verdict reader from ci-status.pr-require-checks.yml:6247, comment at:4029-4051failurecommit status holds the headdo-not-merge(or nothing, when ci-lanes is already red) instead of a separate status context. That drops one hold mechanism and its re-run dependency.pr-require-checks.yml:6186;pr-pipeline/README.md:2863-2869;maintenance-reconcile-lost-runs.yml:93rerun-ci-statusafter each review completescosthold (pr-review-hosted.yml:242-250), so it is a billing control. Simplify the ordering so ci-status does not start red on "review pending". Options: start the review lanes so they gate pr-verify instead of ci-status, or fold the review check into pr-merge's verdict. Either removes the re-run loop.pr-require-checks.yml:6755;pr-feedback-dispatch.yml:575;pr-pipeline/README.md:2872pr-verify-failed, pr-unblock status, lanes body hold)select-testsstep (:958) already fails a held PR "before the lanes", so make it the only queue-time hold read and drop the duplicates inci-status(:5779,:6186,:6247). Better still, dequeue the PR onlabeledinstead of waiting for the group run.pr-require-checks.yml:958,:5779pr-review-main.yml:3,:59workflow_run), about 1.8 per headpr-verify.yml:67-71unlabeledevent and pushunlabeled: 1,011 runs and 206 run-h, mostly waiting on the shared concurrency group (2026-11-01 sample, quoted: "26 s after 21 min queued"). The file says it "does not wake on each CI or review completion"unlabeledas a wake once that event exists, and slow the cron to a backstop. This is the largest PR-side wait.pr-merge.yml:85,:93,:98check_response_timeout_minutes30) re-runs full CI per groupstrictis false)required_review_thread_resolution: truepull_requestrules, one withdismiss_stale_reviews_on_pushand 0 required approvals; signatures; linear historypull_requestrule once someone confirms which ruleset owns the Dependabot approval path (UNVERIFIED which).needs-humanused as a general park by lanes (pr-unblock escalation routes)feedback-disputed/feedback-uncertainroutes to a bounded retry (touchpoint T5).pr-pipeline/README.md:2804; touchpoint audit T5workflow_runon 8 workflows)workflows:to the completions it acts on. It ran about 11 times per merged PR over 7d (19,756 runs, 1,727 successful merge groups).pr-feedback-dispatch.yml:99pr-pipeline/README.md:3682,:3726-3728idle-hours: 2pr-pipeline.yaml:1051Top findings by measured cost
The PR-side wait from ready to queue (ci: add Conventional Commits PR-title gate #13). p50 0.80 h and p90 7.41 h over 7d, against 0.11 h from queue entry to merge. pr-merge wakes on a 15-min cron and
unlabeledonly (pr-merge.yml:93-98). Itsunlabeledruns alone are 206 run-h/7d, mostly concurrency wait. Fix: wake per PR on pr-verify completion.ci-status is red mainly for holds, and holds are enforced in seven places. 71% of sampled red PR runs and 10 of 12 sampled red merge groups were holds, not code.
loop-lane/README.md:460callsdo-not-merge"the only cross-lane hold", yet ci-status enforces several more:needs-humanhuman-merge-holdpr-verify-failedin the queueCollapsing the own-actor ones into
do-not-merge(perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6, docs(playbook): document the local plugin development loop #7, build(dependabot): fast-track first-party melodic-software/* actions #8) removes the verdict reader from ci-status and the half-hourly re-run dependency.Review-pending re-runs. 790 re-run attempts in 7d (14% of PR CI runs) come from
rerun-ci-statusafter each review completes (chore: adopt widened comment-hygiene policy and re-pin ci-workflows @9567b8b #9). Move the review requirement off ci-status's first pass.Trigger fan-out. pr-verify, pr-review-main and pr-feedback-dispatch together ran 21,754 times in 7d (279 run-h), firing on every completion including re-runs and contract-only runs (fix(markdown-formatter): harden hook + simplify plumbing #11, fix(comment-hygiene): exclude '.' from the owner/repo#N leading boundary #12, feat(biome-format): add Biome format/lint-on-edit plugin #19).
Queue ejections on late holds (build(deps): bump ci-workflows pin to b6431a1 (PSSA RULE_ERROR retry fix) #10). 134 failed groups/7d, most of them holds. Dequeue on
labeled, and read holds once, inselect-tests.Outside the merge path but in scope, work-resolve-issue is the largest lane: 694 run-h/7d, 25% failed or cancelled. The baseline's retry-tail lever, a shorter stranded re-fire backoff for transient classes, still applies (lifecycle baseline 2026-11-01, lever 3).
turn-cap-restartsstays untouched (pr-pipeline.yaml:958).2026-10-18 gaps checklist
track done, container close-out)loop-lane/README.md:326-329,:342). A local/work-items:triageverdict is input that intake re-judges.authority: pr-merge,rung: merge,pr-pipeline.yaml:1097-1098). The rung still exists and falls automatically after a lane merge breaks main (pr-pipeline/README.md:1026-1027). No class is read (:1015).pr-pipeline/README.md:3726-3728). The revert-triggered stop is #7528's criterion and is not re-tracked here.pr-pipeline/README.md:3682; runs cachechore(release)version-bump PR fromrelease-publish-plugins.yml. Rollout is not in scope while there are no feature flags.#7612 items 2, 3, 5 and 6
work-class: untrusted-provenancestring (admit.mjs:175,:995-998), and no merge authority reads a class (pr-pipeline/README.md:1015). A lookalike label holds nothing and unlocks nothing. Creating one needs repository write access, and an actor with write access could simply remove the real label, whichuntrustedHoldcatches when a non-human does it.admit.mjs:995-998), and an add by anyone holds, failing closed. No other class gates admission or merge.report-outcome.mjsexact held-class matchreport-outcome.mjsno longer matches work classes (git grepat cb8429f finds none).predicate.mjsnormalizationpredicate.mjs:3).Done-when status
file:lineor run): met by this comment.do-not-merge(perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6-build(dependabot): fast-track first-party melodic-software/* actions #8, build(deps): bump ci-workflows pin to b6431a1 (PSSA RULE_ERROR retry fix) #10, fix(markdown-formatter): guard EPOCHREALTIME for Bash < 5 portability #17)workflow_runfan-out (fix(markdown-formatter): harden hook + simplify plumbing #11, fix(comment-hygiene): exclude '.' from the owner/repo#N leading boundary #12, feat(biome-format): add Biome format/lint-on-edit plugin #19)pull_requestrule (feat(bash-lint): add shell lint/format-on-edit plugin #16)docs/conventions/pr-pipeline/README.mdby PR): not done under this row's no-PR brief. The candidate conclusion: one hold label (do-not-merge) for own-actor holds, with person-only holds kept for critical security, cost and trust paths. It also needs the inconsistency atloop-lane/README.md:460fixed.Target labels: add none; remove none.
kyle-sexton commentedon Nov 3, 2026
Follow-ups for the three findings recommendations that had no issue (criterion 3): row 14, release merge groups scoped to the release diff, is #12633; row 15, the lane resolving answered non-CRITICAL bot threads on owner-login PRs, is #12634; the work-resolve-issue retry-tail lever is #12635. Row 14 is not an owner decision: ADR 0048:124-128 fixes only how a release merges, and narrowing which suites a release group runs touches none of the three owner stop reasons. Agent choice, not an owner answer; written by the issue-triage session.