Skip to content

pr-pipeline: static friction audit of the unattended merge flow and the 2026-10-18 gaps checklist (child of #7531) #12284

Open
3 of 7 issues completed

Description

@kyle-sexton

Refs: #7531

Problem

The 2026-10-26 record on #7531 split #7531. The data retro stays there, gated on live merges. The static friction audit and the 2026-10-18 gaps checklist move to a child issue, because they need no live merge data. That record's first criterion is this child.

The 2026-10-26 amendment changed the deliverable. The findings are posted as a comment on this issue, plus follow-up issues, and any lasting conclusion goes into docs/conventions/pr-pipeline/README.md. No report file is committed (AGENTS.md: one-off investigations stay in the scratchpad, and only the conclusion is committed). The revert-triggered stop is a criterion on #7528's record, so the audit cites #7528 for it.

Routing: host-run. Posting findings as a comment and filing follow-up issues are tracker writes, and the issue lane can only read the tracker. Both #7531 lane runs say so (6536975267, 6537727272).

Scope, from #7531's body: CI/CD performance, the autonomous Actions lanes, and the merge path; find the friction points.

The 2026-10-18 gaps checklist (comment), from that comment:

That comment's last item, the candidates to evaluate on #7531's data, stays with #7531's data retro.

Folded in from #7612 (comment): its items 2, 3, 5 and 6, the label-normalization restrictions on admission.

Done when

  • A findings comment on this issue covers CI/CD performance, the autonomous Actions lanes and the merge path, and names each friction point with the file:line or run that shows it.
  • Each of the six gaps-checklist items above, and each of pr-pipeline: harden issue-lane admission after the ratification marker #7612's items 2, 3, 5 and 6, is answered in that comment as keep, drop, or a follow-up issue number.
  • Every follow-up the findings recommend is a filed issue, cited by number.
  • Any lasting conclusion is written into docs/conventions/pr-pipeline/README.md through a PR. No report file is committed, and no lane behavior changes in this issue.
  • The revert-triggered stop is cited as pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528's criterion, not re-tracked here.

Activity

  1. added theissue type on Nov 2, 2026
  2. added
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 2, 2026
  3. github-actions commented on Nov 2, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Large
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    I understood this as a static audit of the unattended merge flow. It covers CI/CD performance, the autonomous Actions lanes and the merge path. It also answers the six 2026-10-18 gaps-checklist items and #7612's items 2, 3, 5 and 6. The deliverable is a findings comment, follow-up issues for each recommendation, and any lasting conclusion written into the pr-pipeline README through a PR. The done-when is exact, and the tracker writes are already routed to host-run. The audit spans several subsystems, so it may need splitting. I found no likely duplicates among the open issues.

    Automated triage, applied by the lanes App.

  4. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 2, 2026
  5. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

  7. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Claim. Backlog HITL host-run worker is taking this now (read-only static friction audit; findings comment follows).

  8. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Host run. A read-only static friction audit of the unattended path from a ready PR to merge, with run costs measured from the hub's local runs cache and a sampled read of failing ci-status steps.

    What ran

    • Code read at origin/main cb8429f3cf (2026-11-02T15:30Z). Every file:line below is at that commit.
    • Live ruleset read: GET repos/.../rules/branches/main.
    • Run costs come from the hub's local runs cache, which holds every workflow run with its created, started and updated times, conclusion and attempt. Two windows: 7d (2026-10-26T00Z to 11-02T00Z) and 1d (2026-11-01). This used no API calls. "Run-h" means run wall hours (updated_at - run_started_at) over non-skipped runs. On this public repository hosted runners do not bill, so run-h measures runner contention and, for the model lanes, subscription usage. It is not a charge (costs).
    • Failure causes come from a seeded random sample of failed pr-require-checks runs on 11-01, read through actions/runs/{id}/jobs:
      • 25 pull_request runs, of which 24 were readable (one returned HTTP 502).
      • 12 of the 24 merge_group runs.
      • Three runs' ci-status annotations.
    • Prior measurements are reused where they still hold: the 2026-11-01 lifecycle baseline, the 2026-10-28 owner-touchpoint audit, and the 2026-11-01 pr-merge call counts. Their figures are labeled as quoted.
    • About 50 GitHub calls in total, with no rate-limit answer.
    • No gate, workflow, label rule or config was changed, and no PR was opened.

    Where the time goes (measured)

    Workflow (event) Runs 7d Run-h 7d p50 / p90 min Fail share 7d 1d (11-01) runs / run-h
    work-resolve-issue (issues) 1,182 non-skipped of 4,733 694 27.1 / 77.4 227 failed + 71 cancelled (25%) 233 / 110
    pr-require-checks (pull_request) 5,515 291 3.2 / 6.6 1,143 (21%); 790 re-run attempts (14%) 1,162 / 56 (192 re-runs)
    pr-merge (pull_request_target unlabeled) 1,011 206 10.6 / 23.3 0.3% 38 / 7.5
    pr-require-checks (merge_group) 1,885 200 5.4 / 9.5 134 (7.1%) 363 / 38
    pr-merge (15-min schedule) 607 173 16.1 / 31.3 2.6% 94 / 27
    pr-unblock (dispatch) 1,024 133 7.0 / 13.1 118 (12%) 230 / 35
    post-merge-test-mutation 482 123 12.9 / 31.7 30 (6%) 89 / 21
    pr-review-hosted + pr-review-security-hosted 3,089 each 118 + 105 2.0 / 3.6 and 1.7 / 3.2 under 0.5% 648 each / 53
    pr-verify (workflow_run) 5,470 of 6,604 114 0.9 / 2.0 0.1% 1,257 / 19
    pr-feedback-dispatch (workflow_run) 12,345 of 19,756 101 0.3 / 0.8 0.6% 2,786 / 26
    pr-refine (dispatch) 1,035 68 2.6 / 6.9 59 (6%) 221 / 17
    pr-review-main (workflow_run) 3,939 64 0.5 / 2.3 0.1% 1,372 / 20
    pr-require-checks full main (dispatch) 466 68 8.6 / 9.5 13 54 / 9

    About 336 PRs merged through the queue on 11-01 (successful merge_group runs). On that day the merge path after a PR is ready used roughly 300 run-h. The lifecycle baseline (2026-11-01, quoted) puts the PR-side wait at ready -> first queue entry p50 0.80 h / p90 7.41 h over 7d, and queue entry -> merged at p50 0.11 h. The wait before the queue is the binding stage.

    Why ci-status is red (sample)

    Of 24 failed pull_request runs:

    • 7 were real lane failures (lint-repo, lint-shell, check-plugins/check-catalog, test-bash, test-node).
    • 17 (71%) were holds.
    Cause PR runs (of 24) Merge-group runs (of 12) Example runs
    do-not-merge label (intended cross-lane hold) 8 1 41342742932, 41319120018
    pr-unblock failure status on the head 6 1 41332979407, 41344950103
    needs-human label 3 0 41306188307
    Review lanes not yet complete on the head 2 n/a 41327400610
    Lanes App body hold or member veto n/a 6 41348597754, 41324239904
    Hold label added after the PR was enqueued n/a 4 41361037999, 41317739074
    Real lane failure 7 2 41280047305, 41278145401

    A red ci-status from do-not-merge or needs-human is the hold doing its job, not a defect. The friction rows are the others:

    • The pr-unblock stop status.
    • The lanes App body and member hold.
    • Holds that land after enqueue: 10 of the 12 sampled merge-group failures are holds, not code. Each one ejects the PR and rebuilds the group for its peers.
    • The review-pending re-run loop.

    Gate by gate

    Owner direction s10: a hold against our own actors is friction. The only HIGH/CRITICAL boundary is a token or secret leak, an account compromise, or billing. No verdict below loosens one of those.

    # Gate or step Risk it covers Existing control on the same risk Measured cost Verdict Basis
    1 ci-status, the one required check (ruleset required_status_checks, strict_required_status_checks_policy: false) Broken code reaching main Merge-queue re-run (#14) 291 run-h/7d on PR runs; 21% red, 71% of the red sample being holds Keep pr-require-checks.yml:4003; ruleset read
    2 Contract step: title and the do-not-merge label A held or misnamed PR merging None: this is the one documented cross-lane hold 8 of 24 red PR runs Keep pr-require-checks.yml:4795; loop-lane/README.md:460
    3 needs-human label holds A PR a person must see Overlaps #2 for own-actor holds 3 of 24 Keep for the review lanes' cost and critical-security holds. Simplify other uses to do-not-merge (#17) pr-require-checks.yml:5445; pr-pipeline/README.md:2804
    4 Only a trusted person lifts a review-lane hold A bot clearing a critical-security or cost hold None Not seen in the sample Keep (boundary: secret leak, billing) pr-require-checks.yml:5551; pr-pipeline/README.md:2806
    5 Cost-only review hold cleared by an owner decision A real charge merging unseen #4 Not seen in the sample Keep (billing) pr-require-checks.yml:5122
    6 human-merge-hold label, lifted only by a trusted person acting directly A lanes App PR whose body says "Hold: a human merges this" #2 does the same job with one label Not seen in the sample Simplify: fold into do-not-merge. It is an own-actor hold. pr-require-checks.yml:5686; pr-pipeline/README.md:2812
    7 Lanes App body hold or member veto, read by the verdict reader in ci-status A lanes PR a member asked to wait on #2. The babysit gate already reads body holds (AGENTS.md) 6 of 12 merge-group failures. The code comment quotes a cold verdict read in the queue at "about 860" calls and 6-11 min (measured 2026-10-25) Simplify: when a member vetoes, the lanes App adds do-not-merge, and ci-status reads the label only. That removes the verdict reader from ci-status. pr-require-checks.yml:6247, comment at :4029-4051
    8 pr-unblock failure commit status holds the head Merging a head a fixer lane gave up on #2, and the ci-lanes result for real failures 6 of 24 red PR runs, 1 of 12 merge-group. Recovery waits for the half-hourly reconcile re-run Simplify: post do-not-merge (or nothing, when ci-lanes is already red) instead of a separate status context. That drops one hold mechanism and its re-run dependency. pr-require-checks.yml:6186; pr-pipeline/README.md:2863-2869; maintenance-reconcile-lost-runs.yml:93
    9 Both review lanes must have run on the head An unreviewed head merging; the security lane catches token and secret exposure None for security. pr-review-main (#11) for lane-source PRs 223 run-h/7d of review runs; 790 ci-status re-runs/7d (14% of PR runs), driven by rerun-ci-status after each review completes Keep both lanes: the code-review lane also raises the cost hold (pr-review-hosted.yml:242-250), so it is a billing control. Simplify the ordering so ci-status does not start red on "review pending". Options: start the review lanes so they gate pr-verify instead of ci-status, or fold the review check into pr-merge's verdict. Either removes the re-run loop. pr-require-checks.yml:6755; pr-feedback-dispatch.yml:575; pr-pipeline/README.md:2872
    10 Hold checks inside the merge group (labels, pr-verify-failed, pr-unblock status, lanes body hold) A hold added after enqueue #2-#8 on the PR run Holds are 10 of 12 sampled merge-group failures; 134 failed groups/7d at p50 5.4 min, plus peer rebuilds Keep the check. Simplify the order: the earlier select-tests step (:958) already fails a held PR "before the lanes", so make it the only queue-time hold read and drop the duplicates in ci-status (:5779, :6186, :6247). Better still, dequeue the PR on labeled instead of waiting for the group run. pr-require-checks.yml:958, :5779
    11 pr-review-main: re-review from main for PRs that change a lane source A head forging its own review verdict None 3,939 runs and 64 run-h/7d. It fires on every pr-require-checks completion, contract-only runs and re-runs included (1,372 runs on 11-01 for 745 distinct PR heads, about 1.8 per head), and most runs exit early Keep the gate (forgery of a security review). Simplify-later the trigger: skip completions whose ci-status already ran on that head. pr-review-main.yml:3, :59
    12 pr-verify: verifier verdict bound to the head Merging a head the verdict did not judge None 5,470 runs and 114 run-h/7d; also fires on every completion (workflow_run), about 1.8 per head Keep. Simplify-later as #11 pr-verify.yml:67-71
    13 pr-merge lane wake: 15-min cron, the unlabeled event and push Missing a PR that just became mergeable none Ready -> queue p50 0.80 h / p90 7.41 h (7d, quoted). Schedule: 607 runs, 173 run-h. unlabeled: 1,011 runs and 206 run-h, mostly waiting on the shared concurrency group (2026-11-01 sample, quoted: "26 s after 21 min queued"). The file says it "does not wake on each CI or review completion" Simplify: wake per PR on pr-verify completion (per-PR concurrency group). Drop unlabeled as a wake once that event exists, and slow the cron to a backstop. This is the largest PR-side wait. pr-merge.yml:85, :93, :98
    14 Merge queue (ruleset: ALLGREEN, max 5, check_response_timeout_minutes 30) re-runs full CI per group Semantic conflicts between PRs merged with no rebase (strict is false) None 1,885 groups and 200 run-h/7d; 7.1% fail, mostly holds (#10) Keep. Release merge groups widen to the whole tree (quoted baseline: about 38 a day); scope them to the release diff. That is the owner's call, per the baseline. ruleset read; lifecycle baseline 2026-11-01
    15 Ruleset required_review_thread_resolution: true A review finding left unanswered Lanes settle bot threads (pr-unblock / publish-feedback) Owner resolves were about 30 threads over 16 PRs in 23 h on 10-27/28 (quoted touchpoint audit T4); not re-measured Keep. Simplify: let the lane resolve outdated, answered, non-CRITICAL bot threads on owner-login PRs (touchpoint T4). ruleset read
    16 Ruleset: two pull_request rules, one with dismiss_stale_reviews_on_push and 0 required approvals; signatures; linear history Unsigned or forged commits; approval drift none No measured cost; dismissal only affects the trust-path approvals for Dependabot Keep signatures (supply chain). Remove-later the duplicate pull_request rule once someone confirms which ruleset owns the Dependabot approval path (UNVERIFIED which). ruleset read
    17 needs-human used as a general park by lanes (pr-unblock escalation routes) A disputed finding merging unjudged #2 123 PRs with a pr-unblock stop comment in 7d (quoted, 10-28) Simplify: keep person routes only for CRITICAL security or cost. Send other feedback-disputed/feedback-uncertain routes to a bounded retry (touchpoint T5). pr-pipeline/README.md:2804; touchpoint audit T5
    18 Fixer lanes pr-unblock and pr-refine Red CI, review threads, docs drift none 133 + 68 run-h/7d; pr-unblock 12% failed Keep. A failed pr-unblock run is a source of #8's holds. runs cache
    19 pr-feedback-dispatch router (workflow_run on 8 workflows) Routing completions to the next lane none 19,756 runs/7d (7,411 skipped), 101 run-h Simplify-later: narrow workflows: to the completions it acts on. It ran about 11 times per merged PR over 7d (19,756 runs, 1,727 successful merge groups). pr-feedback-dispatch.yml:99
    20 Post-merge: full main run, demotion alert, red-main revert or fix-forward, mutation tests A bad merge staying on main Merge queue (#14) Full main runs 466/7d at p50 8.6 min (now about one per 27 min on 11-01, inside the 30-min cap; the baseline's 14-min gap no longer holds); mutation 123 run-h/7d Keep pr-pipeline/README.md:3682, :3726-3728
    21 Owner-login PR adoption idle wait A person's PR merging while they still work on it Draft state Was 24 h on 10-28; now idle-hours: 2 Already simplified (touchpoint T3) pr-pipeline.yaml:1051

    Top findings by measured cost

    1. The PR-side wait from ready to queue (ci: add Conventional Commits PR-title gate #13). p50 0.80 h and p90 7.41 h over 7d, against 0.11 h from queue entry to merge. pr-merge wakes on a 15-min cron and unlabeled only (pr-merge.yml:93-98). Its unlabeled runs alone are 206 run-h/7d, mostly concurrency wait. Fix: wake per PR on pr-verify completion.

    2. ci-status is red mainly for holds, and holds are enforced in seven places. 71% of sampled red PR runs and 10 of 12 sampled red merge groups were holds, not code. loop-lane/README.md:460 calls do-not-merge "the only cross-lane hold", yet ci-status enforces several more:

      • needs-human
      • human-merge-hold
      • review-lane hold lifts
      • pr-unblock statuses
      • lanes App body holds and vetoes
      • pr-verify-failed in the queue

      Collapsing the own-actor ones into do-not-merge (perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6, docs(playbook): document the local plugin development loop #7, build(dependabot): fast-track first-party melodic-software/* actions #8) removes the verdict reader from ci-status and the half-hourly re-run dependency.

    3. Review-pending re-runs. 790 re-run attempts in 7d (14% of PR CI runs) come from rerun-ci-status after each review completes (chore: adopt widened comment-hygiene policy and re-pin ci-workflows @9567b8b #9). Move the review requirement off ci-status's first pass.

    4. Trigger fan-out. pr-verify, pr-review-main and pr-feedback-dispatch together ran 21,754 times in 7d (279 run-h), firing on every completion including re-runs and contract-only runs (fix(markdown-formatter): harden hook + simplify plumbing #11, fix(comment-hygiene): exclude '.' from the owner/repo#N leading boundary #12, feat(biome-format): add Biome format/lint-on-edit plugin #19).

    5. Queue ejections on late holds (build(deps): bump ci-workflows pin to b6431a1 (PSSA RULE_ERROR retry fix) #10). 134 failed groups/7d, most of them holds. Dequeue on labeled, and read holds once, in select-tests.

    Outside the merge path but in scope, work-resolve-issue is the largest lane: 694 run-h/7d, 25% failed or cancelled. The baseline's retry-tail lever, a shorter stranded re-fire backoff for transient classes, still applies (lifecycle baseline 2026-11-01, lever 3). turn-cap-restarts stays untouched (pr-pipeline.yaml:958).

    2026-10-18 gaps checklist

    Item Answer Basis
    No owner for post-merge bookkeeping (track done, container close-out) Follow-up needed. Merged -> issue closed is p50 0 h (Closes links work). Parent close-out lags: 25 of 67 open parents had every sub-issue closed on 2026-11-01 (quoted baseline; UNVERIFIED today). The owner is work-unblock's parent auto-close; the follow-up is why it skips them. lifecycle baseline 2026-11-01 §2
    Triage in two places Keep, answered. Intake triage owns the stop classification for issues, and admission re-judges before admitting (loop-lane/README.md:326-329, :342). A local /work-items:triage verdict is input that intake re-judges. cited lines
    Merge authority switch and rung lowering (#6202, Q505) Keep, no gap. One authority (authority: pr-merge, rung: merge, pr-pipeline.yaml:1097-1098). The rung still exists and falls automatically after a lane merge breaks main (pr-pipeline/README.md:1026-1027). No class is read (:1015). cited lines
    Post-merge failure: revert or fix forward Keep, decided. The bot reverts only when every guard holds; otherwise fix issues fix main forward through the issue lane (pr-pipeline/README.md:3726-3728). The revert-triggered stop is #7528's criterion and is not re-tracked here. cited lines; #7528
    Post-merge smoke check against a deployed environment (#6202) Drop. This repository deploys no environment. The full main run after merges plus post-merge-test-mutation cover main (#20). pr-pipeline/README.md:3682; runs cache
    Stages outside the plan (build/package, post-deploy monitoring, incidents, postmortems, migrations, a release definition) Drop for this repository, with one exception: "release" is the chore(release) version-bump PR from release-publish-plugins.yml. Rollout is not in scope while there are no feature flags. judgment; workflow list

    #7612 items 2, 3, 5 and 6

    Item Answer Basis
    2. Lookalike-Unicode work-class labels Drop. Admission now matches the exact work-class: untrusted-provenance string (admit.mjs:175, :995-998), and no merge authority reads a class (pr-pipeline/README.md:1015). A lookalike label holds nothing and unlocks nothing. Creating one needs repository write access, and an actor with write access could simply remove the real label, which untrustedHold catches when a non-human does it. cited lines
    3. Who applied a work-class label Drop. A non-human's removal of the untrusted label keeps the hold (admit.mjs:995-998), and an add by anyone holds, failing closed. No other class gates admission or merge. cited lines
    5. report-outcome.mjs exact held-class match Drop (obsolete). report-outcome.mjs no longer matches work classes (git grep at cb8429f finds none). grep
    6. predicate.mjs normalization Drop (obsolete). "No key reads a work class" (predicate.mjs:3). cited line

    Done-when status

    Target labels: add none; remove none.

  9. kyle-sexton commented on Nov 3, 2026

    @kyle-sexton
    MemberAuthor

    Follow-ups for the three findings recommendations that had no issue (criterion 3): row 14, release merge groups scoped to the release diff, is #12633; row 15, the lane resolving answered non-CRITICAL bot threads on owner-login PRs, is #12634; the work-resolve-issue retry-tail lever is #12635. Row 14 is not an owner decision: ADR 0048:124-128 fixes only how a release merges, and narrowing which suites a release group runs touches none of the three owner stop reasons. Agent choice, not an owner answer; written by the issue-triage session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    Work status

    confirmed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions