pr-pipeline: post-V1 audit and retro of the unattended merge flow

Ask

The owner's direction (2026-10-12): ship V1 of unattended merge first, then audit, analyze and run a retro on the whole flow end to end, adopting outside practices (Matt Pocock, poteto) only on data and metrics.

Scope, once `pr-merge` merges live

CI/CD performance, the autonomous Actions lanes, and the merge path; find the friction points.
From day one of live merges: revert and demotion rate per risk bucket, escaped-defect counts, and merged-and-not-reverted outcomes.
Candidates to evaluate on that data: sampled post-merge review, a verifier verdict bound to the head SHA, a mutation pass over changed tests.

Refs: #6202, #4588

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #7528

Decision recorded.
The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

[quote truncated; the full decision is at the link]

#7531 (comment)

The decision settles the open questions. #7531 keeps the data retro, which starts at 30 live pr-merge merges or 30 days, whichever is later. The static audit moves to a child issue. The deliverable is a report under docs/conventions/pr-pipeline/ plus follow-up issues, with no change to lane behavior. The measurement window can't be checked in an unattended run, so that item waits for live merges. Demotion counts depend on #7528.

_{Automated triage, applied by the lanes App.}

What the article adds for this audit, and the owner's 2026-10-16 decisions. Source: https://claude.dev/blog/solving-code-review-at-anthropic (2026-10-15); ids are digest unit and claim.

The article: "We don't yet have circuit breakers that tighten approval when those numbers move; engineers here want them before more code moves to zero-human merges" (04 C70). Noticing a bad merge still depends on the author, on-call and the revert and incident dashboards (04 C69). Revert and incident rates are the guardrail (02 C8; 10 C21).

Owner decision, 2026-10-16: a minimal revert-triggered stop. A revert of an agent-merged PR stops that lane's unattended merges until the owner re-enables them. It stops merges rather than lowering a rung because Phase 9 removes the rung ladder (Q505). It is not a parallel mechanism: a revert of an agent-merged PR feeds the existing post-merge demotion record, post-merge-detect-silent-revert.yml plus the babysit-merge-demotion issues that only a person closes, which already stop unattended merges on a canary or schedule failure. It is built with the #7528 fix and lands before unattended merge widens.

Today the rung falls only when the default branch fails after a lane merge (docs/conventions/pr-pipeline/README.md:344-347); feat(ci): file a demotion issue when a post-merge run fails #7329 files a demotion issue when a post-merge run fails, and fix(source-control): fail closed when evidence runs or demotion issues are deleted #7526 fails closed when evidence runs or demotion issues are deleted. A revert is a new trigger on the same path.
Revert signals already exist: post-merge-detect-silent-revert.yml and revert commits.
pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528: post-merge runs dispatched with GITHUB_TOKEN file no demotion issue, so that path drops events. The revert trigger depends on the fix.

Owner decision, 2026-10-16: this issue owns the revert and incident counts. #6080 and #6974 read them from here rather than keeping their own.

Not adopted: a rate-based breaker. Q567 sends confirmed escaped defects to /corrections:codify, not demotion. That is recorded as a departure from 02 C8 and 04 C70; the SRE error-budget policy (halt changes when the budget is exceeded) sides with the article. This issue's per-bucket revert rate is the data that revisits it: on 2026-10-16 the owner kept Q567's unattended merge on a verifier verdict on condition of revisiting it on this data. The comparison to make: the revert and escaped-defect rate on verifier-approved merges, set against the planted-defect miss rate from Phase 9's B4 run.

Against the originating sources.

pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202 (the 2,000-PRs effort, from poteto's work) already plans rung lowering after a bad merge; the revert stop is its minimal form once the rungs are gone.
This issue's rule, adopt outside practices only on data, agrees with "start strict and loosen with evidence" (10 C14).
Order: the revert stop lands before unattended merge widens, which meets the article's engineers part way (04 C70); the rate breaker stays a departure.

The article's own revert figures are internal and vendor-claimed (00 C10).

Related: #6080, #6974, #6989, #7570, #7528, #7329, #7639, #7642.

Gaps moved from #6989, which is closing as completed (owner decision D18, option a, 2026-10-18). They are lifecycle and post-merge items to weigh in this audit and retro. Source for all: the #6989 body, sections named per item.

No owner for post-merge bookkeeping (track done, container close-out). "Gaps found".
Triage runs in two places (CI intake lane and local /work-items:triage) with no rule for which decides. "Gaps found".
Merge authority moves from the local babysit loop to pr-merge in one switch, and the single merge.authority field with rung lowering after a bad merge (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202). Phase 9 removes the rung ladder (Q505), so check whether this is still a gap. "Gaps found" and "Prior art".
Post-merge failure response: revert or fix forward. The revert stop is decided here; fix-forward is not. "Open, deferred to the pr-unblock design".
Post-merge smoke check against the deployed environment (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202). "Prior art" (the silent-revert canary half already exists as post-merge-detect-silent-revert.yml).
Stages outside the plan: build/package; monitor after deploy with rollback; incident response; postmortem; migrations before and after deploy. Define "release" (version cut) and add rollout if feature flags come into scope. "Stages and gates outside sources include that the plan lacks".
Candidates to evaluate on this issue's data: swarm verifiers with one verdict on the exact SHA (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); a per-app verify-<app> skill beyond testing:run-e2e (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); a learn stage that mines session corrections into skills or lint rules (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); routing new work by size and fog (feat(planning): route new work by diff size and fog: one-shot, /planning:interview, or /planning:wayfind #6729, Epic: improve the planning chain from html-plan, HumanLayer rpi and Pocock sources #6917); a cross-PR model-run cap, to be added only on usage telemetry. "Prior art" and "Decisions".

Already listed here, so not added: the revert-triggered stop and revert and incident counts (2026-10-16 owner decision above), and "start strict and loosen with evidence" (10 C14), which the risk-tier comment ties to this issue's data. Kill-switch coverage of the review and intake lanes is tracked in #7234.

This was generated by AI during attend-queue.

Folded in from #7612 (closed obsolete after #7780 removed the ratification marker): its items 2, 3, 5 and 6, the label-normalization restrictions on admission, belong to this audit.

Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

No description provided.

The work-resolve-issue lane stopped on this issue (stopped): the run stopped for a reason it gave no code for, which names no intent, design, security or cost finding, so the lane runs it again only if the issue has had no retry, for any stop, since a listed human last added agent-ready, and otherwise stops.
It keeps agent-ready, and work-unblock starts it again (retry 1 of 1).

In the run's words:

The data retro this issue keeps cannot start yet. The owner's decision of 2026-10-26 (comment 6412925022) moved the static friction audit and the gaps checklist to a child issue and kept only the data retro here, starting once pr-merge has made 30 live merges or 30 days have passed, whichever is later. The pr-merge lane's merge: block first entered the live config on 2026-10-14 with report-only: true (docs/conventions/pr-pipeline/CHANGELOG.md, 1.11.0 - 2026-10-14), so no live merge could have happened before then. The 30-day window therefore ends 2026-11-13 at the earliest, and today is 2026-10-31. The decision also lists the data retro as a check a person runs once its trigger is met. Nothing was edited. Run this issue again on or after the date when both thresholds are met.

What the run says it did not do:

The data retro (revert and demotion rate per risk bucket, escaped-defect counts, merged-and-not-reverted outcomes, candidate evaluation): its trigger is not met before 2026-11-13, and the owner's decision lists it as a check a person runs once the trigger is met
The decision's first acceptance criterion, a child issue holding the static audit: I did not find that child issue by number, and this lane can only read the tracker, so it could not file one. A person should confirm the child issue exists or file it
The exact date of the first live pr-merge merge (when report-only turned false): neither the shallow checkout nor the pages I read date it, so the earliest possible start, 2026-10-14, bounds the window instead
tidy, review, verify and checks steps: skipped because there is no diff

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41152367960

No description provided.

The work-resolve-issue lane stopped on this issue (stopped): the lane has retried this issue as many times as limits.stop-retries, or the stop's own row, allows.
It removed agent-ready and added no label.

In the run's words:

data-window-not-met: the 2026-10-26 decisions keep only the data retro on ＃7531, gated on 30 live pr-merge merges or 30 days since the first live merge, whichever is later. The pr-merge lane first entered the live config on 2026-10-14, shipping report-only (docs/conventions/pr-pipeline/CHANGELOG.md 1.11.0 and 1.12.1), so no live pr-merge merge can predate 2026-10-14 and the 30-day window ends no earlier than 2026-11-13. Today is 2026-10-31, so the retro cannot start yet. The decision record also lists the data retro under 'Checks a person runs'.

What the run says it did not do:

The data retro report (per-bucket revert and demotion rates, escaped defects, merged-and-not-reverted outcomes, Q567/Q602 revisit, candidate evaluations): its trigger (30 live merges and 30 days, whichever is later) cannot be met before 2026-11-13. Without gh or API access, this lane also cannot rea
The exact date merge.report-only became false was not found: the repository is a shallow clone and the CHANGELOG has no entry for it. The 2026-10-14 lower bound is enough for this stop.
The child issue for the static audit (first 2026-10-26 acceptance criterion) was not filed or checked: this lane cannot create or search issues.
tidy, review, verify: skipped because there is no change

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41157262688

To run it again, add agent-ready.

Status. Two things: the static audit is now filed, and the data retro waits on two conditions. Author: the issue-triage session.

The static friction audit and the 2026-10-18 gaps checklist are filed as #12284. That is the first criterion of the 2026-10-26 record, with the deliverable from the amendment. Both lane runs reported that they could neither find nor file it.

The data retro this issue keeps starts when both of these hold:

30 days have passed since the first live pr-merge merge.
pr-merge has made 30 live merges.

Live merges began when report-only turned off on 2026-10-17. That was #7773, commit ad13c59, which set report-only: false in docs/conventions/pr-pipeline.yaml. The lane runs bounded the window by 2026-10-14, when the merge block entered the config as report-only, and that gave 2026-11-13. No live merge could happen before 2026-10-17, so the earliest start is 2026-11-16.

The record lists the retro under "Checks a person runs", so this issue keeps host-run.

Navigation Menu

Repository navigation

pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531

Ask

Scope, once `pr-merge` merges live

15 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531

Description

Ask

Scope, once pr-merge merges live

Activity

github-actions commented on Oct 15, 2026

kyle-sexton commented on Oct 16, 2026

kyle-sexton commented on Oct 18, 2026

kyle-sexton commented on Oct 22, 2026

kyle-sexton commented on Oct 24, 2026

15 remaining items

github-actions commented on Oct 31, 2026

github-actions commented on Oct 31, 2026

melodic-automation-lanes commented on Oct 31, 2026

github-actions commented on Oct 31, 2026

github-actions commented on Oct 31, 2026

melodic-automation-lanes commented on Oct 31, 2026

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Scope, once `pr-merge` merges live