Repository navigation
feat(verification): make confirm consume affected and add the evidence loop #11389
Description
Parent
Refs #8651. Evidence-verification plan: Brief and acceptance criteria at #11364 (comment); Design and Plan comments are on #11364.
Depends on
Outcome
/verification:confirm takes touched features from affected (presence-gated), keeps the evidence loop (cap 3) in evidence-loop.<branch-slug>.json, and never fails a verdict on a capture that could not run.
Demo: run /skill-quality:check validate-evals confirm; the loop-count, cap, blocked-capture and app-control-absent cases pass.
Plan (phase S2b)
Satisfies: AC3, AC5, AC6, AC7, AC10, AC11
Size Medium. Depends on: S1 (#11384) (S2a (#11388) for the summary shape, by contract only; no S3 (#11387) fields used). Plugin: verification. Design: Contracts C5, T6, T10.
plugins/verification/skills/confirm/SKILL.md: replace the touched-feature parsing atplugins/verification/skills/confirm/SKILL.md:168withaffected, presence-gated: when/app-control:drivedoes not resolve, reporttouched features not matched (app-control not installed)and keep thee2e-*path categories (eval for that case); fresh verifier reads the full local run dir (never uploaded;publishablegates uploads only); at cap with the last verdict NEEDS WORK the manifest saysEvidence loop: 3/3 cap reached, last NEEDS WORK;evidence-loop.jsonkeyed by branch (evidence-loop.<branch-slug>.json), written atomically (write temp, rename), counting NEEDS WORK evidence verdicts on distinct heads (a rebase-only head judged NEEDS WORK counts), cap 3, at cap no loop-back and Next routes to the PR and pipeline park path; manifest lineEvidence loop: <k>/3; evidence dirs kept for the last 3 heads; capture not run → verdict notes why, never NEEDS WORK for that alone.context/outcome.md: report rows for capture status and loop count.- Sanity Check: evals "NEEDS WORK at count 2 → count 3, loop-back", "count 3 → no loop-back, cap line", "capture blocked → verdict notes reason, not NEEDS WORK" and "app-control absent → fallback line" pass under
/skill-quality:check validate-evals confirm;grep -n "Evidence loop" plugins/verification/skills/confirm/SKILL.mdmatches. - Run budget: at most 2 runs per eval case, 1 rerun after a fix; stop on the first usage-limit error.
Design excerpt
- verification / confirm: calls
affected(replacesplugins/verification/skills/confirm/SKILL.md:168parsing); verifier reads the
full local run dir (publishablegates uploads only); ownsevidence-loop.json(cap 3) and local evidence retention (last 3 heads).
components:
schemas:
EvidenceLoop: # C5, <memory_dir>/<slug>/verification/evidence-loop.json
type: object
required: [cap, count, heads]
properties:
cap: {const: 3}
count: {type: integer, minimum: 0, maximum: 3}
heads:
type: array
items: {type: object, properties: {sha: {type: string}, verdict: {enum: [CONFIRMED, NEEDS WORK, NOT VERIFIED]}, at: {type: string, format: date-time}}}- confirm:
evidence-loop.<branch-slug>.json; manifest lineEvidence loop: <k>/3; at 3 no loop-back, Next =
open the PR so pr-verify FAIL →max-verify-fails→ fresh attempt → park
(docs/conventions/pr-pipeline.yaml:1068-1082).
Acceptance criteria
Brief criteria this slice satisfies:
- AC3: Diff-to-feature matching lives only in app-control; confirm and run-e2e call it instead of parsing control skills.
- AC5: If a capture cannot run (blocked, or no usable driver), the verdict records why and does not fail the PR.
- AC6: A failed evidence verdict sends the work back to implementation, up to a cap of 3.
- AC7: While the loop counter is at its cap, no further loop-back runs and the existing park-and-retry path takes over.
- AC10: Every evidence artifact has a retention period.
- AC11: Verdict, evidence-loop count and capture count are recorded per PR.
Done when:
- Sanity Check passes: evals "NEEDS WORK at count 2 → count 3, loop-back", "count 3 → no loop-back, cap line", "capture blocked → verdict notes reason, not NEEDS WORK" and "app-control absent → fallback line" pass under
/skill-quality:check validate-evals confirm;grep -n "Evidence loop" plugins/verification/skills/confirm/SKILL.mdmatches. - The glob-parsing copy at the old
confirm/SKILL.md:168is removed (dead-code removal folded into this slice) -
evidence-loop.<branch-slug>.jsonis written atomically (temp file, rename) - Manifest line
Evidence loop: <k>/3is written - The plugin's version and CHANGELOG are bumped per the repository's release convention
-
ci-statusis green on the PR head
🤖 Generated with Claude Code
Activity
github-actions commentedon Oct 30, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: make
/verification:confirmtake touched features fromaffected(presence-gated on app-control), keep a per-branch evidence loop (cap 3) inevidence-loop.<branch-slug>.jsonwritten atomically, and never fail a verdict on a capture that could not run.The admission note flagged "retention" as a charge. Here it means keeping the last 3 local evidence directories on disk. Nothing bills, so no cost decision is needed.
The stated dependency #11384 is not in the open-issues list, so I treated it as not blocking. The outcome and done-when items are exact. I found no duplicates.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 30, 2026
Not admitted to the issue lane: its title or stated work names a charge (retention), which needs an owner decision. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 30, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: needs-decision; its comment says why.kyle-sexton commentedon Oct 31, 2026
Decision recorded.
Admission held this issue on a charge word that names nothing billable. Asked on 2026-10-31 about the charge-word rule, the owner replied with the standing direction: "Go with whatever is in line with this". Its §3 limits cost to what can bill the owner. On this public repository, standard GitHub-hosted runners are free (GitHub Actions billing), and the retention this issue sets is for local run directories and a retention period on each artifact, which bills nothing and caps storage. The Issue Triage session reads the hold as false and releases the issue; its criteria are unchanged. The word list itself is narrowed separately (
billedmoves to the private-repository list), andretentionkeeps holding elsewhere, because §3 names artifact retention beyond the free quota as a cost.kyle-sexton commentedon Oct 31, 2026
Correction to the record above, from the Issue Triage session: the charge-word list is not changing. Moving billed to the private-repository list would let a public-repo issue about Azure or over-quota storage pass admission, and section 3 names both as cost. This issue's release stands on its own reason: its retention bills nothing.
github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
melodic-automation-lanes commentedon Nov 1, 2026
This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 41231702342, the work-resolve-issue run that
agent-readystarted, before it did any work; that run ended at 2026-10-31T19:27:25Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits longer before each one, up to 24 hours.github-actions commentedon Nov 1, 2026
github-actions commentedon Nov 1, 2026