Repository navigation
attribution: owner rulings on blocked provenance panels and verbatim target #5654
Description
Refs #3465. Split from #5355, which closes as completed: near-verbatim has 13 cases, and verbatim stays at 3 for the reason recorded in the attribution 0.8.2 CHANGELOG.
Summary
Open provenance golden-set items to adjudicate. Adjudication first tries a deterministic grader; otherwise three blind agents label, given the rubric and not the old labels or verdicts. A label is accepted only on unanimous agreement, and any disagreement goes to the owner as one intent question.
Fix
Each item below is adjudicated by the rule above; an agent then converts and re-scores whatever is accepted.
- 10 blocked panels (splits, one owned-content carve-out, one UNKNOWN grade) need owner adjudication.
- 1 verbatim candidate contradicts an earlier recorded adjudication.
- Panel results that disagree with recorded expectations for c23, c26 and c36 were recorded, not edited.
- Decide whether the verbatim target (min_n_per_class) should change. Changing it is outside attribution audit: grow the golden set past min_n_per_class 10 for verbatim and near-verbatim #5355's scope, and the 0.95 precision bar would rest on n=3 if lowered.
No class is fix-eligible.
Verification
Owner rulings are recorded on this issue. Any converted cases are re-scored over the whole set with per-class precision and n in the attribution CHANGELOG.
Related
#5355, #3465, plugins/attribution/CHANGELOG.md [0.8.2]
Done when
-
The provenance golden set is re-scored under copy rubric v5, and the per-class counts and the panels still blocked or disagreeing are posted here.
-
After the whole-set re-score that follows items 2 and 3, put near-verbatim fix-eligibility to the owner as one question. This asks and does not declare; attribution: run the disposition-applying provenance sweep once a class is fix-eligible #9755 and upstream-drift: record the provenance convention entry and add the cleaned-state CI gate #9759 stay cordoned in After zero until the owner rules.
Activity
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during triage.
Agent Brief
Type: Task
Summary: The owner rules on the open provenance golden-set items for the attribution audit; then an agent converts and re-scores whatever was ruled.
Current behavior:
The attribution golden set has 13 near-verbatim cases and 3 verbatim cases. Still open: 10 blocked panels (splits, one owned-content carve-out, one UNKNOWN grade), 1 verbatim candidate that contradicts an earlier adjudication, panel results for c23, c26 and c36 that disagree with their recorded expectations, and whether to lower the verbatim class's minimum per-class count.
Desired behavior:
Each open item has a ruling from the owner, recorded on this issue. Cases that were ruled are converted, and the whole set is re-scored, with per-class precision and n.
Acceptance criteria:
min_n_per_classtarget is recorded, including what the 0.95 precision bar rests on if it is loweredOut of scope:
Routing: human-gated (needs decision)
Each step is an owner adjudication, so this cannot be delegated until the rulings exist. Once they are recorded, the conversion and re-score can be briefed as a separate scoped (C3) task. I kept
needs-humanandstatus: needs-decision, addedpriority: medium, and removedneeds-triage.Basis: judgment. No other needs-triage item shares this decision.
kyle-sexton commentedon Oct 13, 2026
Decision recorded. The owner approved applying independently verified recommendations on 2026-10-12.
Basis: plugins/attribution/CHANGELOG.md:75-79 (0.10.4, fix(attribution): bump the copy rubric to v5 and mark pre-v5 scores invalid (split from #3465) #6194); fix(attribution): bump the copy rubric to v5 and mark pre-v5 scores invalid (split from #3465) #6194 closed
Basis: plugins/attribution/CHANGELOG.md:195-210 (verbatim n=3; no class fix-eligible); judgment for the threshold
The owner must rule on 10 blocked provenance panels, one contradicting verbatim candidate, the c23/c26/c36 disagreements, and whether to lower the verbatim min_n_per_class target. Since filing, attribution 0.10.4 bumped the copy rubric to version 5 and declared the 0.8.2 whole-set score these panels come from invalid, so the panels should be re-taken before the owner adjudicates them.
Verified by an independent fresh-context check (confirmed, risk low): gh issue view 5654 (1 triage comment, no owner ruling); origin/main plugins/attribution/CHANGELOG.md:71-79 (0.10.4: rubric v5, pre-v5 scores incl. 0.8.2 invalid, no re-score); CHANGELOG.md ~195-210 (0.8.2 table: verbatim n=3; near-verbatim 0.92; no class fix-eligible); CHANGELOG.md:350 (min_n_per_class=10); CHANGELOG headings 0.10.5 to 0.11.6: no golden-set re-score recorded; PR #6203 merged (rubric v5); gh pr list for a re-score: none; issue #6194 CLOSED, #3465 OPEN
Admission to an agent lane stays with the work-class rules and owner ratification.
kyle-sexton commentedon Oct 22, 2026
Parked (owner decision 2026-10-21): no eval spend until #7719 resumes hardening. This stays open and resumes then.
kyle-sexton commentedon Oct 24, 2026
Unparked by the owner's HITL session (2026-10-24): the owner's rule is that nothing stays parked unless it is unclear intent, a changed direction, a high security risk or an induced cost, and this hold was none of those.
Only the rubric-v5 re-score is unparked. The owner's adjudication of the blocked panels and the verbatim candidate stays with the owner (this body: "An agent must not resolve them"), so
needs-humanstays on for that step.The runs use the eval lane's MAINTENANCE role (
.github/lane-funding.json, rolemaintenance:keyed: false,MAINTENANCE_CLAUDE_CODE_OAUTH_TOKEN, a flat-fee subscription) or a local run on the operator machine's subscription; neither charges an auto-renewing key or a metered bill. The issue lane cannot run evals, so this goes to the eval session, not the lane.kyle-sexton commentedon Oct 26, 2026
Decision recorded.
This was generated by AI.
The owner decided this in the Backlog HITL session on 2026-10-26 (ledger row Q120 in
.work/backlog-to-zero/interview-checklist.md, a local, untracked file), answering: "go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles".Rule (extends Q114): adjudication first tries a deterministic grader. Otherwise three blind agents label, given the rubric and not the old labels or verdicts. A label is accepted only on unanimous agreement, and any disagreement goes to the owner as one intent question.
This replaces "An agent must not resolve them" in this issue. The body's Summary and Fix steps are edited to match; the rest is unchanged.
needs-humanis removed: the owner's adjudication was the hold, and what remains is the rubric-v5 re-score and the panel runs, which the host runner takes as agent work.62 remaining items
kyle-sexton commentedon Nov 2, 2026
Decision recorded. (owner, 2026-11-02 ~13:3xZ)
Question (Q-NV): may the attribution audit fix near-verbatim copies on its own now? Options: (a) yes, near-verbatim only; (b) not yet, sample more distinct sources; (c) never, keep report-only.
Owner, verbatim: "Lets plan on going with whatever this recommends:" (followed by the standing direction). The recommendation was (a).
Decision: (a). Near-verbatim becomes fix-eligible in the attribution audit; verbatim, paraphrase and restated-fact stay report-only. Basis for the recommendation: near-verbatim n 15, precision 1.00, recall 0.93, binding gate, 6 distinct origin pages (re-score). Caveats the owner accepted: 3 of 9 origin-bearing positives share one page, and the bar of 5 distinct pages was agent-set.
This answers this issue's second done-when item. The change that declares the class fix-eligible follows in a PR that closes this issue. #9755 and #9759 were cordoned until this ruling; the cordon lifts.
github-actions commentedon Nov 2, 2026
github-actions commentedon Nov 2, 2026
melodic-automation-lanes commentedon Nov 2, 2026
The work-resolve-issue lane stopped on this issue (
closed-by-open-pr): an open PR the lanes App or a listed human opened already closes the issue, a race with the earlier-work check, not a finding about the issue; while it stays open the next run stops before any model turn.It keeps
agent-ready, and work-unblock starts it again (retry 1 of 2).In the run's words:
Partial work is on the branch
work-resolve-issue/issue-5654-41456878571-1. The next run continues from it while it is not behind the default branch, and otherwise starts fresh.Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41456878571