Repository navigation
harness-config: concise rewrite of audit, audit-pass, audit-instructions (rollout slice A) #11564
Description
Parent
Depends on
Outcome
audit, audit-pass, audit-instructions meet the skill criteria, with every shipped description and body change backed by a valid measured pass against the frozen cases from #11560.
Demo: the PR body's per-skill decision table, with the before/after numbers behind each row.
Decisions
- Q93/Q94: the harness-config rollout (Concise skills rollout: harness-config (10 skills, 9 model-invoked) #6555) is split into sub-issues with blocked-by edges before any run: outcome eval cases first, then rewrite slices of a few skills each.
- Eval passes run on the Claude subscription; that usage is a capacity limit, not owner cost, so no spend approval gates them.
Key interfaces
Steps 4-8 of the per-plugin runbook; the skill criteria and its measurement protocol decide what ships. The rewriter's brief forbids reading any probe file.
Done when
- Before and after versions are measured together on each target model, with and without the plugin, same
--runs, judge and threshold,--keep-temp; held-out triggering probes run viameasure-invocation emit-plugin-evalwith--ablation none. -
/evals:plugin-evaljudges each result VALID, and anycase dropis restored or the before text ships. -
/docs-hygiene:compress compareran with two independent labelers; the union of losses is resolved. - The PR body carries a per-skill decision table (description and body decided separately) with its numbers; any skill left unmeasured is named with the reason.
-
/skill-quality:checkon the plugin root is no worse than the before-state from harness-config: outcome eval cases and probes for audit, audit-pass, audit-instructions (rollout slice A) #11560, and the plugin's changelog fragment is added.
Out of scope
- Skills outside this slice; changes to the frozen cases from harness-config: outcome eval cases and probes for audit, audit-pass, audit-instructions (rollout slice A) #11560.
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #11560
Understood: rewrite the
audit,audit-passandaudit-instructionsskills in harness-config more concisely (rollout slice A). Each shipped change must be backed by a valid measured eval pass against the frozen cases from #11560. The PR body should carry a per-skill decision table. The issue names #11560 as its dependency, and that issue is still open. Some done-when items (measured eval passes,/evals:plugin-evaljudging) may need a host-run. Admission routes those. No duplicates found.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 31, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 31, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 1, 2026
docs(harness-config): keep audit-pass and audit-instructions at main'…
kyle-sexton commentedon Nov 1, 2026
Lane 1 progress: the
auditrewrite merged in #12105. All 4 of its cases are VALID before and after, and it is non-inferior.audit-passandaudit-instructionskeep main's text for now. Both save state, and their state-saving cases (ap-config-lane-run-state,ai-skills-scope-report-only) had no VALID number in either version. The cause is plugin-data writes through a shell variable, which the validity gate can't set aside. Lane rule R16 lets a state-mutating skill ship a rewrite only on full before/after evidence.Remaining: re-run both skills once #12078 makes those paths measurable. The rewrite and its meaning diff are done, kept on branch
feat/11564-harness-config-concise-aat commit aa06d4c, which still holds both rewrites.kyle-sexton commentedon Nov 1, 2026
Lane 1: re-measured on main after #12090. Both state-saving cases are still INVALID before the rewrite. 4 of the 5 counted denials reach plugin data through a shell variable assigned in the same command, which
run-validity.pycan't resolve. Soaudit-passandaudit-instructionskeep main's text, and this issue is blocked on #12156.The finished rewrite is kept: commit 53d326f on branch
feat/11564-concise-audit-pass-instructions, local and unpushed, composed with #12090 and passing the meaning diff. The next attempt should also writeaudit-pass/SKILL.md:122-125as one literal call.docs(harness-config): write audit-pass's prescribed script calls as l…
kyle-sexton commentedon Nov 2, 2026
Lane 1: closing with both remaining skills kept at main's text, as the Done-when allows ("any skill left unmeasured is named with the reason"; "the before text ships").
Re-measured 2026-11-02, before = main at 444ef9e and after = branch
feat/11564-concise-audit-pass-instructionsat dd88c72. The after version has main merged in, and its prescribed script calls are written as literal commands. Setup: Claude Code 2.1.316, claude-opus-5-5, judge opus,--runs 3,--keep-temp, with and without the plugin. The meaning diff was PASS/PASS for both skills. No drift: the without-arm change was +0.00.Only 2 cases are VALID in both versions, so the pooled compare is
n too small to call.The run-validity fix in #12225 (#12156) changes no verdict here, so this decision doesn't wait for it to merge. The denials that remain are compound commands the model composes around the prescribed
run-state.shcalls. When the plugin call was retried alone, it was allowed. Whether such a denial should count is now #12256. The rewrite stays parked on the branch above; it can be re-measured if #12256 changes the rule.auditshipped in #12105.