Repository navigation
harness-config: outcome eval cases and probes for audit-permission-grants, audit-permission-state, draft-auto-mode-rules (rollout slice B) #11561
Description
Parent
Depends on
- none: on the frontier now
Outcome
harness-config gains frozen outcome eval cases and probe files for audit-permission-grants, audit-permission-state, draft-auto-mode-rules, plus a recorded before-state, so rewrite slice B can be measured against them.
Demo: /evals:validate passes on the new cases, and the smoke-pass record shows each case run against the current (before) skill text.
Decisions
- Q93/Q94: the harness-config rollout (Concise skills rollout: harness-config (10 skills, 9 model-invoked) #6555) is split into sub-issues with blocked-by edges before any run: outcome eval cases first, then rewrite slices of a few skills each.
- Eval passes run on the Claude subscription; that usage is a capacity limit, not owner cost, so no spend approval gates them.
Key interfaces
Steps 1-3 of the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). Case directories carry a short per-skill tag prefix; shared fixtures live under the plugin's eval fixtures directory and are named by a scaffold or seed script.
Done when
- Each of
audit-permission-grants,audit-permission-state,draft-auto-mode-ruleshas outcome cases written by a fresh agent from the before contract via/evals:design, and/evals:validatepasses on them. - Each
llmgrader is calibrated as/evals:plugin-eval"Calibrating a judge" says. - One smoke pass ran on the before version; each failure was classified (real, grader defect, case defect) by a fresh agent and grader or case defects fixed.
- Each skill has a frozen probe set in the plugin's
probes/<skill>.json, checked with/skill-quality:check measure-invocation. - The PR records a digest of these case directories (frozen; unchanged after the first after-version run) and the
/skill-quality:checkbefore-state output for these skills. - No skill body or description changes in this PR.
Out of scope
- Rewriting any skill (rewrite slice B).
- Cases for skills outside this slice.
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: add frozen outcome eval cases and probe files for the three harness-config skills (
audit-permission-grants,audit-permission-state,draft-auto-mode-rules). The PR should also record a before-state, with no skill body or description changes. The issue states its outcome, done-when items and out-of-scope list clearly, and it has no dependencies. Some steps, such as the smoke pass and judge calibration, may need a host-run rather than this lane. No likely duplicates were found among the open issues.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 31, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 31, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 1, 2026