fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns

Observed

Two outcome-pass shapes come back INVALID under run-validity.py for reasons the case cannot fix. This was measured during the #6584 and #6585 rollout passes on 2026-10-31, at Claude Code 2.1.316.

A user-scope read the skill's contract requires is denied. improvement:improve resolves standards through a ladder whose user rung is ~/.claude/standards. The eval sandbox denies that Glob in both arms and both versions, so every improve outcome pass on Opus was INVALID: 4 passes plus a targeted rerun. Concise skills rollout: improvement (3 skills, 2 model-invoked) #6584 therefore shipped main's text and named improve unmeasured on Opus (PR test(improvement): eval cases, trigger probes and graders for find and improve #11719).
A slash-invoked context: fork skill reports 0 turns. The /mcp-tools:audit with-arm runs ran the fork fully, as the trace shows, but the result records turns: 0, so run-validity.py fails them as "never started". The Concise skills rollout: mcp-tools (2 skills, 2 model-invoked) #6585 cases worked around this by naming the skill in the prompt instead of slash-invoking it.

#11624 covers denials in invocation-probe (trigger) runs. These two are outcome runs.

Fix wanted

For 1, one of these, chosen on evidence:
- the harness grants read access to the user-scope paths a case declares;
- run-validity.py warns, rather than fails, on a denial that appears equally in both arms and both versions;
- the case design documents a workspace-level stand-in for the user rung.
For 2: run-validity.py, or the result writer, counts the forked session's turns, so a completed fork is not "never started".

Done when

An improvement improve outcome pass on Opus returns VALID, or only warnings, with the ladder's user rung reachable or handled.
A slash-invoked context: fork case that completes is not failed as "never started".

This was generated by AI.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none

Decision recorded. Both gate fixes the body asks for are on main through #11787. Other merged work covers the later comments' turn-cap, skill-fired and plugin-directory items. What is left is a host run of claude plugin eval, to prove both Done-when items on real passes and, only if one still occurs, to capture a real timeout row so the probe timeout rule can be written against it.

[quote truncated; the full decision is at the link]

#11727 (comment)

The decision settles the open questions. Both gate fixes (the user-scope denial warning and the forked-skill 0-turn check) are already on main. What remains is a host run of claude plugin eval to confirm both Done-when items on real passes. That is a step an unattended lane can't run, so it goes to the host-run route and doesn't hold this issue.

_{Automated triage, applied by the lanes App.}

No description provided.

Two more cases from the #6585 trigger measurement (2026-11-01, Claude Code 2.1.316):

A skill's Glob of its own plugin directory is denied in trigger passes. In mcp-tools:audit-posture positive trigger runs, the with-arm Glob of the plugin copy's root is denied in both versions. That makes every positive pass INVALID, so the description could not be measured and kept main's text (Concise skills rollout: mcp-tools (2 skills, 2 model-invoked) #6585). Granting Read, Glob, Grep in the emitted cases did not help, because the denial comes from the sandbox's path scope, not from the tool grant.
run-validity.py's skill fired check fails any trigger pass with a miss. In a trigger-measurement pass a miss is the measured quantity, so read literally, only a 100% fire rate can ever be VALID. Lane 2 skipped that one check for trigger passes, recorded before reading any number, with every other check kept. The gate should apply it only to outcome cases. check_fired, around lines 757-810 of run-validity.py.

This was generated by AI.

Two more validity gaps seen in the lane-2 trigger passes (#6580 and #6582):

turns is not a turn-cap signal. In the Concise skills rollout: github (3 skills, 2 model-invoked) #6582 smoke (frozen train probes, max_turns: 10), the result rows reported turns 10, 10, 9 and 16. All 4 runs ended with terminal_reason: completed and had no max-turns error. A validity rule written as turns >= max_turns marks runs that completed normally as INVALID. Detect the cap from the error text "Reached maximum number of turns", or from a terminal reason that names max_turns.
Skills that fan out work exceed small turn budgets. In Concise skills rollout: event-storming (2 skills, 2 model-invoked) #6580, event-storming:simulation fired in every positive run of one split and then spawned 7 to 17 persona agents. A 10-turn case budget can't hold that. A trigger pass is graded at the Skill call, so a turn cap reached after that call should not invalidate the run. Concise skills rollout: github (3 skills, 2 model-invoked) #6582 onward record this rule before any pass.

Effect on #6580: simulation stayed on main's description. Its rewrite also fell under the 100-character cutoff, so the decision would not change. #11831 shipped the methodology description.

No description provided.

Host evidence for body item 1 (lane 1, from the #11628 re-measure, 2026-11-01, Claude Code 2.1.316). A user-scope read the skill requires still makes outcome passes INVALID when it is denied in only some runs, which the in_every_run rule (run-validity.py:888-906) does not cover by design.

Skill: visualization:visualize reads ~/.claude/rendered-views.md; the eval sandbox denies it.
Outcome passes, --runs 3, both arms, --disallow-tools Glob Grep Agent:

Pass	Runs with the denied Read	run-validity.py
before, Opus	1	INVALID
before, Sonnet	3	INVALID
after, Opus	2	INVALID
after, Sonnet	(warning only)	VALID

So the denial fires in some runs and not others (the skill reads the file only on some paths), and in_every_run only downgrades a denial seen in every run of both arms. Result rows and traces: lane-1 host run dir cc-plugins-11628-evalrun/out/ (not committed). This bears on step 1's Done-when: an outcome pass of a skill with a conditional user-scope read stays INVALID unless every case has denial-free runs.

Lane 1 holds this issue; the remaining host runs (steps 1-4) are paused while this host's 7-day window is at 90% (Q110). The /run/user/<uid>/ev TMPDIR example the decision comment notes (SKILL.md:246, reference/sandboxes.md:117) gets a docs fix on its own branch.

Released by lane 1: #11727 moves whole to the HITL Backlog session (owner, relayed by the hub). Hand-over: branch docs/11727-eval-tmpdir-on-disk @ c1f89fe (pushed, no PR) moves the plugin-eval TMPDIR example off /run/user to disk; the host evidence for step 1 is the comment above.

Another validity gap, from the #6593 (verification) measurement.

The runbook transform in plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md step 5 sets allowed_tools: [Skill] on trigger cases to stop negative runs from browsing. It does not remove the tools from the session: the init tool list in the negative traces still offers Read, Glob and Grep. Negative runs still try to read outside the sandbox and are denied before their Skill decision.

Effect: under a plan that counts such a denial as invalidating, every negative pass for #6593 was INVALID in both versions (8 of 8), so measure stayed unmeasured and kept main. Plans for #6586, #6591 and #6592 carry an exemption (a pre-decision out-of-sandbox read denial does not invalidate) and measured cleanly.

Options:

Withhold the tools for real (a working disallowed-tools setting on the eval session), then drop the exemption.
Keep the transform and make the exemption part of the runbook's validity rules.

Recommended: 1, with 2 as the interim rule. Basis: the 6593 negative traces (init tool list) and the 6586 pass where the exemption applied.

Follow-up to the Skill-only transform gap, from the #6589 (prototype) measurement.

The transform also leaves Task/Agent in the session (init tool list: Task, Glob, Grep, Read, Skill, TaskStop, ToolSearch). Once prototype:explore-directions fires, the model dispatches general-purpose subagents to build the layout variants, so a trigger run keeps working long after the graded Skill decision.

Measured (passes.log and traces, 2026-11-01):

explore-directions positive passes took 24-32 min each (12 runs, serial) against 3-5 min for its negatives and 3.5-9 min for pressure-test.
8 of 43 explore-directions positive runs hit the 300 s timeout, all after the Skill call.
Positive passes cost 3.2-5.0 USD per 12 runs; one hit the 5 USD per-pass ceiling and returned 7 of 12 rows (partial: true), which voids the pass and forces a rerun split smaller in both versions.

The graded event is the Skill call, so nothing after it adds signal. Options:

Withhold Task/Agent (and the read tools) from trigger-probe sessions for real.
End the run right after the first Skill call (a max-turns cap tight enough to stop after the call).

Recommended: 1, which also fixes the read-denial gap above; 2 as a cheap interim. Basis: the 6589 traces (before-explore-directions-p2 pos-val-05, before-explore-directions-p1 pos-val-01) and passes.log wall times.

Claimed by the Backlog HITL host-run session. The hub moved #11727 here from lane 1, whose host runs never started; lane 1 released it at 16:07Z.

Building on lane 1's record: its branch docs/11727-eval-tmpdir-on-disk (c1f89fe) is carried on host-run/11727-eval-validity, and its 15:13Z host evidence for body item 1 is the starting point. Running the decision comment's host steps 1-4 on Claude Code 2.1.316: the improve outcome pass on Opus, a slash-invoked context: fork case, and one invocation-probe suite on the README run line for the timeout row and the init tool list. Results go in a host-run record on this issue.

This was generated by AI.

Host run. Three claude plugin eval suites on Claude Code 2.1.316 covered the decision comment's host steps 1-4: an improvement improve outcome pass on Opus, a slash-invoked context: fork case, and an invocation-probe suite on the README run line. Both Done-when items are met with no code change.

What ran

CLI: claude --version = 2.1.316 (Claude Code). Model under test claude-opus-5-5 (from the traces). Judge --judge-model opus on the outcome suites.
Every suite ran in its own capped systemd-run --user --scope with TMPDIR=/home/kyle/worktrees/.ev11727. Each ran against a copy of the plugin outside any git repository. A first attempt against the worktree failed every run with a git directory of the plugin registers more linked worktrees than can be screened. The copy is the documented workaround (plugin-eval SKILL.md:130-134).
Validity gate: run-validity.py from origin/main e09dba5.

Suite	Command, short form	n	Gate verdict
1. improve on Opus	`plugins/improvement --tag improve --model opus --scaffold --keep-temp -j 2`, default ablation	4 cases x 3 runs x 2 arms = 24	VALID, warnings only
2. slash fork	mcp-tools copy plus one scratch case `au-slash-fork` (the `au-report-edits-nothing` contract, prompt `/mcp-tools:audit servers/tickets`), `--ablation none --allow-tools Bash --scaffold --keep-temp`, `HOME` set to an empty directory, `CLAUDE_CONFIG_DIR` kept	1 case x 3 runs	`session started` PASS; overall INVALID only on the copied `Skill` grader (see below)
3+4. probes	4 scratch probes for `prototype:explore-directions` (3 should fire, 1 should not), `emit-plugin-eval`, then the README line: `--ablation none --keep-temp --disallow-tools Read Glob Grep Agent -j 2`	4 cases x 3 runs = 12	VALID, warnings: models (no judge recorded, none used)

Results against each item

Done-when 1, an improve outcome pass on Opus returns VALID or warnings only: met.
- Gate verdict: VALID (warnings: permission denials in case imp-dry-run-proposes-diff, imp-meets-standard-no-change, skill fired, ceiling).
- The user-rung denial Glob ~/.claude/standards occurred in 3 of 12 with-arm runs (dry-run run 3, meets-standard runs 2 and 3) and in 0 without-arm runs. It came back as a warning, not a failure: each denied run scored the same as its case's denial-free runs (same_as_clean).
- Suite: 4 of 4 cases passed, overall score 1.00, mean delta +0.28. The delta is +0.37 over the 3 cases whose without-arm is under the ceiling. Duration 345 s.
- Lane 1's 15:13Z table (visualization:visualize) shows the remaining gap: a conditional user-scope read stays INVALID when a denied run scores differently from its clean siblings. in_every_run does not cover that case by design. It did not occur here, so nothing changes in this item.
Done-when 2, a slash-invoked context: fork case that completes is not failed as "never started": met.
- check session started: PASS (... 3 runs at 0 turns ran in a forked session its trace holds). All 3 runs recorded turns: 0, and their traces held 7, 8 and 8 assistant messages.
- Trace shape: top-level type: assistant lines, the same shape assistant_messages() counts. An init line is followed by a user line carrying the loaded skill body, then the assistant/tool lines, then result success with num_turns 0. So the shape fix(evals): warn on denials every run of both arms shares and pass 0-turn forked sessions in run-validity #11787 assumed matches a real fork trace, and assistant_messages() needs no change.
- The suite came back INVALID only because the copied audit-skill-fired grader (tool_used: Skill, min 1) failed in 3 of 3 runs. A slash command loads the skill without a Skill tool call, which is already documented at reference/case-authoring.md:77-82. This is an artifact of the scratch case, not a gate defect. Every outcome grader passed in all 3 runs (C18 FAIL reported, source unchanged, no write or edit).
- Bash ran in the sandbox (tool results carried output), so the 29-character TMPDIR started a shell.
Step 3, a timeout after the Skill call on the README run line: did not occur, so no rule is needed.
- 0 of 12 probe runs timed out (case timeout 300 s, max_turns 8). Each run took 2-3 turns and 13-27 s.
- The skill fired in 9 of 9 should-fire runs and stayed quiet in 3 of 3 should-not-fire runs.
- With Read, Glob, Grep and Agent withheld and no Bash or Write granted, the skill cannot start the mockup build that timed out in Concise skills rollout: prototype (2 skills, 2 model-invoked) #6589/Concise skills rollout: verification (3 skills, 2 model-invoked) #6593. No real timeout row exists to write a check_errors rule against, so per the decision comment none was added.
Step 4, the init tool list of an emitted probe run: met.
- In all 12 traces the init line lists exactly Skill, TaskStop, ToolSearch. None of Read, Glob, Grep, Agent or Task is present.
- check probe tools: PASS (no invocation-probe run could call Read, Glob, Grep or Agent), and there were 0 denied calls in 12 traces.
- So --disallow-tools does remove the tools under claude plugin eval. That answers option 1 of the 17:40Z comment: the working disallowed-tools setting already exists. A runbook that still sets only allowed_tools: [Skill] should switch to the README line.

Side findings

The 14:07Z TMPDIR-leak hypothesis (UNVERIFIED there) got no support. These runs used a TMPDIR under the home directory. The only out-of-sandbox denial was the skill's own ~/.claude/standards rung, in with-arm runs only, and there were no pre-decision home or .claude Glob denials. The likelier cause of Concise skills rollout: harness-memory (2 skills, 2 model-invoked) #6583's denials is still that its sessions could call Glob (decision answer 3).
The first check-affected.sh --tests failed validate-plugins (exit 1). That ran while an eval suite was running, and every test it printed passed. validate-plugins.sh --only evals alone and a second full check-affected --tests both passed (14 passed, 2 skipped). The cause is UNVERIFIED (likely load); reported here, not filed.

Branch

host-run/11727-eval-validity at 2f3a158. It carries lane 1's docs/11727-eval-tmpdir-on-disk commit (c1f89fe, cherry-picked clean onto origin/main e09dba5): the plugin-eval TMPDIR example moves off the /run/user tmpfs, with a changelog fragment. No other code change was needed.
CHECK_DATA_ROOT=<wt> scripts/check-affected.sh --tests on that head: 14 passed, 2 skipped.
The PR is opened by the coordinator.

Target labels: add none; remove host-run.

Navigation Menu

Repository navigation

fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727

Observed

Fix wanted

Done when

13 remaining items

What ran

Results against each item

Side findings

Branch

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727

Description

Observed

Fix wanted

Done when

Activity

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

13 remaining items

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

What ran

Results against each item

Side findings

Branch

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions