[kyle-sexton]

Closes #11727

Summary

Host runs on Claude Code 2.1.316 show that #11787, as merged, meets both done-when items of #11727:

• An improve outcome pass on Opus returns VALID, with the ~/.claude/standards denial reported as a warning.
• A slash-invoked context: fork case passes session started.

This PR carries the remaining docs fix from laptop Lane 1.

Fix

• plugin-eval SKILL.md and reference/sandboxes.md: the short-TMPDIR example moves off the /run/user tmpfs to a short on-disk directory. On the tmpfs, sandboxes kept by --keep-temp filled it and runs failed with ENOSPC.
• Changelog fragment in .changes/evals/.

Verification

• improve on Opus, 4 cases x 3 runs x 2 arms, with --scaffold --keep-temp:
  • run-validity.py VALID, with warnings for permission denials, skill fired and ceiling.
  • Score 1.00, delta +0.28.
• Slash-invoked /mcp-tools:audit fork case, 3 runs: session started PASS, with 0 turns and 7-8 assistant messages in each trace.
• emit-plugin-eval probes on the README run line, 12 runs:
  • VALID, 0 timeouts.
  • The init tool list was Skill, TaskStop, ToolSearch, and probe tools PASS.
• CHECK_DATA_ROOT=<wt> scripts/check-affected.sh --tests: 14 passed, 2 skipped.
• The full record is posted on fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727.

Related

• fix(evals): warn on denials every run of both arms shares and pass 0-turn forked sessions in run-validity #11787 (the gate fixes), fix(evals): run invocation probes without subagents and name the run line in run-validity #11901 (Agent added to --disallow-tools), Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (rollout umbrella)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T20:42:25.774933Z	e89c829	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed e89c829d048db3b0143c11a8b96239d3ccb82809: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12095 (head e89c829)

No security issues found.

I read every hunk in all 3 changed files. None of them match a trusted-path glob.

• plugins/evals/skills/plugin-eval/SKILL.md: the example short TMPDIR moves from /run/user/$(id -u)/ev to an on-disk directory ($HOME/.ev when that path stays within 24 characters). The new text explains why: kept --keep-temp sandboxes can fill the /run/user tmpfs.
• plugins/evals/skills/plugin-eval/reference/sandboxes.md: the same example change, plus the reason (ENOSPC on the tmpfs, which also breaks the session sockets stored there).
• .changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md: a changelog fragment only.

What I checked:

• Instruction surface: the skill text loses no guardrail and no rule is narrowed. The 24-character limit, the "outside the system temp directories" rule, "create it first", and the --trust-plugin and user-agreement steps for the Docker link swap are all still there. Only the example path and its reason changed. I did not consult the protected-class register because no rule was removed.
• Trust controls: none touched. No workflow, action, token request, kill switch, merge hold or trusted-actor file is in the diff.
• Plugin manifests: none changed.
• Sandbox exposure: the new TMPDIR is a dedicated subdirectory the user creates, and it is passed only to the user's local claude plugin eval run. No attacker-controlled input reaches it, and it does not widen what the eval sandbox may write. A closing-issue comment suggests a TMPDIR under $HOME might expose the home path to runs. That comment marks itself UNVERIFIED, and it is about whether eval runs stay valid, not a leak of a token, secret or key. So it is not a security finding.
• PR context: I found no embedded directives aimed at the lane.

critical-findings: 0

[github-actions]

claude-review reviewed e89c829d048db3b0143c11a8b96239d3ccb82809: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

I read the whole change in all 3 files. It changes only documentation. In plugin-eval, the short-TMPDIR example moves from /run/user/<uid>/ev to a directory on disk ($HOME/.ev when that stays within 24 characters), and the text now says why: kept --keep-temp sandboxes can fill the /run/user tmpfs and cause ENOSPC. A patch changelog fragment comes with it.

What I checked

• The two docs agree. SKILL.md:246 and reference/sandboxes.md:116-121 now give the same advice. Both keep the 24-character limit and the "outside the system temp directories" rule. $HOME/.ev fits under 24 characters whenever the username is 14 characters or fewer. The "when that stays within 24 characters" wording covers longer usernames, and preflight's eval_tmpdir refusal still enforces the limit.
• Changelog fragment. .changes/evals/… uses bump: patch, which fits a docs fix. Its text matches the diff.
• Closing issue fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727.
  • Done-when 1 (an improve outcome pass on Opus returns VALID or only warnings) is met by fix(evals): warn on denials every run of both arms shares and pass 0-turn forked sessions in run-validity #11787's gate change. test_run_validity.py:996 (test_a_denial_in_every_run_of_both_arms_is_a_warning) would fail if that broke.
  • Done-when 2 (a slash-invoked fork case is not failed as "never started") is met by check_started. test_run_validity.py:418 (test_a_zero_turn_run_whose_trace_holds_a_forked_session_passes) would fail if that broke.
  • The host-run results in the PR body (VALID, score 1.00, delta +0.28, session started PASS) are claims by the author. I did not run them.
• Related, not a finding. An issue comment (14:07Z, item 3, marked UNVERIFIED) suggested a TMPDIR inside the user's home might expose home paths to runs, and proposed /var/tmp/<short> instead. The later decision comment traced those denials to runs missing --disallow-tools. The host run with TMPDIR under /home/… came back VALID. Also, /var/tmp is a system temp directory, which these docs already say to avoid. So I'm not flagging the $HOME example.

Other notes

• No plugin manifest changed, so there are no manifest lines.
• No closing-issue text was embedded as instructions.
• The prior-findings file is empty (0 rounds), so the round cap does not apply.
• Nothing in this change spends money.