From 2f3a158fe3d4173122f4bb85f689e1cfa9ae11e3 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 1 Nov 2026 10:13:52 -0500 Subject: [PATCH] docs(evals): keep the eval TMPDIR on disk, not the /run/user tmpfs The short-TMPDIR example named /run/user//ev, a small tmpfs that kept --keep-temp sandboxes filled to 100%, failing every eval on the host with ENOSPC and breaking the session sockets stored there. Point at a short disk directory instead and say why. Refs: #11727 Co-Authored-By: Claude Opus 5.5 --- .../evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md | 11 +++++++++++ plugins/evals/skills/plugin-eval/SKILL.md | 2 +- .../evals/skills/plugin-eval/reference/sandboxes.md | 7 +++++-- 3 files changed, 17 insertions(+), 3 deletions(-) create mode 100644 .changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md diff --git a/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md b/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md new file mode 100644 index 00000000000..9afadded365 --- /dev/null +++ b/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md @@ -0,0 +1,11 @@ +--- +bump: patch +--- + +### Fixed + +- `plugin-eval`'s short-`TMPDIR` advice now points at a directory on disk (`$HOME/.ev` when that + stays within 24 characters) instead of `/run/user//ev`, and says why: that tmpfs is small, + and kept `--keep-temp` sandboxes can fill it, failing every eval on the host and breaking the + session sockets stored there + ([#11727](https://github.com/melodic-software/claude-code-plugins/issues/11727)). diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 10699649a99..0b5531efbe1 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -243,7 +243,7 @@ a committed `mocks/.replay/` so agent mocks replay without a model call. script first, since it runs as you outside the sandbox (see Gotchas), then add `--scaffold`. Without it those cases run against an unstaged workspace, both arms score the same, and their delta reads 0, so that share of the pass is wasted. When a script is one you would not run on - your own machine, select the other cases with `--case ` or `--tag `. On Linux and WSL2, run the CLI as `env TMPDIR= claude plugin eval …`, where `` is a directory of at most 24 characters outside the system temp directories that you create first (for example `/run/user/$(id -u)/ev`). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). + your own machine, select the other cases with `--case ` or `--tag `. On Linux and WSL2, run the CLI as `env TMPDIR= claude plugin eval …`, where `` is a directory of at most 24 characters outside the system temp directories that you create first, on disk (for example `$HOME/.ev` when that stays within 24 characters; not the `/run/user` tmpfs, which kept sandboxes can fill). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). Without `--trust-plugin` a non-TTY run against an untrusted directory is refused with exit 1. `--keep-temp` keeps each run's trace for the validity gate under "Reading the delta". A run diff --git a/plugins/evals/skills/plugin-eval/reference/sandboxes.md b/plugins/evals/skills/plugin-eval/reference/sandboxes.md index 10b25a7b8e9..f3a2fa6b2f7 100644 --- a/plugins/evals/skills/plugin-eval/reference/sandboxes.md +++ b/plugins/evals/skills/plugin-eval/reference/sandboxes.md @@ -114,8 +114,11 @@ python3 "/scripts/docker-links.py" run -- env TMPDIR= claude plu ``` `` is the run step's directory: at most 24 characters, outside the system temp directories, -created first (for example `/run/user//ev`). A longer one is untested, and at 63 characters -no run started a shell sandbox ([Sandbox start](#sandbox-start)). +created first, on disk (for example `$HOME/.ev` when that path stays within 24 characters). A +longer one is untested, and at 63 characters no run started a shell sandbox +([Sandbox start](#sandbox-start)). Not `/run/user/`: it is a small tmpfs that kept +`--keep-temp` sandboxes can fill, which fails every eval on the host with ENOSPC and breaks the +session sockets stored there. It changes `~/.docker` outside the repository, so it runs only after the user agrees. It replaces every link the preflight's `find` lists, or `~/.docker` when that is itself a link, with a copy of