diff --git a/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md b/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md new file mode 100644 index 00000000000..9afadded365 --- /dev/null +++ b/.changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md @@ -0,0 +1,11 @@ +--- +bump: patch +--- + +### Fixed + +- `plugin-eval`'s short-`TMPDIR` advice now points at a directory on disk (`$HOME/.ev` when that + stays within 24 characters) instead of `/run/user//ev`, and says why: that tmpfs is small, + and kept `--keep-temp` sandboxes can fill it, failing every eval on the host and breaking the + session sockets stored there + ([#11727](https://github.com/melodic-software/claude-code-plugins/issues/11727)). diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 10699649a99..0b5531efbe1 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -243,7 +243,7 @@ a committed `mocks/.replay/` so agent mocks replay without a model call. script first, since it runs as you outside the sandbox (see Gotchas), then add `--scaffold`. Without it those cases run against an unstaged workspace, both arms score the same, and their delta reads 0, so that share of the pass is wasted. When a script is one you would not run on - your own machine, select the other cases with `--case ` or `--tag `. On Linux and WSL2, run the CLI as `env TMPDIR= claude plugin eval …`, where `` is a directory of at most 24 characters outside the system temp directories that you create first (for example `/run/user/$(id -u)/ev`). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). + your own machine, select the other cases with `--case ` or `--tag `. On Linux and WSL2, run the CLI as `env TMPDIR= claude plugin eval …`, where `` is a directory of at most 24 characters outside the system temp directories that you create first, on disk (for example `$HOME/.ev` when that stays within 24 characters; not the `/run/user` tmpfs, which kept sandboxes can fill). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). Without `--trust-plugin` a non-TTY run against an untrusted directory is refused with exit 1. `--keep-temp` keeps each run's trace for the validity gate under "Reading the delta". A run diff --git a/plugins/evals/skills/plugin-eval/reference/sandboxes.md b/plugins/evals/skills/plugin-eval/reference/sandboxes.md index 10b25a7b8e9..f3a2fa6b2f7 100644 --- a/plugins/evals/skills/plugin-eval/reference/sandboxes.md +++ b/plugins/evals/skills/plugin-eval/reference/sandboxes.md @@ -114,8 +114,11 @@ python3 "/scripts/docker-links.py" run -- env TMPDIR= claude plu ``` `` is the run step's directory: at most 24 characters, outside the system temp directories, -created first (for example `/run/user//ev`). A longer one is untested, and at 63 characters -no run started a shell sandbox ([Sandbox start](#sandbox-start)). +created first, on disk (for example `$HOME/.ev` when that path stays within 24 characters). A +longer one is untested, and at 63 characters no run started a shell sandbox +([Sandbox start](#sandbox-start)). Not `/run/user/`: it is a small tmpfs that kept +`--keep-temp` sandboxes can fill, which fails every eval on the host with ENOSPC and breaks the +session sockets stored there. It changes `~/.docker` outside the repository, so it runs only after the user agrees. It replaces every link the preflight's `find` lists, or `~/.docker` when that is itself a link, with a copy of