Skip to content

docs(evals): move the short-TMPDIR example off the /run/user tmpfs (#11727) #12095

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kyle-sexton merged 2 commits into from
Nov 1, 2026
Merged

docs(evals): move the short-TMPDIR example off the /run/user tmpfs (#11727) #12095

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f3a158
docs(evals): keep the eval TMPDIR on disk, not the /run/user tmpfs
kyle-sexton Nov 1, 2026
e89c829
Merge branch 'main' into host-run/11727-eval-validity
kyle-sexton Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changes/evals/docs-11727-eval-tmpdir-on-disk-4c1e7a90.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
bump: patch
---

### Fixed

- `plugin-eval`'s short-`TMPDIR` advice now points at a directory on disk (`$HOME/.ev` when that
stays within 24 characters) instead of `/run/user/<uid>/ev`, and says why: that tmpfs is small,
and kept `--keep-temp` sandboxes can fill it, failing every eval on the host and breaking the
session sockets stored there
([#11727](https://github.com/melodic-software/claude-code-plugins/issues/11727)).
2 changes: 1 addition & 1 deletion plugins/evals/skills/plugin-eval/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ a committed `mocks/.replay/` so agent mocks replay without a model call.
script first, since it runs as you outside the sandbox (see Gotchas), then add `--scaffold`.
Without it those cases run against an unstaged workspace, both arms score the same, and their
delta reads 0, so that share of the pass is wasted. When a script is one you would not run on
your own machine, select the other cases with `--case <glob>` or `--tag <tag>`. On Linux and WSL2, run the CLI as `env TMPDIR=<dir> claude plugin eval …`, where `<dir>` is a directory of at most 24 characters outside the system temp directories that you create first (for example `/run/user/$(id -u)/ev`). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start).
your own machine, select the other cases with `--case <glob>` or `--tag <tag>`. On Linux and WSL2, run the CLI as `env TMPDIR=<dir> claude plugin eval …`, where `<dir>` is a directory of at most 24 characters outside the system temp directories that you create first, on disk (for example `$HOME/.ev` when that stays within 24 characters; not the `/run/user` tmpfs, which kept sandboxes can fill). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start).

Without `--trust-plugin` a non-TTY run against an untrusted directory is refused with exit 1.
`--keep-temp` keeps each run's trace for the validity gate under "Reading the delta". A run
Expand Down
7 changes: 5 additions & 2 deletions plugins/evals/skills/plugin-eval/reference/sandboxes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,11 @@ python3 "<skill-dir>/scripts/docker-links.py" run -- env TMPDIR=<dir> claude plu
```

`<dir>` is the run step's directory: at most 24 characters, outside the system temp directories,
created first (for example `/run/user/<uid>/ev`). A longer one is untested, and at 63 characters
no run started a shell sandbox ([Sandbox start](#sandbox-start)).
created first, on disk (for example `$HOME/.ev` when that path stays within 24 characters). A
longer one is untested, and at 63 characters no run started a shell sandbox
([Sandbox start](#sandbox-start)). Not `/run/user/<uid>`: it is a small tmpfs that kept
`--keep-temp` sandboxes can fill, which fails every eval on the host with ENOSPC and breaks the
session sockets stored there.

It changes `~/.docker` outside the repository, so it runs only after the user agrees. It replaces
every link the preflight's `find` lists, or `~/.docker` when that is itself a link, with a copy of
Expand Down
Loading