Add Pool harness support for Laguna S 2.1

[krlvi]

What this adds

• Adds Pool as an opt-in benchmark agent and pins Laguna to poolside/laguna-s-2.1.
• Parses Pool transcripts and native trajectories, verifies the observed model, and reconciles version-control calls against the harness command trace.
• Gives each Pool run an isolated home, generated settings, a credential proxy, and a macOS host sandbox.
• Preserves rate-limit attempts and records the batch inputs needed for a reproducible resume.

Prerequisites

• macOS with sandbox-exec (the Pool host boundary is enabled by default).
• Node.js 20+ and Git.
• Pool CLI on PATH, or pass --pool-bin; this was developed against Pool CLI 1.0.14.
• Standalone Pool credentials created with pool login --api-key "$POOLSIDE_API_KEY". The harness reads $HOME/.config/poolside/credentials.json.
• Jujutsu on PATH, or pass --jj-bin; this was developed against Jujutsu 0.42.0.
• Either a GitButler checkout with Rust/Cargo available, or an existing but binary plus the checkout's crates/but/skill directory.
• Network access for Pool and, unless --jj-skill-dir is supplied, the pinned Jujutsu skill download.

Run only Pool / Laguna

This runs six tasks x three tool arms x three repetitions: 54 runs.

npm run matrix:run -- \
  --agents pool \
  --k 3 \
  --pool-model poolside/laguna-s-2.1 \
  --pool-bin /absolute/path/to/pool \
  --jj-bin /absolute/path/to/jj \
  --gitbutler-root /absolute/path/to/gitbutler \
  --batch-name pool-laguna-s21-full-k3 \
  --pool-rate-limit-retries 6 \
  --pool-rate-limit-backoff-ms 60000 \
  --fail-on-failures true

--pool-bin and --jj-bin can be omitted when those commands are already on PATH.

To use a prebuilt GitButler binary, replace --gitbutler-root with:

--build-but false \
--but-bin /absolute/path/to/but \
--skill-dir /absolute/path/to/gitbutler/crates/but/skill

Use a fresh batch name. batch-config.json records the model, binaries, skills, runtime, and harness fingerprint; a resume with different inputs is rejected.

Debug traces

Everything is under tmp/pilot-runs/<batch-name>/.

Batch-level files:

• batch-config.json: exact batch inputs and harness fingerprint.
• plan.tsv and progress.tsv: planned runs, completions, retries, and failures.
• <run-id>.log: runner output for each run.
• aggregate.json, manifest.tsv, and report.md: the current batch summary.
• retry-attempts/<run-id>/attempt-N/: complete failed run directories and runner.log files retained before rate-limit retries.

Per-run files:

• result.json: authoritative failure class, verifier result, parsed Pool output, observed model, and sandbox/auth details.
• command-trace.tsv: wrapped Git, GitButler, and Jujutsu invocations.
• pool-trajectory.ndjson: retained Pool trajectory.
• pool-home/Library/Application Support/poolside/trajectories/*.ndjson: Pool's native trajectory files.
• pool-home/Library/Application Support/poolside/logs/*.log: Pool runtime logs.
• agent-stdout.txt, agent-stderr.txt, prompt.txt, pool-auth-proxy.log, pool-host-sandbox.sb, and workspace/.
• tmp/pool-shell-*/output_*.txt: shell-session output created inside the run.

Start diagnosis with:

jq '{run_failure_class, agent_result, agent_output, verifier, agent_config}' \
  tmp/pilot-runs/<batch-name>/<run-id>/result.json

A passing repository verifier is not enough when agent_output.output_error is set; transcript, model, command-boundary, or credential-cleanup failures deliberately invalidate the run.

Pool CLI note

Pool CLI 1.0.14 rejects its advertised -m option as obsolete. The harness therefore selects Laguna through generated isolated settings and verifies poolside/laguna-s-2.1 from the native trajectory before accepting a run.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
version-control-bench	Ready	Preview, Comment	Jul 22, 2026 7:15am