Add Pool harness support for Laguna S 2.1

README.md

@@ -76,7 +76,7 @@ Each scenario is a pre-built Git repository (a commit history plus uncommitted c
 
 ## How it's scored
 
-- **Identical instruction across tools.** Each task ships as one prepared fixture repo with one plain-English instruction. The tool's name does not appear in the prompt; the agent decides how to carry it out.
+- **Identical task across tools.** Each task ships as one prepared fixture repo with one plain-English instruction. Harness policy identifies the allowed command wrappers for the selected arm, but the requested Git-visible outcome is unchanged.
 - **Deterministic grader.** Correctness is checked by a hidden, scripted verifier that inspects the final Git state: commit boundaries, branch topology, and what stayed uncommitted. It is not an LLM judge and does not compare commands against a reference — two different command sequences pass if they produce the same history.
 - **Timing boundary.** Fixture build, workspace prep, skill installation, and dirty-state application all happen before timing begins; the measured figures cover only the agent's work on the task.
 - **Git write restriction.** In GitButler and Jujutsu runs, raw git write commands are blocked so the agent must use the tool under test. Git calls a tool makes internally count as tool-internal work, not agent commands.
@@ -90,7 +90,7 @@ Full method docs: [benchmark design](docs/benchmark-design.md), [scoring and val
 - Synthetic TypeScript fixtures generated by `scripts/create-pilot*-fixture.mjs`.
 - Hidden oracle verifiers under `scripts/verify-pilot*.mjs`.
 - Reference `git` and `but` solutions in each task directory.
-- An agent runner for Codex and Claude: `scripts/run-pilot-agent.mjs`.
+- An agent runner for Codex, Claude, and Pool: `scripts/run-pilot-agent.mjs`.
 - Tool-policy wrappers that block the wrong write tool per arm and split measurements into task, platform, and tool-internal commands.
 - Checked-in result summaries under [docs/results/](docs/results/).
 - The [vcbench.dev](https://vcbench.dev/) results site under [web/](web/).
@@ -130,7 +130,7 @@ npm run pilot:agent -- --task pilot-5-squash-commits --agent claude --arm 'but+s
 npm run pilot:agent -- --task pilot-5-squash-commits --agent claude --arm 'jj+skill'
 ```
 
-Defaults are `--task pilot-1-selective-validation`, `--agent codex`, `--arm git`, Codex model `gpt-5.5`, and Claude model `claude-opus-4-8` (a versioned model ID, so reruns hit the same model; the runner warns if you pass a floating alias like `opus`). Use `--model <name>` to override, or `--codex-model` / `--claude-model` on the matrix runner. The published k=7 results predate this default and used `claude-opus-4-1-20250805`.
+Defaults are `--task pilot-1-selective-validation`, `--agent codex`, `--arm git`, Codex model `gpt-5.5`, Claude model `claude-opus-4-8`, and Pool model `poolside/laguna-s-2.1`. Use `--model <name>` to override a single run, or `--codex-model`, `--claude-model`, and `--pool-model` on the matrix runner. The runner records and verifies the concrete model observed in each Pool trajectory.
 
 Claude auth gotcha: the runner copies the keychain OAuth token into an isolated config that cannot refresh itself. The keychain access token expires after roughly eight hours, and once it does every Claude run fails within seconds as `AGENT_RUNTIME_ERROR` with a 401 in `agent-output.json`. Run any plain `claude -p "ok"` outside the harness to refresh the keychain token, then rerun the affected cells. Before long batches (and periodically during overnight sessions), refresh proactively.
 
@@ -153,6 +153,40 @@ npm run matrix:run -- --k 5 --batch-name full-k5
 
 Use `--tasks`, `--agents`, or `--arms` with comma-separated values to run a subset. Plans, progress, aggregate data, and the generated report land under `tmp/pilot-runs/<batch-name>/`.
 
+### Pool / Laguna S 2.1
+
+Pool runs currently require macOS because the host boundary uses `sandbox-exec`. Install Node 20+, Git, Pool CLI, and Jujutsu. This path was developed against Pool CLI 1.0.14 and Jujutsu 0.42.0. Authenticate Pool in standalone mode, supplying the key through your shell or secret manager:
+
+```bash
+pool login --api-key "$POOLSIDE_API_KEY"
+```
+
+The harness reads the resulting standalone credential from `$HOME/.config/poolside/credentials.json`.
+
+The full Pool-only k=3 suite is 54 runs: six tasks, three tool arms, and three repetitions. Use a fresh batch name:
+
+```bash
+npm run matrix:run -- \
+  --agents pool \
+  --k 3 \
+  --pool-model poolside/laguna-s-2.1 \
+  --pool-bin /absolute/path/to/pool \
+  --jj-bin /absolute/path/to/jj \
+  --gitbutler-root /absolute/path/to/gitbutler \
+  --batch-name pool-laguna-s21-full-k3 \
+  --pool-rate-limit-retries 6 \
+  --pool-rate-limit-backoff-ms 60000 \
+  --fail-on-failures true
+```
+
+`--pool-bin` and `--jj-bin` can be omitted when both commands are on `PATH`. The default path builds `but` from the GitButler checkout and reads its skill from `crates/but/skill`. To use an existing binary instead, replace `--gitbutler-root` with:
+
+```bash
+--build-but false --but-bin /absolute/path/to/but --skill-dir /absolute/path/to/gitbutler/crates/but/skill
+```
+
+Network access is needed for Pool and, unless `--jj-skill-dir` points at a local copy, the pinned Jujutsu skill download. A batch writes `batch-config.json` with its inputs and harness fingerprint; resuming with different inputs is rejected.
+
 ### Local GitButler build
 
 Build `but` from the local GitButler checkout and pass it to the runner:
@@ -187,13 +221,26 @@ npm run pilot:agent -- --agent codex --arm git --codex-clean-config false
 
 Claude trials use a temporary config directory, load only project/local settings, disable user plugins, and ignore ambient MCP configuration. OAuth is passed through the process environment and the temporary config is deleted after the run; credentials are never written into retained benchmark artifacts. Pass `--claude-clean-config false` to opt out.
 
+### Pool isolation
+
+Pool trials use a per-run home, generated settings, a credential proxy, and a macOS sandbox. The model receives only a dummy credential; the real API key remains in the proxy process and is not written into retained artifacts. The runner pins Laguna through generated Pool settings because Pool CLI 1.0.14 rejects its advertised `-m` flag, then verifies the observed trajectory model before grading the run.
+
 ### Outputs
 
 Run artifacts are written under `tmp/pilot-runs/` and ignored by Git. A run directory contains the sandbox workspace, `result.json`, the command trace, generated instruction files, and verifier output.
 
 `npm run runs:export -- --batch <batch-name>` packages the publishable evidence for a batch — per-run agent transcripts, command traces, prompts, `result.json`, runner logs, and a final Git-state snapshot per run — into a tarball under `tmp/exports/` (the k=7 batch exports to ~2 MB). The export scans for credential-shaped strings and aborts if any are found. Batches referenced in public results should have their artifact bundle attached to a GitHub release so every reported number can be traced to raw evidence.
 Claude runs also write the raw JSON CLI result to `agent-output.json`; `result.json` records the configured model alias and the concrete observed model from Claude's `modelUsage`.
 
+Pool batches retain the evidence needed to diagnose harness or model failures:
+
+- Batch-level: `batch-config.json`, `plan.tsv`, `progress.tsv`, `<run-id>.log`, `aggregate.json`, `manifest.tsv`, and `report.md`.
+- Rate-limit retries: `retry-attempts/<run-id>/attempt-N/`, containing the complete failed run directory and `runner.log`.
+- Per-run: `result.json`, `command-trace.tsv`, `pool-trajectory.ndjson`, `agent-stdout.txt`, `agent-stderr.txt`, `prompt.txt`, `pool-auth-proxy.log`, `pool-host-sandbox.sb`, and `workspace/`.
+- Pool-native files: `pool-home/Library/Application Support/poolside/trajectories/*.ndjson` and `pool-home/Library/Application Support/poolside/logs/*.log`.
+
+Start with `result.json`: `agent_output.output_error` can explain a failed Pool interaction even when the repository verifier itself passes.
+
 The useful measurement block is `measurement`, not the older coarse `metrics` block. It separates:
 
 - task-relevant VC commands

package.json

@@ -26,6 +26,7 @@
     "pilot6:check": "node scripts/run-pilot6-checks.mjs",
     "matrix:run": "node scripts/run-full-matrix.mjs",
     "matrix:merge": "node scripts/merge-matrix-batches.mjs",
+    "pool:check": "node scripts/test-pool-auth-proxy.mjs && node scripts/run-pilot-agent.mjs --self-test-metrics && node scripts/run-full-matrix.mjs --self-test true",
     "runs:export": "node scripts/export-batch-artifacts.mjs",
     "web:data": "node scripts/build-web-data.mjs"
   },

scripts/lib/pool-auth-proxy.mjs

@@ -0,0 +1,118 @@
+#!/usr/bin/env node
+import http from "node:http";
+import https from "node:https";
+import { appendFileSync, renameSync, writeFileSync } from "node:fs";
+
+function arg(name) {
+  const index = process.argv.indexOf(name);
+  return index >= 0 ? process.argv[index + 1] : null;
+}
+
+const upstream = new URL(arg("--api-url"));
+const portFile = arg("--port-file");
+const logFile = arg("--log-file");
+const parentPid = Number(arg("--parent-pid"));
+let token = "";
+
+const log = (event, detail = {}) => {
+  if (!logFile) return;
+  appendFileSync(logFile, `${JSON.stringify({ timestamp: new Date().toISOString(), event, ...detail })}\n`, { mode: 0o600 });
+};
+
+const hopByHopHeaders = new Set([
+  "connection",
+  "keep-alive",
+  "proxy-authenticate",
+  "proxy-authorization",
+  "te",
+  "trailer",
+  "transfer-encoding",
+  "upgrade",
+]);
+
+process.stdin.setEncoding("utf8");
+process.stdin.on("data", (chunk) => {
+  token += chunk;
+});
+process.stdin.on("end", () => {
+  if (!token || !portFile) process.exit(2);
+
+  const transport = upstream.protocol === "https:" ? https : http;
+  const upstreamBasePath = upstream.pathname.replace(/\/$/, "");
+  const sockets = new Set();
+  const forwardedRequests = new Set();
+  let shuttingDown = false;
+  const server = http.createServer((request, response) => {
+    const requestTarget = request.url ?? "";
+    if (!requestTarget.startsWith("/") || requestTarget.startsWith("//")) {
+      log("request_rejected", { reason: "non_origin_form" });
+      response.writeHead(400).end("invalid request target");
+      return;
+    }
+    const target = new URL(requestTarget, upstream.origin);
+    const hasExpectedBasePath = !upstreamBasePath
+      || target.pathname === upstreamBasePath
+      || target.pathname.startsWith(`${upstreamBasePath}/`);
+    if (target.origin !== upstream.origin || !hasExpectedBasePath) {
+      log("request_rejected", { reason: "upstream_boundary" });
+      response.writeHead(400).end("invalid upstream target");
+      return;
+    }
+    const headers = Object.fromEntries(
+      Object.entries(request.headers).filter(([name]) => !hopByHopHeaders.has(name.toLowerCase())),
+    );
+    headers.host = target.host;
+    headers.authorization = `Bearer ${token}`;
+    headers["x-api-key"] = token;
+    const forwarded = transport.request(target, { method: request.method, headers }, (upstreamResponse) => {
+      response.writeHead(upstreamResponse.statusCode ?? 502, upstreamResponse.headers);
+      upstreamResponse.pipe(response);
+    });
+    forwardedRequests.add(forwarded);
+    forwarded.once("close", () => forwardedRequests.delete(forwarded));
+    forwarded.setTimeout(20 * 60 * 1000, () => forwarded.destroy(new Error("upstream request timed out")));
+    forwarded.on("error", () => {
+      log("upstream_error");
+      if (!response.headersSent) response.writeHead(502);
+      response.end("upstream request failed");
+    });
+    request.pipe(forwarded);
+  });
+  server.on("connection", (socket) => {
+    sockets.add(socket);
+    socket.once("close", () => sockets.delete(socket));
+  });
+
+  server.listen(0, "127.0.0.1", () => {
+    const address = server.address();
+    const pendingPortFile = `${portFile}.tmp`;
+    writeFileSync(pendingPortFile, JSON.stringify({ port: address.port }), { mode: 0o600 });
+    renameSync(pendingPortFile, portFile);
+    log("ready", { port: address.port, upstream_origin: upstream.origin, upstream_base_path: upstreamBasePath });
+  });
+
+  const shutdown = () => {
+    if (shuttingDown) return;
+    shuttingDown = true;
+    token = "";
+    log("shutdown");
+    for (const forwarded of forwardedRequests) forwarded.destroy();
+    if (typeof server.closeAllConnections === "function") server.closeAllConnections();
+    for (const socket of sockets) socket.destroy();
+    server.close(() => process.exit(0));
+    setTimeout(() => process.exit(0), 250).unref();
+  };
+
+  if (Number.isInteger(parentPid) && parentPid > 1) {
+    setInterval(() => {
+      try {
+        process.kill(parentPid, 0);
+      } catch {
+        shutdown();
+      }
+    }, 1000).unref();
+  }
+
+  process.on("SIGTERM", shutdown);
+  process.on("SIGINT", shutdown);
+});