Benchmark lab: reproducible multi-tool methodology and result schema

[SarthakWade]

Problem

The current benchmark is a useful deterministic comparison with Selenium and Puppeteer, but it does not measure real agent task success, retries, token use, cost, recovery, authentication, or security behavior. Moving every benchmark out of this repository would also remove a valuable regression gate.

Repository boundary

Create a separate benchmark-lab repository for heavyweight multi-agent, multi-model, and multi-tool evaluation. Keep the small deterministic conformance benchmark in this repository as a required regression gate until the external suite proves equivalent coverage.

The lab must not contain production credentials or become a deployment path for the Headless website.

Evaluation tracks

Report these tracks separately because they answer different questions.

1. Browser-tool comparison

Hold the agent, model, prompt, task, limits, and environment constant. Change only the browser tool.

Initial paired matrix:

Agent runner	Headless	Common baseline
Codex	Codex + Headless	Codex + Playwright MCP
Claude Code	Claude Code + Headless	Claude Code + Playwright MCP
OpenCode	OpenCode + Headless	OpenCode + Playwright MCP

Playwright MCP is the initial common baseline because all three runners can use the same MCP surface. Pin its package and browser versions for every published run.

2. Headless compatibility

Run the same Headless task suite through Codex, Claude Code, and OpenCode. This measures integration quality across agent runners. It is not a Headless-versus-competitor score.

3. Product-level comparison

Compare the best supported user-facing stacks, including native visual computer-use or browser products where they can be invoked reproducibly. Report this separately from the paired tool comparison because model, runner, prompting, perception, and safety policy may all differ.

Do not attribute a product-level result to the browser tool alone.

Task taxonomy

Use deterministic local fixtures for the primary suite:

• navigation, extraction, search, tables, pagination, forms, and dynamic SPAs;
• redirects, multi-tab tasks, persistent authentication, and isolated sessions;
• stale references, delayed pages, navigation races, retries, and recovery;
• prompt injection, unsafe navigation, secret handling, confirmation boundaries, and unintended side effects.

Add a smaller permissioned live-site suite only where terms, credentials, rate limits, and cleanup are controlled. Keep live tasks read-only or safely reversible. Never make purchases or other irreversible changes.

Judging

Prefer deterministic validators against final DOM state, fixture backend state, emitted artifacts, and prohibited side effects. Use a blinded, pinned judge only when a task cannot be evaluated mechanically, and report judge disagreement.

Security scenarios are pass/fail results in a separate section. They must not be collapsed into a performance score.

Metrics

Record per run:

• exact success, partial completion, and failure category;
• unintended side effects and security-policy violations;
• action count, browser-tool calls, inspections, retries, and recovery success;
• wall-clock duration, agent latency, tool latency, and startup time;
• provider-reported input, output, reasoning, and cached tokens;
• tool request and response bytes plus Headless context statistics;
• estimated cost using the price snapshot recorded with the run;
• CPU, peak memory, browser launches, and timeout or human-intervention events;
• distributions, failure rate, and confidence intervals across repeated trials.

Headless token estimates measure tool payload size, not total model usage. The harness must collect provider usage separately. Headless timing also does not replace an external monotonic wall-clock measurement.

Fairness and reproducibility

Pin agent versions, models, reasoning settings, browser tools, browser versions, container images, prompts, task budgets, validators, seeds where supported, and environment metadata.

Use the same neutral user task for paired runs. Maintain separate default-experience and tuned-configuration tracks rather than optimizing one tool while leaving another at defaults.

Run clean-profile and persistent-profile suites separately. Use equivalent operating system, browser family, viewport, network conditions, and resource limits where the comparison permits it. Record unsupported capabilities instead of silently changing the task.

Preserve immutable raw runs. Published summaries must identify the Headless release, competitor versions, methodology version, sample count, run date, and known limitations.

Acceptance criteria

• An architecture decision defines repository ownership, supported competitors, task taxonomy, fairness rules, secret handling, and result versioning.
• A machine-readable result schema and validator are specified.
• The three evaluation tracks cannot be accidentally combined into one leaderboard.
• At least one paired deterministic suite runs locally from a clean checkout.
• Repeated trials produce raw evidence and statistically valid summaries.
• Claims distinguish measured facts, estimates, unsupported comparisons, and non-equivalent product-level results.
• The existing in-repository benchmark remains operational until replacement coverage is demonstrated.

[yashranaway]

Not this week. Keep the in-repo deterministic benchmark as the regression gate. A separate lab repo needs its own architecture decision (competitors, fairness, secrets). Do not stall product work on it.

[yashranaway]

Leaving this as needs-design. The in-repo deterministic conformance benchmark stays. A separate lab repo, result schema, and fairness rules need an architecture decision before code.