Add Hermes browser benchmark harness to Stagehand evals

[shrey150]

What changed

• Adds hermes as a first-class executable harness in packages/evals.
• Supports three surfaces through the existing eval CLI: hermes_browser_legacy, hermes_browser_exec (Browser Use), and hermes_stagehand_batch (Stagehand V4 experimental_batch).
• Adds a frozen ten-task Online-Mind2Web hard slice, counterbalanced A/B/D runner, strict canary gate, resumable records, Vercel AI Gateway routing, and Browserbase execution.
• Converts retained Hermes state plus independently captured URL/screenshot evidence into normal Stagehand verifier trajectories.
• Adds cached-rubric grading, identity-bound verifier recovery, benchmark:hermes-public-hard, and regrade:hermes.

Why

The public Nous/Hermes result did not publish its six prompts, trajectories, grader, or website snapshots. This adapter tests the same tool-surface hypothesis through Stagehand's existing eval infrastructure while preserving Hermes's agent loop.

Companion native runner and audited report: browserbase/hermes-browser-agent-benchmarks#1.

Final matched result

Two independent 30-trajectory matrices completed over the same ten tasks and schedule.

Orchestrator	A · Hermes legacy	B · Browser Use	D · Stagehand batch
Hermes-native strict scheduled success	6/10	7/10	8/10
Stagehand evals strict scheduled success	5/10	5/10	8/10

Relative to Browser Use:

• Native: Stagehand used 19.6% fewer tool calls, but 55.5% more trajectory tokens and cost 17.5% more.
• Stagehand evals: Stagehand used 22.4% fewer tool calls and 9.1% less agent wall time, but 17.2% more trajectory tokens and cost 40.0% more.

Stagehand improved scheduled reliability and tool-call count, but did not reproduce the Nous 60–66% token reduction or satisfy the conjunctive accuracy-and-token-efficiency win rule.

Claim boundary

• These are ten frozen public proxies, not the unreleased six Nous tasks.
• Current Hermes exposes twelve legacy tools; the published Nous baseline reportedly had eight.
• Native B/D and evals B lack complete verifier coverage. Scheduled success counts those rows as failures.
• A first-write rubric-cache race produced two retained rubric digests for one task. No full-matrix trajectory was retried, discarded, or regraded; the report preserves both and withholds a strong claim.
• The executed matrix remains provenance-bound to its original run checkout; this PR ports the same harness cleanly onto current main.

Validation

• Full eval suite: 57 files / 442 tests passed.
• Evals typecheck and build, SDK TypeScript build, extension build, Oxfmt, and clean diff checks passed.
• Strict three-arm live canary passed; retained verifier recovery did not rerun the agent or browser.
• Full Stagehand-evals matrix completed 30/30 scheduled trajectories.

Published report

Open the SSO-gated matched HTML report.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 49ae12c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
stagehand	🟢 Ready	View Preview	Aug 12, 2026, 10:53 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.