Benchmark publishing: validated results PRs for docs and website

[SarthakWade]

Problem

Benchmark results currently feed repository documentation, but the external benchmark lab needs a controlled publication path. Directly mutating production docs from benchmark runs would allow noisy, incomplete, or compromised results to become public claims.

Trigger policy

A published stable Headless release matching the approved v* policy triggers a benchmark run for that exact release. Prereleases, draft releases, duplicate delivery events, and unverified tags do not publish results.

Use a narrowly scoped GitHub App or equivalent least-privilege mechanism for the cross-repository trigger and PR creation. A scheduled competitor-refresh run may use the same methodology without pretending that a new Headless release occurred.

Manual runs remain available for methodology validation and recovery, but must produce the same provenance as automated runs.

Release workflow

1. Receive the release tag, source commit, release URL, and expected artifact identifiers.
2. Verify the tag, commit, downloadable artifact checksums, and supported platform before executing untrusted release output.
3. Deduplicate by Headless release, methodology version, platform, and benchmark configuration.
4. Run the pinned matrix and repeated trials defined by Benchmark lab: reproducible multi-tool methodology and result schema #161.
5. Validate task coverage, raw evidence, sample counts, statistical stability, and the versioned result schema.
6. Store the raw result bundle as an immutable benchmark-lab artifact with a content digest.
7. Generate a bounded canonical result update for this repository.
8. Open or update one reviewable PR against the existing benchmark provenance artifact.
9. Let this repository's CI validate the canonical data, generated documentation, and website build.
10. Publish through the existing Vercel GitHub integration only after the PR is reviewed and merged.

The benchmark repository must never push directly to main, mutate production documentation, or deploy the website. Do not add a second website deployment workflow.

Published data

The website must identify:

• the tested Headless version and source commit;
• every agent, model, browser tool, browser, and container version;
• methodology and result-schema versions;
• run timestamp, environment, sample count, and confidence information;
• whether the result belongs to the paired tool, Headless compatibility, or product-level track;
• known limitations, unsupported capabilities, human interventions, and links to immutable evidence.

Keep accuracy, efficiency, cost, and security results separate. Do not generate one weighted score that hides safety failures or non-equivalent configurations.

Failure behavior

A failed, incomplete, statistically unstable, or unauthorized run leaves the last accepted website result unchanged. Surface the failure in the benchmark repository without opening a result PR.

A later retry for the same release updates the existing automation branch or PR rather than creating duplicates. Rollback restores a previously reviewed canonical result and requires no benchmark rerun.

The website should clearly show the last successfully tested release and date. It must not imply that a failed or pending release has benchmark coverage.

Security and permissions

• The publishing identity may create a bounded branch and PR but cannot push to protected branches or deploy.
• Benchmark outputs are untrusted input. They cannot modify workflows, executable code, prompts, validators, or deployment configuration through the publishing path.
• Pin actions and executable dependencies according to repository policy.
• Never use production credentials in tasks, fixtures, logs, raw artifacts, generated prose, or PR content.
• Reject symlinked, oversized, malformed, unsigned, incomplete, or schema-incompatible result bundles.
• Preserve an audit trail from the published release through the raw bundle digest to the canonical website data.

Acceptance criteria

• A verified stable release triggers exactly one benchmark configuration per approved platform and methodology version.
• Trigger retries and duplicate release events are idempotent.
• The benchmark runs the matrix defined by Benchmark lab: reproducible multi-tool methodology and result schema #161 and records complete provenance.
• Publication rejects incomplete suites, schema drift, missing evidence, implausible values, unstable summaries, and unauthorized sources.
• Failed runs never replace the last accepted baseline.
• Automation opens a reviewable PR and cannot push to main or deploy.
• CI verifies canonical data, docs generation, and the website before merge.
• The existing Vercel integration remains the only website deployment path.
• The website displays the tested release, versions, methodology, samples, date, limitations, and evidence.
• Rollback is documented and requires no benchmark rerun.

Blocked by #161.

[yashranaway]

Blocked on #161. Do not publish lab numbers onto the website by a side channel. When the lab exists, results come in as a reviewed PR against the existing provenance artifact.

[yashranaway]

Still blocked on #161. No second website deploy path and no unreviewed result PRs until the lab schema exists.