Skip to content

feat: headline-first benchmarks with explicit context and telemetry metadata - #28

Merged
IsuraManchanayake merged 2 commits into
rc-0.1.2from
feat/benchmark-protocol-v2
Sep 16, 2026
Merged

IsuraManchanayake merged 2 commits into
rc-0.1.2from
feat/benchmark-protocol-v2

Conversation

@IsuraManchanayake

@IsuraManchanayake IsuraManchanayake commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Run the standard PP512/TG128 headline before the larger prefill sweep; custom sweeps without PP512 use their first requested PP size.
  • Capability-gate BaseRT's new --headline-first option (features.headline_context_capacity, basert-throughput-protocol/2). The headline reserves 4K, TG starts from one untimed seed token, and remaining PP sizes run afterward. Validate capacity, history, order and requested repetitions before signing.
  • Preserve BaseRT harness warmup, repetitions, timers and telemetry. Do not copy basert-bench's adaptive warmup or five-repetition default. basert-bench itself is unchanged in the companion runtime PR.
  • Use the user's unmodified llama-bench. Both cooldown and standard modes run PP512, then TG128, then the remaining PP sizes; retain native warmup and token timing.
  • Stock llama-bench has no independent 4K reservation option. Retain its native PP+TG+depth capacity request and explicitly record that the 4K target was not applied. Do not substitute -d 4096, which would add actual token history. No patched runtime, fork, or special installer.
  • Keep native/external same-run telemetry, host OS/GPU/power/memory-pressure/swap snapshots, signed raw samples, binary provenance, and existing report commands.

Reports and compatibility

BaseRT headline reports normalize to computearena-throughput/3 with full native basert-bench-capacity/1 evidence. Stock llama-bench remains /2 with additive execution-order and context-request metadata. Runtime-specific warmup, repetition, capacity, telemetry and cooldown differences remain in the signed JSON; matching workload labels alone does not promise identical rates.

Older BaseRT harnesses keep their existing isolated or legacy path, with no new unsupported flags. Historical report envelopes and signatures are unchanged; offline reports remain verifiable/submittable. The web follow-up accepts the new protocol/layout without adding a depth selector, splitting historical rankings, or migrating existing rows.

Deploy web compatibility PR https://github.com/basecompute/computearena/pull/69 before publishing this CLI update. Companion BaseRT PR: https://github.com/basecompute/baseRT-internal/pull/489 . No PR should be merged automatically.

Validation

  • cargo test --workspace -- --test-threads=4: 128 passing unit, black-box fake-runtime, signing, malformed result, legacy capability, headline ordering, runtime setup and offline-upgrade compatibility tests.
  • cargo clippy --workspace --all-targets -- -D warnings.
  • cargo fmt and git diff --check.

Tests use fake runtimes and tiny synthetic model headers; no GPU inference was run. Real Metal/CUDA throughput comparisons remain necessary before making a numerical-parity claim.

@IsuraManchanayake IsuraManchanayake changed the title feat: align benchmark context and host telemetry feat: headline-first benchmarks with explicit context and telemetry metadata Sep 15, 2026
@IsuraManchanayake
IsuraManchanayake merged commit 6b892f8 into rc-0.1.2 Sep 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant