feat: headline-first benchmarks with explicit context and telemetry metadata - #28
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--headline-firstoption (features.headline_context_capacity,basert-throughput-protocol/2). The headline reserves 4K, TG starts from one untimed seed token, and remaining PP sizes run afterward. Validate capacity, history, order and requested repetitions before signing.basert-benchitself is unchanged in the companion runtime PR.-d 4096, which would add actual token history. No patched runtime, fork, or special installer.Reports and compatibility
BaseRT headline reports normalize to
computearena-throughput/3with full nativebasert-bench-capacity/1evidence. Stock llama-bench remains/2with additive execution-order and context-request metadata. Runtime-specific warmup, repetition, capacity, telemetry and cooldown differences remain in the signed JSON; matching workload labels alone does not promise identical rates.Older BaseRT harnesses keep their existing isolated or legacy path, with no new unsupported flags. Historical report envelopes and signatures are unchanged; offline reports remain verifiable/submittable. The web follow-up accepts the new protocol/layout without adding a depth selector, splitting historical rankings, or migrating existing rows.
Deploy web compatibility PR https://github.com/basecompute/computearena/pull/69 before publishing this CLI update. Companion BaseRT PR: https://github.com/basecompute/baseRT-internal/pull/489 . No PR should be merged automatically.
Validation
cargo test --workspace -- --test-threads=4: 128 passing unit, black-box fake-runtime, signing, malformed result, legacy capability, headline ordering, runtime setup and offline-upgrade compatibility tests.cargo clippy --workspace --all-targets -- -D warnings.cargo fmtandgit diff --check.Tests use fake runtimes and tiny synthetic model headers; no GPU inference was run. Real Metal/CUDA throughput comparisons remain necessary before making a numerical-parity claim.