feat: headline-first benchmarks with explicit context and telemetry metadata

README.md

@@ -155,12 +155,16 @@ and piped input must use `--yes`. Details are in
 [docs/benchmark-profiles.md](docs/benchmark-profiles.md).
 
 Telemetry is automatic for both runtimes and needs no flag, credential, or
-sudo. ComputeArena observes the single process it launches for current BaseRT
-and llama.cpp builds: resident memory, operating-system temperature sensors,
-power state, and NVIDIA or ROCm device snapshots where those vendor tools exist.
-A future BaseRT harness can advertise native same-run telemetry, which the CLI
-will use without a CLI release or version-string rule. Coverage and limitations
-are in [docs/telemetry.md](docs/telemetry.md).
+sudo. ComputeArena observes the processes it launches: resident memory,
+operating-system temperature sensors, power state, memory pressure and swap,
+plus NVIDIA or ROCm device snapshots where those vendor tools exist. Signed
+environment boundaries also record the OS/kernel, CPU layout, host memory, and
+available GPU configuration; static macOS display configuration is cached so
+`system_profiler` is not rerun at every boundary. A BaseRT harness can
+advertise native same-run telemetry, which the CLI uses without a CLI release
+or version-string rule. Runtime-native detail is preferred when it is more
+accurate; portable host boundaries remain available for cross-runtime analysis.
+Coverage and limitations are in [docs/telemetry.md](docs/telemetry.md).
 
 ## Runtimes
 
@@ -214,10 +218,19 @@ reads its GGUF header only. The history lives in `recent-gguf.json` in the
 data directory, is never part of a report, and can be deleted to reset the
 list; a corrupt or unwritable history never blocks a benchmark.
 
-llama.cpp measurements carry their own protocol identifiers and record native
-warmup, zero context depth, and the exclusion of sampling and tokenization, so
-they are never presented as BaseRT numbers. The measurement contract is in
-[docs/runtime-adapters.md](docs/runtime-adapters.md).
+The standard headline is PP512 followed by TG128 **before** the remaining PP
+sweep. PP starts empty; TG starts with one untimed seed token. A capable BaseRT
+harness reserves 4K for the headline (`computearena-throughput/3`), without
+changing its existing warmup, repetitions, timing or telemetry. Older harnesses
+retain their existing invocation and recorded protocol.
+
+ComputeArena uses the user's unmodified llama-bench build. Its native capacity
+is retained: stock llama-bench has no independent 4K reservation option, and
+`-d 4096` would add real history instead. This difference, actual workload order,
+warmup and context requests are signed in the JSON. There is no claim of exact
+cross-runtime equivalence. Existing reports remain verifiable and submittable;
+protocol differences are not a new leaderboard filter.
+The measurement contract is in [docs/runtime-adapters.md](docs/runtime-adapters.md).
 
 ## Reports and signatures
 
@@ -334,12 +347,19 @@ reports, sessions, and the signing key.
 - Report envelope: `computearena-benchmark/1`
 - BaseRT harness output: `basert-benchmark-harness/1`; the older
   `basert-harness/1` is still accepted by the server
-- llama.cpp measurements: `computearena-measurements/1`, executed as
-  `llama-bench-independent-pp-tg/1` or, with cooldown,
-  `llama-bench-conditioned-pp-tg/1`
+- llama.cpp measurements: `computearena-measurements/1`
+- Comparable throughput semantics: `computearena-throughput/2`, with native
+  evidence retained as `basert-throughput-protocol/1` or
+  `llama-bench-json/1`. Older BaseRT results use
+  `computearena-throughput-legacy/1` and are marked non-comparable.
+- Headline-first BaseRT capacity: `computearena-throughput/3`, native evidence
+  `basert-throughput-protocol/2` / `basert-bench-capacity/1`. The historical
+  `comparable` metadata is not a guarantee of identical measured performance
+  and does not exclude older reports from the website.
 - Telemetry: `computearena-telemetry/1` for externally observed BaseRT and
   llama.cpp runs. A BaseRT harness advertising `features.same_run_telemetry`
   uses native `basert-telemetry/4` instead.
+- Host environment boundaries: `computearena-environment/1`
 - Signing: Ed25519 over `computearena-json-v1` canonical JSON
 
 ## Development

crates/computearena-cli/src/adapters/basert.rs

@@ -1,5 +1,8 @@
 use super::{BenchmarkRequest, RuntimeAdapter, RuntimeOutput};
-use crate::protocol::BASERT_SAME_RUN_TELEMETRY_SCHEMA;
+use crate::protocol::{
+    BASERT_ISOLATED_PROTOCOL_SCHEMA, BASERT_SAME_RUN_TELEMETRY_SCHEMA,
+    DECODE_INITIAL_CONTEXT_TOKENS, THROUGHPUT_PROTOCOL_ID,
+};
 use crate::ui::TerminalUi;
 use anyhow::{bail, Context, Result};
 use serde_json::{json, Value};
@@ -14,6 +17,27 @@ enum TelemetryMode {
     NativeSameRun,
 }
 
+fn supports_isolated_workloads(descriptor: &Value) -> bool {
+    descriptor
+        .pointer("/features/isolated_workload_contexts")
+        .and_then(Value::as_bool)
+        == Some(true)
+}
+
+fn supports_headline_capacity(descriptor: &Value) -> Result<bool> {
+    let supported = descriptor
+        .pointer("/features/headline_context_capacity")
+        .and_then(Value::as_bool)
+        == Some(true);
+    if supported
+        && descriptor["capacity_protocol_schema"].as_str()
+            != Some(crate::protocol::BASERT_CAPACITY_PROTOCOL_SCHEMA)
+    {
+        bail!("BaseRT advertises an unsupported headline context protocol");
+    }
+    Ok(supported)
+}
+
 fn telemetry_mode(descriptor: &Value) -> Result<TelemetryMode> {
     if descriptor
         .pointer("/features/same_run_telemetry")
@@ -81,6 +105,8 @@ impl RuntimeAdapter for BaseRtAdapter {
     ) -> Result<RuntimeOutput> {
         let ui = TerminalUi::detect();
         let mode = telemetry_mode(descriptor)?;
+        let isolated_workloads = supports_isolated_workloads(descriptor);
+        let headline_capacity = supports_headline_capacity(descriptor)?;
         let suite_conditioning = if mode == TelemetryMode::ExternalWholeProcess && r.cooldown {
             let mut cooldown = crate::conditioning::Cooldown::new();
             Some(cooldown.prepare("BaseRT benchmark suite"))
@@ -97,6 +123,13 @@ impl RuntimeAdapter for BaseRtAdapter {
             .arg(r.reps.to_string())
             .arg("-w")
             .arg(r.warmup.to_string());
+        if headline_capacity {
+            command.arg("--headline-first");
+        } else if isolated_workloads {
+            command.arg("--isolated-workloads");
+        }
+        let native_environment_before =
+            (mode == TelemetryMode::NativeSameRun).then(crate::telemetry::capture_environment);
         let (output, external_telemetry) = match mode {
             TelemetryMode::ExternalWholeProcess => {
                 println!("{}", ui.neutral("Telemetry: observing the BaseRT harness process and available device sensors (whole run, 1-second sampling; no telemetry replays)."));
@@ -119,6 +152,10 @@ impl RuntimeAdapter for BaseRtAdapter {
         if !output.status.success() {
             bail!("BaseRT benchmark exited with {}", output.status);
         }
+        let native_environment = native_environment_before.map(|before| {
+            json!({"schema":"computearena-environment/1","measurement_relation":"outside_runtime_execution",
+                "before":before,"after":crate::telemetry::capture_environment()})
+        });
         let mut benchmark: Value =
             serde_json::from_slice(&output.stdout).context("BaseRT returned invalid JSON")?;
         let expected_schema = match mode {
@@ -127,6 +164,10 @@ impl RuntimeAdapter for BaseRtAdapter {
         };
         crate::benchmark::validate_harness_result(&benchmark, expected_schema)?;
         validate_requested_workloads(&benchmark, r)?;
+        normalize_protocol(&mut benchmark, r, isolated_workloads, headline_capacity)?;
+        if let Some(environment) = native_environment {
+            crate::telemetry::attach_environment(&mut benchmark, environment);
+        }
         if let Some(mut telemetry) = external_telemetry {
             let mut policy = if r.cooldown {
                 crate::conditioning::before_suite_policy()
@@ -162,6 +203,127 @@ impl RuntimeAdapter for BaseRtAdapter {
     }
 }
 
+fn normalize_protocol(
+    benchmark: &mut Value,
+    request: &BenchmarkRequest<'_>,
+    isolated_workloads: bool,
+    headline_capacity: bool,
+) -> Result<()> {
+    let runtime_protocol = benchmark.get("protocol").cloned().unwrap_or(Value::Null);
+    benchmark["params"]["decode_context_tokens"] = json!(DECODE_INITIAL_CONTEXT_TOKENS);
+    if headline_capacity {
+        return normalize_headline_protocol(benchmark, request, runtime_protocol);
+    }
+    if !isolated_workloads {
+        benchmark["protocol"] = json!({
+            "id":"computearena-throughput-legacy/1",
+            "comparable":false,
+            "reason":"The installed BaseRT harness does not advertise isolated workload contexts",
+            "context_isolation":"shared_sweep_context",
+            "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS},
+            "runtime_protocol":runtime_protocol
+        });
+        return Ok(());
+    }
+
+    if runtime_protocol["schema"].as_str() != Some(BASERT_ISOLATED_PROTOCOL_SCHEMA)
+        || runtime_protocol["context_isolation"].as_str() != Some("per_workload")
+        || runtime_protocol["context_capacity_policy"].as_str() != Some("minimum_required")
+        || runtime_protocol["model_load_in_timing"].as_bool() != Some(false)
+    {
+        bail!("BaseRT advertised isolated contexts but returned incompatible protocol metadata");
+    }
+    for prompt in request.pp.split(',') {
+        let tokens = prompt.parse::<u64>()?;
+        let workload = &runtime_protocol["prefill"][prompt];
+        if workload["initial_context_tokens"].as_u64() != Some(0)
+            || workload["context_capacity_tokens"].as_u64() != Some(tokens)
+        {
+            bail!("BaseRT returned incompatible PP{tokens} context metadata");
+        }
+    }
+    if runtime_protocol["decode"]["initial_context_tokens"].as_u64()
+        != Some(DECODE_INITIAL_CONTEXT_TOKENS)
+        || runtime_protocol["decode"]["context_capacity_tokens"].as_u64()
+            != Some(u64::from(request.tg) + DECODE_INITIAL_CONTEXT_TOKENS)
+    {
+        bail!(
+            "BaseRT returned incompatible TG{} context metadata",
+            request.tg
+        );
+    }
+    benchmark["protocol"] = json!({
+        "id":THROUGHPUT_PROTOCOL_ID,
+        "comparable":true,
+        "context_isolation":"per_workload",
+        "context_capacity_policy":"minimum_required",
+        "model_load_in_timing":false,
+        "prefill":{"initial_context_tokens":0},
+        "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS},
+        "tokenization_timed":false,
+        "sampling_timed":false,
+        "runtime_protocol":runtime_protocol
+    });
+    Ok(())
+}
+
+fn normalize_headline_protocol(
+    benchmark: &mut Value,
+    request: &BenchmarkRequest<'_>,
+    runtime: Value,
+) -> Result<()> {
+    use crate::protocol::{
+        headline_order, headline_prefill, BASERT_CAPACITY_PROTOCOL_SCHEMA,
+        HEADLINE_CONTEXT_CAPACITY, HEADLINE_PROTOCOL_ID,
+    };
+    let headline = headline_prefill(request.pp).parse::<u64>()?;
+    let capacity = (headline + u64::from(request.tg)).max(HEADLINE_CONTEXT_CAPACITY);
+    if runtime["schema"] != BASERT_CAPACITY_PROTOCOL_SCHEMA
+        || runtime["profile"] != "basert-bench-capacity/1"
+        || runtime["context_isolation"] != "headline_then_per_prefill"
+        || runtime["context_capacity_policy"] != "basert_bench_default"
+        || runtime["model_load_in_timing"] != false
+        || runtime["execution_layout"] != "headline_then_prefill_processes"
+        || runtime["execution_order"] != json!(headline_order(request.pp, request.tg))
+    {
+        bail!("BaseRT returned incompatible headline capacity/order metadata");
+    }
+    for prompt in request.pp.split(',') {
+        let tokens = prompt.parse::<u64>()?;
+        let expected = if tokens == headline {
+            capacity
+        } else {
+            tokens.max(HEADLINE_CONTEXT_CAPACITY)
+        };
+        if runtime["prefill"][prompt]["initial_context_tokens"] != 0
+            || runtime["prefill"][prompt]["context_capacity_tokens"].as_u64() != Some(expected)
+        {
+            bail!("BaseRT returned incompatible PP{tokens} capacity metadata");
+        }
+    }
+    if runtime["decode"]["initial_context_tokens"] != DECODE_INITIAL_CONTEXT_TOKENS
+        || runtime["decode"]["context_capacity_tokens"] != capacity
+        || runtime["decode"]["seed_prefill_in_timing"] != false
+        || runtime["measurement"]["timed_repetitions"] != request.reps
+        || runtime["measurement"]["requested_warmup_repetitions"] != request.warmup
+        || benchmark["params"]["ctx"] != capacity
+    {
+        bail!("BaseRT returned incompatible headline measurement metadata");
+    }
+    benchmark["protocol"] = json!({
+        "id":HEADLINE_PROTOCOL_ID, "comparable":true,
+        "profile":"headline-first-capacity/1", "context_isolation":"headline_then_per_prefill",
+        "context_capacity_policy":"basert_bench_default", "model_load_in_timing":false,
+        "prefill":{"initial_context_tokens":0},
+        "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS,"context_capacity_tokens":capacity},
+        "execution_order":runtime["execution_order"], "execution_layout":runtime["execution_layout"],
+        "measurement":runtime["measurement"], "cooldown_enabled":request.cooldown,
+        "tokenization_timed":false, "sampling_timed":false,
+        "runtime_protocol":runtime
+    });
+    Ok(())
+}
+
 /// A compatible harness must also have completed the work the user requested.
 /// Keep raw samples unchanged; reject partial runs instead of signing them.
 fn validate_requested_workloads(value: &Value, r: &BenchmarkRequest<'_>) -> Result<()> {
@@ -219,6 +381,70 @@ fn validate_requested_workloads(value: &Value, r: &BenchmarkRequest<'_>) -> Resu
 mod tests {
     use super::*;
 
+    #[test]
+    fn headline_capacity_is_capability_gated_and_does_not_change_warmup() {
+        assert!(!supports_headline_capacity(&json!({})).unwrap());
+        assert!(supports_headline_capacity(
+            &json!({"features":{"headline_context_capacity":true}})
+        )
+        .is_err());
+        assert!(
+            supports_headline_capacity(&json!({"features":{"headline_context_capacity":true},
+            "capacity_protocol_schema":crate::protocol::BASERT_CAPACITY_PROTOCOL_SCHEMA}))
+            .unwrap()
+        );
+        let request = BenchmarkRequest {
+            model: Path::new("test.base"),
+            pp: "128,512",
+            tg: 128,
+            reps: 3,
+            warmup: 0,
+            cooldown: false,
+        };
+        let original = json!({"params":{"ctx":4096},"protocol":{
+            "schema":"basert-throughput-protocol/2","profile":"basert-bench-capacity/1",
+            "context_isolation":"headline_then_per_prefill","context_capacity_policy":"basert_bench_default",
+            "model_load_in_timing":false,"execution_layout":"headline_then_prefill_processes",
+            "execution_order":["pp512","tg128","pp128"],
+            "prefill":{"128":{"initial_context_tokens":0,"context_capacity_tokens":4096},
+                       "512":{"initial_context_tokens":0,"context_capacity_tokens":4096}},
+            "decode":{"initial_context_tokens":1,"context_capacity_tokens":4096,"seed_prefill_in_timing":false},
+            "measurement":{"timed_repetitions":3,"requested_warmup_repetitions":0}
+        }});
+        let mut report = original.clone();
+        normalize_protocol(&mut report, &request, true, true).unwrap();
+        assert_eq!(
+            report["protocol"]["id"],
+            crate::protocol::HEADLINE_PROTOCOL_ID
+        );
+        assert_eq!(
+            report["protocol"]["measurement"]["requested_warmup_repetitions"],
+            0
+        );
+        assert_eq!(report["protocol"]["runtime_protocol"], original["protocol"]);
+        for (pointer, value) in [
+            ("/protocol/decode/initial_context_tokens", json!(4096)),
+            ("/protocol/decode/context_capacity_tokens", json!(129)),
+            ("/protocol/prefill/128/context_capacity_tokens", json!(128)),
+            ("/protocol/measurement/timed_repetitions", json!(5)),
+            (
+                "/protocol/measurement/requested_warmup_repetitions",
+                json!(12),
+            ),
+            (
+                "/protocol/execution_order",
+                json!(["pp128", "pp512", "tg128"]),
+            ),
+        ] {
+            let mut invalid = original.clone();
+            *invalid.pointer_mut(pointer).unwrap() = value;
+            assert!(
+                normalize_protocol(&mut invalid, &request, true, true).is_err(),
+                "{pointer}"
+            );
+        }
+    }
+
     #[test]
     fn replay_based_harnesses_use_external_observation() {
         let descriptor = json!({"telemetry_schema":"basert-telemetry/3",
@@ -245,4 +471,37 @@ mod tests {
             "features":{"same_run_telemetry":true}});
         assert!(telemetry_mode(&descriptor).is_err());
     }
+    #[test]
+    fn isolated_harness_metadata_normalizes_to_the_shared_protocol() {
+        let request = BenchmarkRequest {
+            model: Path::new("model.base"),
+            pp: "128,512",
+            tg: 128,
+            reps: 3,
+            warmup: 3,
+            cooldown: false,
+        };
+        let mut benchmark = json!({
+            "params":{},
+            "protocol":{
+                "schema":BASERT_ISOLATED_PROTOCOL_SCHEMA,
+                "context_isolation":"per_workload",
+                "context_capacity_policy":"minimum_required",
+                "model_load_in_timing":false,
+                "prefill":{
+                    "128":{"initial_context_tokens":0,"context_capacity_tokens":128},
+                    "512":{"initial_context_tokens":0,"context_capacity_tokens":512}
+                },
+                "decode":{"initial_context_tokens":1,"context_capacity_tokens":129}
+            }
+        });
+        normalize_protocol(&mut benchmark, &request, true, false).unwrap();
+        assert_eq!(benchmark["protocol"]["id"], THROUGHPUT_PROTOCOL_ID);
+        assert_eq!(benchmark["protocol"]["comparable"], true);
+        assert_eq!(benchmark["params"]["decode_context_tokens"], 1);
+        assert_eq!(
+            benchmark["protocol"]["runtime_protocol"]["decode"]["context_capacity_tokens"],
+            129
+        );
+    }
 }