diff --git a/README.md b/README.md index 09c4ff2..21e6158 100644 --- a/README.md +++ b/README.md @@ -155,12 +155,16 @@ and piped input must use `--yes`. Details are in [docs/benchmark-profiles.md](docs/benchmark-profiles.md). Telemetry is automatic for both runtimes and needs no flag, credential, or -sudo. ComputeArena observes the single process it launches for current BaseRT -and llama.cpp builds: resident memory, operating-system temperature sensors, -power state, and NVIDIA or ROCm device snapshots where those vendor tools exist. -A future BaseRT harness can advertise native same-run telemetry, which the CLI -will use without a CLI release or version-string rule. Coverage and limitations -are in [docs/telemetry.md](docs/telemetry.md). +sudo. ComputeArena observes the processes it launches: resident memory, +operating-system temperature sensors, power state, memory pressure and swap, +plus NVIDIA or ROCm device snapshots where those vendor tools exist. Signed +environment boundaries also record the OS/kernel, CPU layout, host memory, and +available GPU configuration; static macOS display configuration is cached so +`system_profiler` is not rerun at every boundary. A BaseRT harness can +advertise native same-run telemetry, which the CLI uses without a CLI release +or version-string rule. Runtime-native detail is preferred when it is more +accurate; portable host boundaries remain available for cross-runtime analysis. +Coverage and limitations are in [docs/telemetry.md](docs/telemetry.md). ## Runtimes @@ -214,10 +218,19 @@ reads its GGUF header only. The history lives in `recent-gguf.json` in the data directory, is never part of a report, and can be deleted to reset the list; a corrupt or unwritable history never blocks a benchmark. -llama.cpp measurements carry their own protocol identifiers and record native -warmup, zero context depth, and the exclusion of sampling and tokenization, so -they are never presented as BaseRT numbers. The measurement contract is in -[docs/runtime-adapters.md](docs/runtime-adapters.md). +The standard headline is PP512 followed by TG128 **before** the remaining PP +sweep. PP starts empty; TG starts with one untimed seed token. A capable BaseRT +harness reserves 4K for the headline (`computearena-throughput/3`), without +changing its existing warmup, repetitions, timing or telemetry. Older harnesses +retain their existing invocation and recorded protocol. + +ComputeArena uses the user's unmodified llama-bench build. Its native capacity +is retained: stock llama-bench has no independent 4K reservation option, and +`-d 4096` would add real history instead. This difference, actual workload order, +warmup and context requests are signed in the JSON. There is no claim of exact +cross-runtime equivalence. Existing reports remain verifiable and submittable; +protocol differences are not a new leaderboard filter. +The measurement contract is in [docs/runtime-adapters.md](docs/runtime-adapters.md). ## Reports and signatures @@ -334,12 +347,19 @@ reports, sessions, and the signing key. - Report envelope: `computearena-benchmark/1` - BaseRT harness output: `basert-benchmark-harness/1`; the older `basert-harness/1` is still accepted by the server -- llama.cpp measurements: `computearena-measurements/1`, executed as - `llama-bench-independent-pp-tg/1` or, with cooldown, - `llama-bench-conditioned-pp-tg/1` +- llama.cpp measurements: `computearena-measurements/1` +- Comparable throughput semantics: `computearena-throughput/2`, with native + evidence retained as `basert-throughput-protocol/1` or + `llama-bench-json/1`. Older BaseRT results use + `computearena-throughput-legacy/1` and are marked non-comparable. +- Headline-first BaseRT capacity: `computearena-throughput/3`, native evidence + `basert-throughput-protocol/2` / `basert-bench-capacity/1`. The historical + `comparable` metadata is not a guarantee of identical measured performance + and does not exclude older reports from the website. - Telemetry: `computearena-telemetry/1` for externally observed BaseRT and llama.cpp runs. A BaseRT harness advertising `features.same_run_telemetry` uses native `basert-telemetry/4` instead. +- Host environment boundaries: `computearena-environment/1` - Signing: Ed25519 over `computearena-json-v1` canonical JSON ## Development diff --git a/crates/computearena-cli/src/adapters/basert.rs b/crates/computearena-cli/src/adapters/basert.rs index 5261fc9..8335606 100644 --- a/crates/computearena-cli/src/adapters/basert.rs +++ b/crates/computearena-cli/src/adapters/basert.rs @@ -1,5 +1,8 @@ use super::{BenchmarkRequest, RuntimeAdapter, RuntimeOutput}; -use crate::protocol::BASERT_SAME_RUN_TELEMETRY_SCHEMA; +use crate::protocol::{ + BASERT_ISOLATED_PROTOCOL_SCHEMA, BASERT_SAME_RUN_TELEMETRY_SCHEMA, + DECODE_INITIAL_CONTEXT_TOKENS, THROUGHPUT_PROTOCOL_ID, +}; use crate::ui::TerminalUi; use anyhow::{bail, Context, Result}; use serde_json::{json, Value}; @@ -14,6 +17,27 @@ enum TelemetryMode { NativeSameRun, } +fn supports_isolated_workloads(descriptor: &Value) -> bool { + descriptor + .pointer("/features/isolated_workload_contexts") + .and_then(Value::as_bool) + == Some(true) +} + +fn supports_headline_capacity(descriptor: &Value) -> Result { + let supported = descriptor + .pointer("/features/headline_context_capacity") + .and_then(Value::as_bool) + == Some(true); + if supported + && descriptor["capacity_protocol_schema"].as_str() + != Some(crate::protocol::BASERT_CAPACITY_PROTOCOL_SCHEMA) + { + bail!("BaseRT advertises an unsupported headline context protocol"); + } + Ok(supported) +} + fn telemetry_mode(descriptor: &Value) -> Result { if descriptor .pointer("/features/same_run_telemetry") @@ -81,6 +105,8 @@ impl RuntimeAdapter for BaseRtAdapter { ) -> Result { let ui = TerminalUi::detect(); let mode = telemetry_mode(descriptor)?; + let isolated_workloads = supports_isolated_workloads(descriptor); + let headline_capacity = supports_headline_capacity(descriptor)?; let suite_conditioning = if mode == TelemetryMode::ExternalWholeProcess && r.cooldown { let mut cooldown = crate::conditioning::Cooldown::new(); Some(cooldown.prepare("BaseRT benchmark suite")) @@ -97,6 +123,13 @@ impl RuntimeAdapter for BaseRtAdapter { .arg(r.reps.to_string()) .arg("-w") .arg(r.warmup.to_string()); + if headline_capacity { + command.arg("--headline-first"); + } else if isolated_workloads { + command.arg("--isolated-workloads"); + } + let native_environment_before = + (mode == TelemetryMode::NativeSameRun).then(crate::telemetry::capture_environment); let (output, external_telemetry) = match mode { TelemetryMode::ExternalWholeProcess => { println!("{}", ui.neutral("Telemetry: observing the BaseRT harness process and available device sensors (whole run, 1-second sampling; no telemetry replays).")); @@ -119,6 +152,10 @@ impl RuntimeAdapter for BaseRtAdapter { if !output.status.success() { bail!("BaseRT benchmark exited with {}", output.status); } + let native_environment = native_environment_before.map(|before| { + json!({"schema":"computearena-environment/1","measurement_relation":"outside_runtime_execution", + "before":before,"after":crate::telemetry::capture_environment()}) + }); let mut benchmark: Value = serde_json::from_slice(&output.stdout).context("BaseRT returned invalid JSON")?; let expected_schema = match mode { @@ -127,6 +164,10 @@ impl RuntimeAdapter for BaseRtAdapter { }; crate::benchmark::validate_harness_result(&benchmark, expected_schema)?; validate_requested_workloads(&benchmark, r)?; + normalize_protocol(&mut benchmark, r, isolated_workloads, headline_capacity)?; + if let Some(environment) = native_environment { + crate::telemetry::attach_environment(&mut benchmark, environment); + } if let Some(mut telemetry) = external_telemetry { let mut policy = if r.cooldown { crate::conditioning::before_suite_policy() @@ -162,6 +203,127 @@ impl RuntimeAdapter for BaseRtAdapter { } } +fn normalize_protocol( + benchmark: &mut Value, + request: &BenchmarkRequest<'_>, + isolated_workloads: bool, + headline_capacity: bool, +) -> Result<()> { + let runtime_protocol = benchmark.get("protocol").cloned().unwrap_or(Value::Null); + benchmark["params"]["decode_context_tokens"] = json!(DECODE_INITIAL_CONTEXT_TOKENS); + if headline_capacity { + return normalize_headline_protocol(benchmark, request, runtime_protocol); + } + if !isolated_workloads { + benchmark["protocol"] = json!({ + "id":"computearena-throughput-legacy/1", + "comparable":false, + "reason":"The installed BaseRT harness does not advertise isolated workload contexts", + "context_isolation":"shared_sweep_context", + "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS}, + "runtime_protocol":runtime_protocol + }); + return Ok(()); + } + + if runtime_protocol["schema"].as_str() != Some(BASERT_ISOLATED_PROTOCOL_SCHEMA) + || runtime_protocol["context_isolation"].as_str() != Some("per_workload") + || runtime_protocol["context_capacity_policy"].as_str() != Some("minimum_required") + || runtime_protocol["model_load_in_timing"].as_bool() != Some(false) + { + bail!("BaseRT advertised isolated contexts but returned incompatible protocol metadata"); + } + for prompt in request.pp.split(',') { + let tokens = prompt.parse::()?; + let workload = &runtime_protocol["prefill"][prompt]; + if workload["initial_context_tokens"].as_u64() != Some(0) + || workload["context_capacity_tokens"].as_u64() != Some(tokens) + { + bail!("BaseRT returned incompatible PP{tokens} context metadata"); + } + } + if runtime_protocol["decode"]["initial_context_tokens"].as_u64() + != Some(DECODE_INITIAL_CONTEXT_TOKENS) + || runtime_protocol["decode"]["context_capacity_tokens"].as_u64() + != Some(u64::from(request.tg) + DECODE_INITIAL_CONTEXT_TOKENS) + { + bail!( + "BaseRT returned incompatible TG{} context metadata", + request.tg + ); + } + benchmark["protocol"] = json!({ + "id":THROUGHPUT_PROTOCOL_ID, + "comparable":true, + "context_isolation":"per_workload", + "context_capacity_policy":"minimum_required", + "model_load_in_timing":false, + "prefill":{"initial_context_tokens":0}, + "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS}, + "tokenization_timed":false, + "sampling_timed":false, + "runtime_protocol":runtime_protocol + }); + Ok(()) +} + +fn normalize_headline_protocol( + benchmark: &mut Value, + request: &BenchmarkRequest<'_>, + runtime: Value, +) -> Result<()> { + use crate::protocol::{ + headline_order, headline_prefill, BASERT_CAPACITY_PROTOCOL_SCHEMA, + HEADLINE_CONTEXT_CAPACITY, HEADLINE_PROTOCOL_ID, + }; + let headline = headline_prefill(request.pp).parse::()?; + let capacity = (headline + u64::from(request.tg)).max(HEADLINE_CONTEXT_CAPACITY); + if runtime["schema"] != BASERT_CAPACITY_PROTOCOL_SCHEMA + || runtime["profile"] != "basert-bench-capacity/1" + || runtime["context_isolation"] != "headline_then_per_prefill" + || runtime["context_capacity_policy"] != "basert_bench_default" + || runtime["model_load_in_timing"] != false + || runtime["execution_layout"] != "headline_then_prefill_processes" + || runtime["execution_order"] != json!(headline_order(request.pp, request.tg)) + { + bail!("BaseRT returned incompatible headline capacity/order metadata"); + } + for prompt in request.pp.split(',') { + let tokens = prompt.parse::()?; + let expected = if tokens == headline { + capacity + } else { + tokens.max(HEADLINE_CONTEXT_CAPACITY) + }; + if runtime["prefill"][prompt]["initial_context_tokens"] != 0 + || runtime["prefill"][prompt]["context_capacity_tokens"].as_u64() != Some(expected) + { + bail!("BaseRT returned incompatible PP{tokens} capacity metadata"); + } + } + if runtime["decode"]["initial_context_tokens"] != DECODE_INITIAL_CONTEXT_TOKENS + || runtime["decode"]["context_capacity_tokens"] != capacity + || runtime["decode"]["seed_prefill_in_timing"] != false + || runtime["measurement"]["timed_repetitions"] != request.reps + || runtime["measurement"]["requested_warmup_repetitions"] != request.warmup + || benchmark["params"]["ctx"] != capacity + { + bail!("BaseRT returned incompatible headline measurement metadata"); + } + benchmark["protocol"] = json!({ + "id":HEADLINE_PROTOCOL_ID, "comparable":true, + "profile":"headline-first-capacity/1", "context_isolation":"headline_then_per_prefill", + "context_capacity_policy":"basert_bench_default", "model_load_in_timing":false, + "prefill":{"initial_context_tokens":0}, + "decode":{"initial_context_tokens":DECODE_INITIAL_CONTEXT_TOKENS,"context_capacity_tokens":capacity}, + "execution_order":runtime["execution_order"], "execution_layout":runtime["execution_layout"], + "measurement":runtime["measurement"], "cooldown_enabled":request.cooldown, + "tokenization_timed":false, "sampling_timed":false, + "runtime_protocol":runtime + }); + Ok(()) +} + /// A compatible harness must also have completed the work the user requested. /// Keep raw samples unchanged; reject partial runs instead of signing them. fn validate_requested_workloads(value: &Value, r: &BenchmarkRequest<'_>) -> Result<()> { @@ -219,6 +381,70 @@ fn validate_requested_workloads(value: &Value, r: &BenchmarkRequest<'_>) -> Resu mod tests { use super::*; + #[test] + fn headline_capacity_is_capability_gated_and_does_not_change_warmup() { + assert!(!supports_headline_capacity(&json!({})).unwrap()); + assert!(supports_headline_capacity( + &json!({"features":{"headline_context_capacity":true}}) + ) + .is_err()); + assert!( + supports_headline_capacity(&json!({"features":{"headline_context_capacity":true}, + "capacity_protocol_schema":crate::protocol::BASERT_CAPACITY_PROTOCOL_SCHEMA})) + .unwrap() + ); + let request = BenchmarkRequest { + model: Path::new("test.base"), + pp: "128,512", + tg: 128, + reps: 3, + warmup: 0, + cooldown: false, + }; + let original = json!({"params":{"ctx":4096},"protocol":{ + "schema":"basert-throughput-protocol/2","profile":"basert-bench-capacity/1", + "context_isolation":"headline_then_per_prefill","context_capacity_policy":"basert_bench_default", + "model_load_in_timing":false,"execution_layout":"headline_then_prefill_processes", + "execution_order":["pp512","tg128","pp128"], + "prefill":{"128":{"initial_context_tokens":0,"context_capacity_tokens":4096}, + "512":{"initial_context_tokens":0,"context_capacity_tokens":4096}}, + "decode":{"initial_context_tokens":1,"context_capacity_tokens":4096,"seed_prefill_in_timing":false}, + "measurement":{"timed_repetitions":3,"requested_warmup_repetitions":0} + }}); + let mut report = original.clone(); + normalize_protocol(&mut report, &request, true, true).unwrap(); + assert_eq!( + report["protocol"]["id"], + crate::protocol::HEADLINE_PROTOCOL_ID + ); + assert_eq!( + report["protocol"]["measurement"]["requested_warmup_repetitions"], + 0 + ); + assert_eq!(report["protocol"]["runtime_protocol"], original["protocol"]); + for (pointer, value) in [ + ("/protocol/decode/initial_context_tokens", json!(4096)), + ("/protocol/decode/context_capacity_tokens", json!(129)), + ("/protocol/prefill/128/context_capacity_tokens", json!(128)), + ("/protocol/measurement/timed_repetitions", json!(5)), + ( + "/protocol/measurement/requested_warmup_repetitions", + json!(12), + ), + ( + "/protocol/execution_order", + json!(["pp128", "pp512", "tg128"]), + ), + ] { + let mut invalid = original.clone(); + *invalid.pointer_mut(pointer).unwrap() = value; + assert!( + normalize_protocol(&mut invalid, &request, true, true).is_err(), + "{pointer}" + ); + } + } + #[test] fn replay_based_harnesses_use_external_observation() { let descriptor = json!({"telemetry_schema":"basert-telemetry/3", @@ -245,4 +471,37 @@ mod tests { "features":{"same_run_telemetry":true}}); assert!(telemetry_mode(&descriptor).is_err()); } + #[test] + fn isolated_harness_metadata_normalizes_to_the_shared_protocol() { + let request = BenchmarkRequest { + model: Path::new("model.base"), + pp: "128,512", + tg: 128, + reps: 3, + warmup: 3, + cooldown: false, + }; + let mut benchmark = json!({ + "params":{}, + "protocol":{ + "schema":BASERT_ISOLATED_PROTOCOL_SCHEMA, + "context_isolation":"per_workload", + "context_capacity_policy":"minimum_required", + "model_load_in_timing":false, + "prefill":{ + "128":{"initial_context_tokens":0,"context_capacity_tokens":128}, + "512":{"initial_context_tokens":0,"context_capacity_tokens":512} + }, + "decode":{"initial_context_tokens":1,"context_capacity_tokens":129} + } + }); + normalize_protocol(&mut benchmark, &request, true, false).unwrap(); + assert_eq!(benchmark["protocol"]["id"], THROUGHPUT_PROTOCOL_ID); + assert_eq!(benchmark["protocol"]["comparable"], true); + assert_eq!(benchmark["params"]["decode_context_tokens"], 1); + assert_eq!( + benchmark["protocol"]["runtime_protocol"]["decode"]["context_capacity_tokens"], + 129 + ); + } } diff --git a/crates/computearena-cli/src/adapters/llama_cpp.rs b/crates/computearena-cli/src/adapters/llama_cpp.rs index a842966..554319e 100644 --- a/crates/computearena-cli/src/adapters/llama_cpp.rs +++ b/crates/computearena-cli/src/adapters/llama_cpp.rs @@ -85,12 +85,12 @@ impl RuntimeAdapter for LlamaCppAdapter { ) -> Result { validate_model(r.model)?; let ui = TerminalUi::detect(); - println!("{}", ui.neutral("Telemetry: observing process memory and available device sensors (whole run, 1-second sampling).")); + println!("{}", ui.neutral("Telemetry: observing each runtime process and available device sensors (1-second sampling).")); let (rows, telemetry) = if r.cooldown { let mut cooldown = crate::conditioning::Cooldown::new(); run_conditioned(executable, r, |label| cooldown.prepare(label))? } else { - run_native(executable, r, r.pp, r.tg)? + run_unconditioned(executable, r)? }; let mut result = normalize(&rows, r)?; ui.section("Benchmark results"); @@ -111,8 +111,7 @@ impl RuntimeAdapter for LlamaCppAdapter { ); result.benchmark["protocol"]["telemetry_available"] = json!(telemetry.get("observer").is_some() || telemetry.get("workloads").is_some()); - result.benchmark["protocol"]["measurement_observer"] = - json!("external_whole_process_sampler"); + result.benchmark["protocol"]["measurement_observer"] = json!("external_process_sampler"); if let Some(peak) = crate::telemetry::attach_whole_process(&mut result.benchmark, telemetry) { println!("{}", ui.neutral(format!("Telemetry: observed peak process memory {peak:.0} MiB (includes loading and warmup)."))); @@ -120,7 +119,6 @@ impl RuntimeAdapter for LlamaCppAdapter { println!("{}", ui.neutral("Telemetry: process memory unavailable; see sensor coverage in the saved report.")); } if r.cooldown { - result.benchmark["protocol"]["id"] = json!("llama-bench-conditioned-pp-tg/1"); result.benchmark["protocol"]["cooldown_enabled"] = json!(true); result.benchmark["protocol"]["execution_layout"] = json!("one_process_per_workload"); result.benchmark["protocol"]["conditioning"] = @@ -144,6 +142,7 @@ fn run_native( r: &BenchmarkRequest<'_>, pp: &str, tg: u32, + depth: u32, ) -> Result<(Value, Value)> { let mut command = Command::new(executable); command @@ -151,7 +150,9 @@ fn run_native( .arg(r.model) .args(["-p", pp, "-n"]) .arg(tg.to_string()) - .args(["-d", "0", "-r"]) + .arg("-d") + .arg(depth.to_string()) + .arg("-r") .arg(r.reps.to_string()) .args(["-o", "json"]); if r.warmup == 0 { @@ -170,22 +171,128 @@ fn run_native( Ok((rows, telemetry)) } +fn aggregate_observations( + scope: &str, + ordered: Vec<(String, Value)>, + conditioning: Option>, +) -> Value { + let mut observations = Map::new(); + let mut peak: Option = None; + let mut before = None; + let mut after = None; + for (label, mut telemetry) in ordered { + if before.is_none() { + before = telemetry.pointer("/boundaries/before").cloned(); + } + after = telemetry.pointer("/boundaries/after").cloned(); + if let Some(value) = telemetry + .pointer("/process_memory/statistics/peak") + .and_then(Value::as_f64) + { + peak = Some(peak.map_or(value, |current| current.max(value))); + } + telemetry["scope"] = json!("one_runtime_process"); + telemetry["includes"] = json!([ + "model_loading", + "warmup", + "requested_workloads_for_this_process", + "runtime_teardown" + ]); + observations.insert(label, telemetry); + } + let mut telemetry = json!({ + "schema":"computearena-telemetry/1", + "coverage":"basic", + "scope":scope, + "measurement_relation":"concurrent_observer", + "workloads":observations, + "process_memory":{ + "metric":"resident_set_size", + "unit":"MiB", + "statistics":{"available":peak.is_some(),"peak":peak}, + "note":"Maximum observed peak across separate process windows including loading and warmup" + }, + "boundaries":{ + "schema":"computearena-environment/1", + "measurement_relation":"outside_runtime_execution", + "before":before, + "after":after + } + }); + if let Some(conditioning) = conditioning { + telemetry["conditioning"] = crate::conditioning::policy(); + telemetry["conditioning_workloads"] = Value::Object(conditioning); + } + telemetry +} + +fn run_unconditioned(executable: &Path, r: &BenchmarkRequest<'_>) -> Result<(Value, Value)> { + let headline = crate::protocol::headline_prefill(r.pp); + let (prefill, prefill_telemetry) = run_native(executable, r, headline, 0, 0)?; + let (decode, decode_telemetry) = run_native( + executable, + r, + "0", + r.tg, + crate::protocol::DECODE_INITIAL_CONTEXT_TOKENS as u32, + )?; + let mut rows = prefill + .as_array() + .context("llama.cpp prefill output must be an array")? + .clone(); + rows.extend( + decode + .as_array() + .context("llama.cpp decode output must be an array")? + .iter() + .cloned(), + ); + let mut observations = vec![ + (format!("pp{headline}"), prefill_telemetry), + (format!("tg{}", r.tg), decode_telemetry), + ]; + let remaining = + r.pp.split(',') + .filter(|pp| *pp != headline) + .collect::>() + .join(","); + if !remaining.is_empty() { + let (sweep, telemetry) = run_native(executable, r, &remaining, 0, 0)?; + rows.extend( + sweep + .as_array() + .context("llama.cpp sweep output must be an array")? + .iter() + .cloned(), + ); + observations.push(("remaining_prefill_sweep".into(), telemetry)); + } + let telemetry = aggregate_observations("headline_then_prefill_processes", observations, None); + Ok((Value::Array(rows), telemetry)) +} + fn run_conditioned( executable: &Path, r: &BenchmarkRequest<'_>, mut prepare: impl FnMut(&str) -> Value, ) -> Result<(Value, Value)> { - let schedule: Vec<(u32, u32)> = - r.pp.split(',') - .map(|v| v.parse::().map(|pp| (pp, 0))) - .chain(std::iter::once(Ok((0, r.tg)))) - .collect::>()?; + let headline = crate::protocol::headline_prefill(r.pp); + let mut schedule = vec![ + (headline.parse::()?, 0, 0), + ( + 0, + r.tg, + crate::protocol::DECODE_INITIAL_CONTEXT_TOKENS as u32, + ), + ]; + for pp in r.pp.split(',').filter(|pp| *pp != headline) { + schedule.push((pp.parse::()?, 0, 0)); + } let mut rows = Vec::new(); - let mut observations = Map::new(); + let mut observations = Vec::new(); let mut conditioning = Map::new(); - let mut peak: Option = None; let ui = TerminalUi::detect(); - for (index, (pp, tg)) in schedule.iter().enumerate() { + for (index, (pp, tg, depth)) in schedule.iter().enumerate() { let label = if *pp > 0 { format!("pp{pp}") } else { @@ -208,33 +315,27 @@ fn run_conditioned( r.reps )) ); - let (native, telemetry) = run_native(executable, r, &pp.to_string(), *tg)?; + let (native, telemetry) = run_native(executable, r, &pp.to_string(), *tg, *depth)?; let native = native .as_array() .context("llama.cpp benchmark output must be an array")?; if native.len() != 1 || native[0]["n_prompt"].as_u64() != Some(u64::from(*pp)) || native[0]["n_gen"].as_u64() != Some(u64::from(*tg)) + || native[0]["n_depth"].as_u64() != Some(u64::from(*depth)) { bail!("llama.cpp returned an unexpected workload for {label}; no report was signed"); } rows.extend(native.iter().cloned()); - if let Some(value) = telemetry - .pointer("/process_memory/statistics/peak") - .and_then(Value::as_f64) - { - peak = Some(peak.map_or(value, |p| p.max(value))); - } - observations.insert(label, telemetry); + observations.push((label, telemetry)); } Ok(( Value::Array(rows), - json!({"schema":"computearena-telemetry/1","coverage":"basic", - "scope":"separate_workload_processes","measurement_relation":"concurrent_observer", - "conditioning":crate::conditioning::policy(),"conditioning_workloads":conditioning, - "workloads":observations,"process_memory":{"metric":"resident_set_size","unit":"MiB", - "statistics":{"available":peak.is_some(),"peak":peak}, - "note":"Maximum observed peak across separate process windows including loading and warmup"}}), + aggregate_observations( + "separate_workload_processes", + observations, + Some(conditioning), + ), )) } @@ -297,6 +398,8 @@ pub(crate) fn normalize(value: &Value, r: &BenchmarkRequest<'_>) -> Result) -> Result 0 && tg == 0; + let label = if is_pp { + format!("pp{pp}") + } else { + format!("tg{tg}") + }; + execution_order.push(label.clone()); + let expected_depth = if is_pp { + 0 + } else { + crate::protocol::DECODE_INITIAL_CONTEXT_TOKENS + }; + if row["n_depth"].as_u64() != Some(expected_depth) { + bail!("llama.cpp returned an unexpected context depth"); + } + // Stock llama-bench derives the requested context from PP+TG+depth. + // The engine may pad allocations; never claim unreported physical capacity. + context_requests.insert( + label, + json!({ + "initial_context_tokens":expected_depth, + "requested_context_capacity_tokens":pp+tg+expected_depth, + "capacity_source":"llama_bench_native_workload_parameters" + }), + ); if !(is_pp && expected.contains(&pp) || pp == 0 && tg == u64::from(r.tg)) { bail!("llama.cpp returned an unexpected workload PP{pp}/TG{tg}"); } @@ -411,12 +535,24 @@ pub(crate) fn normalize(value: &Value, r: &BenchmarkRequest<'_>) -> Result &str { + pp.split(',') + .find(|size| *size == "512") + .unwrap_or_else(|| pp.split(',').next().unwrap_or("512")) +} + +pub(crate) fn headline_order(pp: &str, tg: u32) -> Vec { + let headline = headline_prefill(pp); + let mut order = vec![format!("pp{headline}"), format!("tg{tg}")]; + order.extend( + pp.split(',') + .filter(|size| *size != headline) + .map(|size| format!("pp{size}")), + ); + order +} +pub(crate) const DECODE_INITIAL_CONTEXT_TOKENS: u64 = 1; pub(crate) const SIGNATURE_DOMAIN: &[u8] = b"computearena-benchmark/1\0"; pub(crate) const SIGNATURE_ALGORITHM: &str = "ed25519"; diff --git a/crates/computearena-cli/src/telemetry.rs b/crates/computearena-cli/src/telemetry.rs index 84aa7c6..3c06a02 100644 --- a/crates/computearena-cli/src/telemetry.rs +++ b/crates/computearena-cli/src/telemetry.rs @@ -166,7 +166,7 @@ impl Sampler { eprintln!( "{}", crate::ui::TerminalUi::detect().neutral(format!( - "Benchmark running — {}s elapsed; {memory} (whole run)", + "Benchmark running — {}s elapsed; {memory} (runtime process)", started.elapsed().as_secs() )) ); @@ -231,13 +231,32 @@ fn unavailable(reason: &str) -> Value { /// Attach runtime-neutral whole-process telemetry and retain the compatibility /// memory field consumed by existing local summaries and the web projection. /// Detailed scope and limitations remain on the telemetry object. +pub(crate) fn capture_environment() -> Value { + snapshots::capture() +} + +pub(crate) fn attach_environment(benchmark: &mut Value, environment: Value) { + benchmark["environment"] = environment; +} + pub(crate) fn attach_whole_process(benchmark: &mut Value, telemetry: Value) -> Option { + if let Some(environment) = telemetry.get("boundaries").cloned() { + attach_environment(benchmark, environment); + } + let memory_scope = if telemetry["scope"] + .as_str() + .is_some_and(|scope| scope.starts_with("separate_")) + { + "all_runtime_processes" + } else { + "whole_runtime_process" + }; let peak = telemetry .pointer("/process_memory/statistics/peak") .and_then(Value::as_f64); if let Some(peak) = peak { benchmark["memory"] = json!({"process_peak_rss_mb":peak, - "measurement_relation":"concurrent_observer","scope":"whole_runtime_process", + "measurement_relation":"concurrent_observer","scope":memory_scope, "unit":"MiB","note":"Observed sampled peak, including loading and warmup; not a kernel high-water mark"}); } benchmark["telemetry"] = telemetry; @@ -264,7 +283,8 @@ pub(crate) fn run_observed(command: &mut Command) -> Result<(Output, Value)> { Ok(sampler) => sampler.finish(), Err(error) => unavailable(&format!("Could not start telemetry worker: {error}")), }; - telemetry["boundaries"] = json!({"measurement_relation":"outside_runtime_execution", + telemetry["boundaries"] = json!({"schema":"computearena-environment/1", + "measurement_relation":"outside_runtime_execution", "before":before,"after":snapshots::capture()}); Ok((output?, telemetry)) } diff --git a/crates/computearena-cli/src/telemetry/snapshots.rs b/crates/computearena-cli/src/telemetry/snapshots.rs index ab33439..096d1bf 100644 --- a/crates/computearena-cli/src/telemetry/snapshots.rs +++ b/crates/computearena-cli/src/telemetry/snapshots.rs @@ -5,8 +5,11 @@ use std::fs; use std::io::{Read, Seek, SeekFrom}; use std::path::Path; use std::process::{Command, Stdio}; +#[cfg(target_os = "macos")] +use std::sync::OnceLock; use std::thread; use std::time::{Duration, Instant}; +use sysinfo::System; const PROBE_TIMEOUT: Duration = Duration::from_secs(2); const MAX_OUTPUT: u64 = 65_536; @@ -55,6 +58,132 @@ fn number(value: &str) -> Option { .filter(|v| v.is_finite() && *v >= 0.0) } +fn mebibytes(bytes: u64) -> f64 { + bytes as f64 / 1_048_576.0 +} + +fn operating_system() -> Value { + json!({ + "family": std::env::consts::OS, + "architecture": std::env::consts::ARCH, + "version": System::long_os_version(), + "kernel_version": System::kernel_version() + }) +} + +fn host_resources() -> Value { + let system = System::new_all(); + let total_memory = system.total_memory(); + let available_memory = system.available_memory(); + json!({ + "logical_cpu_count": system.cpus().len(), + "physical_cpu_count": System::physical_core_count(), + "memory": { + "unit": "MiB", + "total": mebibytes(total_memory), + "available": mebibytes(available_memory), + "available_percent": if total_memory > 0 { + Some(available_memory as f64 * 100.0 / total_memory as f64) + } else { + None + } + }, + "swap": { + "unit": "MiB", + "total": mebibytes(system.total_swap()), + "used": mebibytes(system.used_swap()), + "active": system.used_swap() > 0 + } + }) +} + +#[cfg(target_os = "macos")] +fn memory_pressure() -> Value { + let text = query( + Path::new("/usr/bin/memory_pressure"), + &["-Q"], + PROBE_TIMEOUT, + ); + let free_percent = text.as_deref().and_then(|text| { + text.lines().find_map(|line| { + line.strip_prefix("System-wide memory free percentage:") + .and_then(|value| number(value.trim_end_matches('%'))) + }) + }); + json!({"provider":"memory_pressure","available":free_percent.is_some(), + "system_free_percent":free_percent, + "reason_if_unavailable":"memory_pressure -Q failed or returned unsupported output"}) +} + +#[cfg(target_os = "linux")] +fn memory_pressure() -> Value { + let text = fs::read_to_string("/proc/pressure/memory").ok(); + let mut classes = Map::new(); + if let Some(text) = text.as_deref() { + for line in text.lines() { + let mut fields = line.split_whitespace(); + let Some(class) = fields.next() else { + continue; + }; + let mut values = Map::new(); + for field in fields { + let Some((key, value)) = field.split_once('=') else { + continue; + }; + if let Some(value) = number(value) { + values.insert(key.to_owned(), json!(value)); + } + } + classes.insert(class.to_owned(), Value::Object(values)); + } + } + json!({"provider":"linux_psi","available":!classes.is_empty(),"classes":classes, + "reason_if_unavailable":"/proc/pressure/memory is unavailable"}) +} + +#[cfg(not(any(target_os = "macos", target_os = "linux")))] +fn memory_pressure() -> Value { + json!({"available":false,"reason":"No memory-pressure provider for this OS yet"}) +} + +#[cfg(target_os = "macos")] +fn apple_accelerators() -> Vec { + static DEVICES: OnceLock> = OnceLock::new(); + DEVICES + .get_or_init(|| { + let Some(text) = query( + Path::new("/usr/sbin/system_profiler"), + &["SPDisplaysDataType", "-json"], + PROBE_TIMEOUT, + ) else { + return Vec::new(); + }; + let Ok(value) = serde_json::from_str::(&text) else { + return Vec::new(); + }; + value["SPDisplaysDataType"] + .as_array() + .into_iter() + .flatten() + .take(32) + .map(|display| { + json!({ + "name": display["sppci_model"].as_str().or_else(|| display["_name"].as_str()), + "core_count": display["sppci_cores"].as_str().and_then(|value| value.parse::().ok()), + "device_type": display["sppci_device_type"].as_str(), + "metal_support": display["spdisplays_metal"].as_str() + }) + }) + .collect() + }) + .clone() +} + +#[cfg(not(target_os = "macos"))] +fn apple_accelerators() -> Vec { + Vec::new() +} + fn nvidia(text: &str) -> Vec { text.lines() .filter_map(|line| { @@ -114,6 +243,11 @@ fn rocm(text: &str) -> Vec { fn accelerators() -> Value { let mut providers = Vec::new(); + let apple_devices = apple_accelerators(); + if !apple_devices.is_empty() { + providers + .push(json!({"provider":"system_profiler","available":true,"devices":apple_devices})); + } for (program, args) in [ ("nvidia-smi", vec!["--query-gpu=index,name,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu","--format=csv,noheader,nounits"]), ("rocm-smi", vec!["--showtemp","--showpower","--showuse","--showmeminfo","vram","--json"]), @@ -142,20 +276,39 @@ fn power_state() -> Value { } }); let settings = query(Path::new("/usr/bin/pmset"), &["-g"], PROBE_TIMEOUT); - let low_power = settings.as_deref().and_then(|text| { - text.lines().find_map(|line| { - let mut fields = line.split_whitespace(); - if fields.next()? != "lowpowermode" { - return None; - } - match fields.next()? { - "1" => Some(true), - "0" => Some(false), - _ => None, - } + let setting = |name: &str| { + settings.as_deref().and_then(|text| { + text.lines().find_map(|line| { + let mut fields = line.split_whitespace(); + (fields.next()? == name).then(|| fields.next()).flatten() + }) }) + }; + let low_power = setting("lowpowermode").and_then(|value| match value { + "1" => Some(true), + "0" => Some(false), + _ => None, + }); + let power_mode = setting("powermode").and_then(|value| value.parse::().ok()); + let high_power = setting("highpowermode").and_then(|value| match value { + "1" => Some(true), + "0" => Some(false), + _ => None, }); - json!({"provider":"pmset","power_source":source,"low_power_mode":low_power}) + let performance_mode = if low_power == Some(true) || power_mode == Some(1) { + Some("low_power") + } else if high_power == Some(true) || power_mode == Some(2) { + Some("high_power") + } else if power_mode == Some(0) { + Some("automatic") + } else if low_power == Some(false) { + Some("standard_or_automatic") + } else { + None + }; + json!({"provider":"pmset","power_source":source,"low_power_mode":low_power, + "performance_mode":performance_mode,"performance_mode_raw":power_mode, + "high_power_mode":high_power}) } #[cfg(target_os = "linux")] @@ -188,9 +341,14 @@ fn power_state() -> Value { pub(super) fn capture() -> Value { let started = Instant::now(); + let operating_system = operating_system(); + let host = host_resources(); + let pressure = memory_pressure(); let power = power_state(); let accelerators = accelerators(); - json!({"power_state":power,"accelerators":accelerators,"probe_elapsed_ms":started.elapsed().as_secs_f64()*1000.0}) + json!({"operating_system":operating_system,"host":host,"memory_pressure":pressure, + "power_state":power,"accelerators":accelerators, + "probe_elapsed_ms":started.elapsed().as_secs_f64()*1000.0}) } #[cfg(test)] @@ -217,4 +375,15 @@ mod tests { assert!(query(Path::new("/bin/sleep"), &["2"], Duration::from_millis(30)).is_none()); assert!(start.elapsed() < Duration::from_secs(1)); } + #[test] + fn host_snapshot_reports_os_memory_swap_and_cpu_shape() { + let os = operating_system(); + let host = host_resources(); + assert_eq!(os["family"], std::env::consts::OS); + assert_eq!(os["architecture"], std::env::consts::ARCH); + assert!(host["logical_cpu_count"].as_u64().unwrap_or(0) > 0); + assert!(host["memory"]["total"].as_f64().unwrap_or(0.0) > 0.0); + assert_eq!(host["swap"]["unit"], "MiB"); + assert!(host["swap"]["active"].is_boolean()); + } } diff --git a/crates/computearena-cli/tests/adapter_contract.rs b/crates/computearena-cli/tests/adapter_contract.rs index 997f5e9..5e05dd5 100644 --- a/crates/computearena-cli/tests/adapter_contract.rs +++ b/crates/computearena-cli/tests/adapter_contract.rs @@ -68,9 +68,10 @@ fn telemetry_is_collected_for_the_child_summarized_and_signature_protected() { let mut report = f.signed(); let telemetry = &report["benchmark"]["telemetry"]; assert_eq!(telemetry["schema"], "computearena-telemetry/1"); - assert_eq!(telemetry["observer"]["requested_interval_ms"], 1000); + let observed = &telemetry["workloads"]["pp512"]; + assert_eq!(observed["observer"]["requested_interval_ms"], 1000); assert!( - telemetry["process_memory"]["statistics"]["sample_count"] + observed["process_memory"]["statistics"]["sample_count"] .as_u64() .unwrap() >= 1 @@ -79,13 +80,14 @@ fn telemetry_is_collected_for_the_child_summarized_and_signature_protected() { report["benchmark"]["memory"]["process_peak_rss_mb"], telemetry["process_memory"]["statistics"]["peak"] ); - assert_eq!(telemetry["energy"]["available"], false); - assert_eq!(telemetry["per_workload"]["available"], false); + assert_eq!(observed["energy"]["available"], false); + assert_eq!(observed["per_workload"]["available"], false); success(&f.verify()); if let Some(path) = std::env::var_os("COMPUTEARENA_TELEMETRY_TEST_REPORT") { fs::copy(&f.report, path).unwrap(); } - report["benchmark"]["telemetry"]["observer"]["requested_interval_ms"] = json!(25); + report["benchmark"]["telemetry"]["workloads"]["pp512"]["observer"]["requested_interval_ms"] = + json!(25); fs::write(&f.report, serde_json::to_vec_pretty(&report).unwrap()).unwrap(); failure(&f.verify(), "signature verification failed"); } @@ -393,7 +395,7 @@ impl Fixture { Value::Array([(128,0),(512,0),(0,128)].into_iter().map(|(pp,tg)| json!({ "build_commit":"abc123","build_number":123,"model_type":"Qwen3 Q4_K_M", "model_filename":self.model,"model_size":24,"model_n_params":4000000000u64, - "n_prompt":pp,"n_gen":tg,"n_depth":0,"n_gpu_layers":0,"backends":"CPU", + "n_prompt":pp,"n_gen":tg,"n_depth":if pp == 0 { 1 } else { 0 },"n_gpu_layers":0,"backends":"CPU", "cpu_info":"Test CPU","gpu_info":"","samples_ns":[100000000,200000000], // Reported aggregates are deliberately bogus: native samples are authoritative. "avg_ts":999999.0,"avg_ns":1 @@ -412,8 +414,45 @@ impl Fixture { let descriptor = json!({"schema":"basert-benchmark-harness-descriptor/1", "runtime":{"name":"basert","version":"0.2.4"},"result_schema":"basert-benchmark-harness/1", "telemetry_schema":telemetry_schema, - "features":{"telemetry":true,"same_run_telemetry":native_same_run}}); - let script = format!("#!/bin/sh\ncase \"$1\" in\n describe) printf '%s\\n' '{descriptor}';;\n --help) printf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json';;\n *) printf '%s\\n' \"$@\" > \"$ARENA_TEST_ARGS\"\n{before_result}\n/bin/cat <<'RESULT'\n{result}\nRESULT\n;;\nesac\n"); + "capacity_protocol_schema":"basert-throughput-protocol/2", + "features":{"telemetry":true,"same_run_telemetry":native_same_run, + "headline_context_capacity":result.pointer("/protocol/schema").and_then(Value::as_str)==Some("basert-throughput-protocol/2")}}); + let result_script = if self.runtime == "llama-cpp" { + let rows = result.as_array().unwrap(); + let prefill: Value = rows + .iter() + .filter(|row| row["n_prompt"].as_u64().unwrap_or(0) > 0) + .cloned() + .collect(); + let decode: Value = rows + .iter() + .filter(|row| row["n_gen"].as_u64().unwrap_or(0) > 0) + .cloned() + .collect(); + // This fixture has PP128 and PP512. Keep their original row + // positions even when a test deliberately corrupts a token count. + let headline: Value = prefill + .as_array() + .unwrap() + .iter() + .skip(1) + .take(1) + .cloned() + .collect(); + let remaining: Value = prefill + .as_array() + .unwrap() + .iter() + .take(1) + .cloned() + .collect(); + format!( + "tg=0\npp=0\nwhile [ \"$#\" -gt 0 ]; do\ncase \"$1\" in\n-n) shift; tg=\"$1\";;\n-p) shift; pp=\"$1\";;\nesac\nshift\ndone\nif [ \"$tg\" != 0 ]; then\n/bin/cat <<'RESULT'\n{decode}\nRESULT\nelif [ \"$pp\" = 512 ]; then\n/bin/cat <<'RESULT'\n{headline}\nRESULT\nelse\n/bin/cat <<'RESULT'\n{remaining}\nRESULT\nfi" + ) + } else { + format!("/bin/cat <<'RESULT'\n{result}\nRESULT") + }; + let script = format!("#!/bin/sh\ncase \"$1\" in\n describe) printf '%s\\n' '{descriptor}';;\n --help) printf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json';;\n *) printf '%s\\n' \"$@\" >> \"$ARENA_TEST_ARGS\"\n{before_result}\n{result_script}\n;;\nesac\n"); fs::write(&self.executable, script).unwrap(); fs::set_permissions(&self.executable, fs::Permissions::from_mode(0o755)).unwrap(); } @@ -499,11 +538,11 @@ fn runtimes_agree_on_samples_units_and_rates_without_faking_protocol_equivalence } assert_eq!(b["benchmark"]["metrics"]["pp512_t_s"], 3840.0); assert_eq!(b["benchmark"]["protocol"]["warmup"], "runtime_native"); - assert_eq!(b["benchmark"]["params"]["decode_context_tokens"], 0); + assert_eq!(b["benchmark"]["params"]["decode_context_tokens"], 1); assert_eq!(b["benchmark"]["protocol"]["telemetry_available"], true); assert_eq!( b["benchmark"]["telemetry"]["scope"], - "whole_runtime_process" + "headline_then_prefill_processes" ); assert!(b["benchmark"]["telemetry"].get("memory_replay").is_none()); assert_eq!( @@ -538,12 +577,26 @@ fn runtimes_agree_on_samples_units_and_rates_without_faking_protocol_equivalence .contains(fixture.dir.path().to_str().unwrap())); success(&fixture.verify()); let args = fs::read_to_string(fixture.dir.path().join("args")).unwrap(); - assert!(args.contains("\n128,512\n")); + if fixture.runtime == "basert" { + assert!(args.contains("\n128,512\n")); + } else { + let prompts: Vec<_> = args + .lines() + .collect::>() + .windows(2) + .filter(|pair| pair[0] == "-p") + .map(|pair| pair[1]) + .collect(); + assert_eq!(prompts, ["512", "0", "128"]); + assert!(!args.contains("--ctx")); + assert!(!args.contains("-d\n4096\n")); + } assert!(args.contains(fixture.model.to_str().unwrap())); assert!(!args.contains("--cooldown")); assert!(!args.contains("--telemetry")); if fixture.runtime == "llama-cpp" { assert!(args.contains("-d\n0\n")); + assert!(args.contains("-d\n1\n")); assert!(args.contains("-o\njson\n")); } } @@ -570,6 +623,38 @@ fn basert_native_same_run_telemetry_is_selected_by_capability_not_version() { assert!(args.contains("--telemetry")); } +#[test] +fn headline_capable_basert_signs_new_metadata_without_changing_requested_repetitions() { + let f = Fixture::new("basert"); + let mut result = f.result(); + result["params"]["ctx"] = json!(4096); + result["protocol"] = json!({"schema":"basert-throughput-protocol/2","profile":"basert-bench-capacity/1", + "context_isolation":"headline_then_per_prefill","context_capacity_policy":"basert_bench_default", + "model_load_in_timing":false,"execution_layout":"headline_then_prefill_processes", + "execution_order":["pp512","tg128","pp128"], + "prefill":{"128":{"initial_context_tokens":0,"context_capacity_tokens":4096}, + "512":{"initial_context_tokens":0,"context_capacity_tokens":4096}}, + "decode":{"initial_context_tokens":1,"context_capacity_tokens":4096,"seed_prefill_in_timing":false}, + "measurement":{"timed_repetitions":2,"requested_warmup_repetitions":3, + "warmup_policy":"fixed_repetitions","minimum_warmup_s":0, + "telemetry":"disabled","cooldown":false,"timing":"harness_existing_token_operations"}}); + f.install(&result, ""); + let signed = f.signed(); + assert_eq!( + signed["benchmark"]["protocol"]["id"], + "computearena-throughput/3" + ); + assert_eq!(signed["benchmark"]["raw_samples"], result["raw_samples"]); + let args = fs::read_to_string(f.dir.path().join("args")).unwrap(); + assert!(args.contains("--headline-first")); + assert!(!args.contains("--isolated-workloads")); + assert!(args.contains("-r\n2\n-w\n3\n")); + success(&f.verify()); + if let Some(path) = std::env::var_os("COMPUTEARENA_HEADLINE_TEST_REPORT") { + fs::copy(&f.report, path).unwrap(); + } +} + #[test] fn signed_fields_cannot_be_tampered_with_in_either_runtime() { for runtime in RUNTIMES { diff --git a/crates/computearena-cli/tests/runtime_flow.rs b/crates/computearena-cli/tests/runtime_flow.rs index 6455be6..603fd29 100644 --- a/crates/computearena-cli/tests/runtime_flow.rs +++ b/crates/computearena-cli/tests/runtime_flow.rs @@ -14,7 +14,7 @@ fn llama_cpp_offline_run_signs_runtime_identity_and_remains_verifiable_after_upg header.extend_from_slice(&0_u64.to_le_bytes()); fs::write(&model, header).unwrap(); - let rows: Vec = [(128, 0), (512, 0), (0, 128)] + let pp_rows: Vec = [(128, 0), (512, 0)] .into_iter() .map(|(pp, tg)| { json!({ @@ -26,10 +26,20 @@ fn llama_cpp_offline_run_signs_runtime_identity_and_remains_verifiable_after_upg }) }) .collect(); + let tg_rows = vec![json!({ + "build_commit":"abc123", "build_number":123, "model_type":"Qwen3 Q4_K_M", + "model_filename":model, "model_size":24, "model_n_params":4000000000u64, + "n_prompt":0, "n_gen":128, "n_depth":1, "n_gpu_layers":99, + "gpu_info":"Apple M5 Pro", "cpu_info":"Apple M5 Pro", "backends":"Metal", + "samples_ns":[100000000,200000000] + })]; + let executable = dir.path().join("llama-bench"); fs::write(&executable, format!( - "#!/bin/sh\nif [ \"$1\" = --help ]; then\nprintf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json'\nelse\ncat <<'JSON'\n{}\nJSON\nfi\n", - serde_json::to_string(&rows).unwrap() + "#!/bin/sh\nif [ \"$1\" = --help ]; then\nprintf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json'\nelse\ncase \" $* \" in\n *\" -d 1 \"*) printf '%s\\n' '{}' ;;\n *\" -p 512 \"*) printf '%s\\n' '{}' ;;\n *) printf '%s\\n' '{}' ;;\nesac\nfi\n", + serde_json::to_string(&tg_rows).unwrap(), + serde_json::to_string(&pp_rows[1..]).unwrap(), + serde_json::to_string(&pp_rows[..1]).unwrap() )).unwrap(); fs::set_permissions(&executable, fs::Permissions::from_mode(0o755)).unwrap(); let report = dir.path().join("report.json"); diff --git a/crates/computearena-cli/tests/runtime_setup.rs b/crates/computearena-cli/tests/runtime_setup.rs index d4faee2..2e03598 100644 --- a/crates/computearena-cli/tests/runtime_setup.rs +++ b/crates/computearena-cli/tests/runtime_setup.rs @@ -33,7 +33,7 @@ impl Sandbox { fn fake_runtime(&self, at: &Path) { let descriptor = json!({"schema":"basert-benchmark-harness-descriptor/1", "runtime":{"name":"basert","version":"0.2.4"},"result_schema":"basert-benchmark-harness/1"}); - let rows: Vec = [(128, 0), (512, 0), (0, 128)] + let pp_rows: Vec = [(128, 0), (512, 0)] .into_iter() .map(|(pp, tg)| { json!({"build_commit":"abc123","build_number":123,"model_type":"Qwen3 Q4_K_M", @@ -42,9 +42,16 @@ impl Sandbox { "cpu_info":"Test CPU","gpu_info":"","samples_ns":[100000000,200000000]}) }) .collect(); + let tg_rows = vec![json!({"build_commit":"abc123","build_number":123, + "model_type":"Qwen3 Q4_K_M","model_filename":"model.gguf","model_size":24, + "model_n_params":4000000000u64,"n_prompt":0,"n_gen":128,"n_depth":1, + "n_gpu_layers":0,"backends":"CPU","cpu_info":"Test CPU","gpu_info":"", + "samples_ns":[100000000,200000000]})]; let script = format!( - "#!/bin/sh\ncase \"$1\" in\n describe) printf '%s\\n' '{descriptor}';;\n --help) printf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json';;\n *) /bin/cat <<'RESULT'\n{}\nRESULT\n;;\nesac\n", - Value::Array(rows) + "#!/bin/sh\ncase \"$1\" in\n describe) printf '%s\\n' '{descriptor}';;\n --help) printf '%s\\n' '--n-prompt --n-gen --n-depth --repetitions --no-warmup json';;\n *) case \" $* \" in\n *\" -d 1 \"*) printf '%s\\n' '{}' ;;\n *\" -p 512 \"*) printf '%s\\n' '{}' ;;\n *) printf '%s\\n' '{}' ;;\n esac;;\nesac\n", + Value::Array(tg_rows), + json!(&pp_rows[1..]), + json!(&pp_rows[..1]) ); fs::create_dir_all(at.parent().unwrap()).unwrap(); fs::write(at, script).unwrap(); diff --git a/docs/benchmark-profiles.md b/docs/benchmark-profiles.md index 5a96b49..d812112 100644 --- a/docs/benchmark-profiles.md +++ b/docs/benchmark-profiles.md @@ -26,6 +26,11 @@ belongs to the harness's multi-pass telemetry protocol. The wait adds 10 seconds A future harness can provide native per-workload conditioning together with native same-run telemetry. ComputeArena uses that path only when the descriptor advertises `features.same_run_telemetry: true` and `telemetry_schema: basert-telemetry/4`. +`features.headline_context_capacity: true` plus the supported capacity schema +enables `--headline-first`: PP512/TG128 at 4K reservation before the larger sweep. +The native harness's warmup, repetition counts, telemetry and optional cooldown +remain unchanged. Older `isolated_workload_contexts` support and legacy paths +remain available and are identified in each report. ## llama.cpp cooldown semantics @@ -33,7 +38,16 @@ llama-bench has no external pause hook between workload phases in a running swee ComputeArena launches each requested PP size with TG0 and the requested TG workload with PP0 in a **fresh process**. It waits before each launch, then loading and native warmup occur. This is not equivalent to BaseRT's one pre-suite wait or a reset immediately -before a measured repetition. Standard mode still uses one process. +before a measured repetition. Both modes run the headline PP512, then TG128, +then remaining PP sizes. Standard mode groups the remaining PP sizes into one +process. PP starts empty; TG has one untimed seed token. Custom sweeps without +PP512 use the first requested PP size as headline. + +The user's unmodified llama-bench controls reservation. Native context requests +are PP+TG+depth; there is no independent 4K capacity flag. The signed report +explicitly records that the 4K target was not applied. `-d 4096` is never used +as a substitute because it would add actual history. Warmup and timers are +unchanged; no fork or replacement llama.cpp build is required. The controller uses a fixed set of initially readable CPU/GPU/die-labelled temperature sensors. It excludes battery/storage/ambient labels. The first stable window establishes @@ -57,10 +71,12 @@ The CLI calculates these wait estimates from the selected sweep. There is no cal total runtime prediction yet. It says so instead of reusing BaseRT's replay-duration estimate, which would be incorrect for llama.cpp. -Conditioned reports use `llama-bench-conditioned-pp-tg/1` with -`execution_layout: one_process_per_workload`. The private backend must include support -for this protocol before accepting those submissions. Existing standard reports still -use `llama-bench-independent-pp-tg/1`. No database migration is needed. +llama.cpp profiles normalize to `computearena-throughput/2`. Conditioned reports use +`execution_layout: one_process_per_workload`; new standard reports use +`headline_then_prefill_processes`. BaseRT's new capacity profile uses `/3`. +The runtime-specific protocol evidence and +cooldown metadata remain signed. The private backend must accept the normalized +protocol before these submissions are published. No database migration is required. ## Audit of remaining differences diff --git a/docs/runtime-adapters.md b/docs/runtime-adapters.md index 511ae8f..850d39b 100644 --- a/docs/runtime-adapters.md +++ b/docs/runtime-adapters.md @@ -3,8 +3,10 @@ ComputeArena owns menus, authentication, signing, report storage, and submission. RuntimeAdapter implementations own executable discovery, capabilities, model selection, benchmark execution, and translation into the shared raw sample/metric structure. -BaseRT keeps its native harness schema; llama.cpp uses computearena-measurements/1. -Both retain the computearena-benchmark/1 signed envelope, so old reports remain readable. +BaseRT keeps its native harness evidence; llama.cpp uses computearena-measurements/1. +Adapters retain versioned timing/context semantics in their protocol metadata while +retaining their runtime-specific evidence. Both retain the computearena-benchmark/1 +signed envelope, so old reports remain readable. ## Commands @@ -37,24 +39,36 @@ Both retain the computearena-benchmark/1 signed envelope, so old reports remain All adapters store positive per-repetition token counts and elapsed nanoseconds. The server recomputes arithmetic mean throughput from these samples. llama.cpp must return every requested independent PP/TG workload exactly once, -with the expected repetition count and zero initial context depth. Mixed model, -build, device, and runtime settings in one run are rejected. +with the expected repetition count. PP runs with `n_depth=0`; TG runs in a +separate process with `n_prompt=0` and `n_depth=1`, creating one untimed seed +token before timed generation. Mixed model, build, device, and runtime settings +in one report are rejected. Build identity comes from llama-bench JSON's build_number and build_commit. Capability probing uses --help because --version is not implemented consistently. The initial llama.cpp adapter uses native warmup and records it as runtime_native. --warmup 0 disables it; any positive value enables native warmup, not that number -of repetitions. The plan states this before execution. Automatic external telemetry -is collected over the whole process (see telemetry.md). Optional cooldown runs each -workload in a fresh process using llama-bench-conditioned-pp-tg/1 (see benchmark-profiles.md). -Native effective settings are preserved. +of repetitions. The plan states this before execution. Standard mode uses a PP512 +process, then a TG process, then a remaining-PP-sweep process; optional cooldown uses one fresh process per +workload (see benchmark-profiles.md). Automatic external telemetry is collected +for each process and native effective settings are preserved. No equivalence between BaseRT and GGUF quantization names is assumed. Both adapters emit the runtime-neutral `computearena-model/1` identity described in model-identity.md. -The PP/TG counts alone do not establish equivalent timing semantics. llama.cpp -records a distinct protocol ID and its exclusion of sampling/tokenization. -Consumers must retain runtime, quantization, context depth, and protocol when -comparing results. Existing BaseRT measurements are not retroactively relabelled. +The PP/TG counts alone do not establish equivalent timing semantics. BaseRT's +new `features.headline_context_capacity` selects `--headline-first`: 4K reserved +capacity for PP512/TG128 first, then the remaining PP sweep. It produces +`computearena-throughput/3` with nested `basert-throughput-protocol/2` evidence. +Only capacity and order change, not the harness's warmup or timed operations. +Older isolated harnesses still use `/2`; legacy harnesses retain their old path. + +llama.cpp remains `/2` with additional order and context-request evidence. Its +unmodified native capacity request is PP+TG+initial-depth, not forced to 4K. +Physical allocation padding is not inferred. No patched runtime, extra depth, +or unsupported flag is introduced. Both runtimes run the requested headline +first when supported, and use their existing native warmup/timing policies. +Consumers retain runtime, quantization, protocol and signed runtime evidence. +Existing measurements are not retroactively relabelled or excluded from rankings. ## Binary provenance diff --git a/docs/telemetry.md b/docs/telemetry.md index 035aec8..3d8d3b1 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -22,30 +22,42 @@ therefore record `computearena-telemetry/1` under `benchmark.telemetry`: - Observer timing: requested interval, attempts, missed deadlines, elapsed window, and time spent reading sensors. Aggregate storage is bounded (up to 128 temperature sensors), rather than accumulating an unlimited time series. +- Signed before/after `computearena-environment/1` boundaries: OS family, + architecture, version and kernel; logical/physical CPU count; total and + available host memory; swap use; macOS memory pressure or Linux PSI; + normalized power/performance mode; and available accelerator configuration. + Apple GPU name/core count and Metal capability come from `system_profiler`; + that static configuration is cached once per CLI process. Dynamic memory, + pressure, swap and power fields are refreshed at each boundary. ## Measurement boundaries The worker is initialized before launching the runtime and samples every second until -it exits. For a standard BaseRT or llama.cpp run it covers model loading, runtime -warmup, the entire PP/TG sweep, and teardown. These are **concurrently observed -whole-run measurements**, not per-PP diagnostic results. One-second sampling can miss -short peaks or a very short-lived process. +it exits. A legacy BaseRT suite is one process. Standard llama.cpp uses one PP-sweep +process plus one TG process; thermally controlled llama.cpp uses one process per +workload. Each observation covers that process's model loading, native warmup, +measured work, and teardown. These are **concurrently observed process-window +measurements**, not runtime allocator counters. One-second sampling can miss short +peaks or a very short-lived process. The existing `benchmark.memory.process_peak_rss_mb` compatibility field carries the observed peak alongside its scope, units, and caveat, so existing CLI/backend memory summaries can consume it. It is not a process-lifetime kernel high-water mark. Detailed temperature and power data remain in the signed telemetry object. -Vendor tools run only outside the runtime process window. Each invocation has a +Vendor tools and other slow environment probes run only outside a runtime process +window. Static macOS GPU configuration is cached. Every command invocation has a two-second timeout, a 64-KiB output cap checked while running, and is killed/reaped on timeout. Temporary output files are removed automatically. The worker owns its OS sensor handles and is stopped/joined on completion or failure. Energy, runtime allocator/KV-cache queries, and per-workload attribution are explicitly unavailable in the external-observer path. Power snapshots are not energy measurements, -and a whole-run sensor reading is not a per-token metric. Optional llama.cpp cooldown -groups telemetry by isolated workload process, still including loading/warmup. Current -BaseRT cooldown performs one external wait before the suite. See benchmark-profiles.md. +and a process-window sensor reading is not a per-token metric. llama.cpp groups +telemetry by the PP/TG process layout, still including loading and warmup. Current +BaseRT cooldown performs one external wait before the suite. Native BaseRT fields +remain available when its harness reports them from the recorded runs. See +benchmark-profiles.md. ## BaseRT capability transition @@ -57,11 +69,16 @@ Runtime selection is capability-based, not tied to a BaseRT version string: `telemetry_schema: basert-telemetry/4` selects the native `--telemetry` path. - Advertising native same-run telemetry with an unknown or missing schema fails closed instead of silently changing measurement semantics. +- `features.isolated_workload_contexts: true` selects `--isolated-workloads`. + The CLI validates the returned `basert-throughput-protocol/1` capacities and + initial-context metadata before marking the result comparable. Missing support + retains the legacy report but marks its normalized protocol non-comparable. -The future native schema may contain energy, KV-cache, allocator, and per-workload -data, but only if those values are collected during the same runs that produce the -signed throughput samples. This descriptor contract lets a new BaseRT release plug -in without requiring a corresponding ComputeArena release. + +The native schema may contain energy, KV-cache, allocator, and per-workload data, +but only when those values are collected during the same runs that produce the +signed throughput samples. These descriptor contracts let a BaseRT release plug in +without requiring a corresponding ComputeArena release. ## Overhead and validation diff --git a/docs/testing.md b/docs/testing.md index d9d83e4..acaeab2 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -35,11 +35,11 @@ They do not run GPU workloads. See [telemetry.md](telemetry.md) for collector co | CLI UX | Runtime-scoped help, no nested runtime selectors, one-per-line runtime chooser skipped when a single runtime is installed, menu exit, common list/inspect/verify commands, NO_COLOR output, explicit install/capability guidance | | Discovery | PATH discovery, explicit executable path override, paths containing spaces, missing/incompatible executables, executable feedback on selection, ComputeArena-installed copies, BaseRT's default install location | | Installation | Plan shown before installing, local bundle unpacking, install records, replacement of earlier copies, refusal without a terminal or --yes, non-interactive runs pointing at install | -| Runtime invocation | Requested PP sweep forwarded, separate PP/TG samples, llama.cpp depth zero and JSON output, no replay telemetry flag for current BaseRT, no default cooldown, native warmup disablement | +| Runtime invocation | Requested PP sweep forwarded, separate PP/TG processes, llama.cpp PP depth zero and TG depth one, JSON output, no replay telemetry flag for current BaseRT, no default cooldown, native warmup disablement | | Result consistency | Identical raw measurements produce identical token/second metrics and units across adapters; bogus llama.cpp aggregate rates are ignored | -| Protocol differences | Current BaseRT and llama.cpp use concurrent whole-run telemetry; future BaseRT native same-run telemetry is selected only by an advertised capability; llama.cpp conditioned runs use a distinct per-workload-process protocol | +| Protocol compatibility | Isolated BaseRT metadata and llama.cpp normalize to computearena-throughput/2; legacy BaseRT remains readable but non-comparable; native same-run telemetry is selected only by an advertised capability | | Runtime failures | Nonzero exit, malformed JSON, executable mutation during a benchmark: no report signed | -| Measurement validation | Missing workloads, token-count mismatches, repetition mismatches, invalid/unsafe durations, nonzero llama.cpp depth, inconsistent build identity | +| Measurement validation | Missing workloads, token-count mismatches, repetition mismatches, invalid/unsafe durations, incorrect llama.cpp PP/TG depth, incompatible BaseRT context metadata, inconsistent build identity | | Binary identity | Signed SHA-256 equals the executable bytes; platform identity recorded; report remains verifiable after the executable is removed or upgraded | | Report integrity | Changes to runtime identity, digest, version, timing, rates, token sizes, model, timestamp, or run ID invalidate the signature | | Signature format | Unsigned/malformed reports, bad algorithm/canonicalization/key ID/signature rejected; whitespace and key-order changes accepted |