diff --git a/README.md b/README.md index 5a6a7df..7f2ffb2 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,21 @@ of `--runtime-path`. BaseRT 0.2.4 and newer can also start this client with BaseRT remains responsible for choosing a compatible backend artifact, downloading split files, conversion, and writing `hub.json` provenance. +ComputeArena says when the BaseRT it found is worth updating, and never +refuses to run an older one. A harness that does not advertise the +headline-first protocol (BaseRT 0.2.4 and older) is named before the benchmark +plan, with what its report will be signed as; this needs no network, because +the harness describes itself. A newer BaseRT release is mentioned the same way. +That lookup asks GitHub for BaseRT's latest release in the background, keeps +the answer for 24 hours in `basert-update-check.json`, and never delays or +fails a run; when it has not answered before a benchmark starts, the notice +follows the run instead. `computearena basert install` installs the latest +release where the official installer does, and the full-screen interface +offers the same with `u` on its menu, asking for a second press before it +replaces anything. A harness chosen with `--runtime-path` or +`COMPUTEARENA_BASERT_HARNESS` is yours to update, and on platforms without a +prebuilt BaseRT the notice points at the release to build from instead. + ### llama.cpp The adapter asks for a GGUF file rather than scanning the disk, and lists the @@ -346,6 +361,7 @@ minimumClientVersion for an actionable upgrade message. | `COMPUTEARENA_API_URL` | API base URL, same as `--api-url` | | `COMPUTEARENA_BASERT_HARNESS` | Path to `basert-benchmark-harness`, same as `--runtime-path` for BaseRT | | `BASERT_INSTALL_DIR` | Where BaseRT is looked for and installed; `~/.basert` by default | +| `COMPUTEARENA_BASERT_RELEASE_API` | Where BaseRT's latest release is looked up, for mirrors and tests; GitHub's API for `basecompute/baseRT` by default. Only a version number is read from the answer | | `BASERT_MODELS_DIR` | Where installed BaseRT models are listed from; BaseRT's own model cache by default | | `CUDA_VISIBLE_DEVICES` | Respected by the CUDA chip fallback; a mask leaves the chip unresolved | | `NO_COLOR` | Plain output | diff --git a/crates/computearena-cli/src/adapters/basert.rs b/crates/computearena-cli/src/adapters/basert.rs index 8335606..ecabf77 100644 --- a/crates/computearena-cli/src/adapters/basert.rs +++ b/crates/computearena-cli/src/adapters/basert.rs @@ -17,14 +17,14 @@ enum TelemetryMode { NativeSameRun, } -fn supports_isolated_workloads(descriptor: &Value) -> bool { +pub(crate) fn supports_isolated_workloads(descriptor: &Value) -> bool { descriptor .pointer("/features/isolated_workload_contexts") .and_then(Value::as_bool) == Some(true) } -fn supports_headline_capacity(descriptor: &Value) -> Result { +pub(crate) fn supports_headline_capacity(descriptor: &Value) -> Result { let supported = descriptor .pointer("/features/headline_context_capacity") .and_then(Value::as_bool) diff --git a/crates/computearena-cli/src/basert_updates.rs b/crates/computearena-cli/src/basert_updates.rs new file mode 100644 index 0000000..a4cc5a7 --- /dev/null +++ b/crates/computearena-cli/src/basert_updates.rs @@ -0,0 +1,437 @@ +//! What ComputeArena says about the BaseRT it found. +//! +//! Two things are worth a sentence before a benchmark is spent on them: a +//! newer BaseRT release exists, or the installed harness predates the +//! benchmark protocol current reports use. The second needs no network, because +//! the harness says what it supports. Neither ever stops a run: an older BaseRT +//! keeps working exactly as before, and its report records what was used. + +use crate::adapters::basert::{supports_headline_capacity, supports_isolated_workloads}; +use crate::reports::Paths; +use crate::runtimes::{Source, BASERT_INSTALL_SCRIPT, BASERT_RELEASES}; +use crate::ui::TerminalUi; +use crate::updates::{LatestRelease, ReleaseCheck}; +use semver::Version; +use serde_json::Value; + +/// The first BaseRT release whose harness runs the headline-first protocol. +const HEADLINE_FIRST_SINCE: Version = Version::new(0, 2, 5); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Protocol { + /// PP512 and TG128 first, in a freshly loaded model with a 4K reservation. + HeadlineFirst, + /// One context per workload, without the headline order. + IsolatedWorkloads, + /// One context shared by the whole sweep; signed as not comparable. + SharedContext, +} + +fn protocol(descriptor: &Value) -> Protocol { + // A harness advertising a capacity protocol this client cannot read is + // newer than the client, not older: the run itself reports that. + if supports_headline_capacity(descriptor).unwrap_or(true) { + Protocol::HeadlineFirst + } else if supports_isolated_workloads(descriptor) { + Protocol::IsolatedWorkloads + } else { + Protocol::SharedContext + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Advice { + /// Stands on its own: a status line, a plan row. + pub(crate) summary: String, + /// What an older protocol means for the report; empty for a plain update. + pub(crate) consequence: Option, + /// What to do about it. + pub(crate) action: String, + /// Whether `computearena basert install` is that action. + pub(crate) installable: bool, + /// The older protocol in a few words, for a benchmark plan. + pub(crate) plan_note: Option<&'static str>, +} + +/// What to say about this BaseRT, if anything. `latest` is the newest release +/// when that is known; an older protocol is reported either way. +pub(crate) fn advice( + descriptor: &Value, + latest: Option<&Version>, + source: Source, + prebuilt_for_platform: bool, +) -> Option { + let installed_text = descriptor + .pointer("/runtime/version") + .and_then(Value::as_str) + .filter(|version| !version.is_empty()); + let installed = installed_text.and_then(|version| Version::parse(version).ok()); + let newer = match (&installed, latest) { + (Some(installed), Some(latest)) if latest > installed => Some(latest), + _ => None, + }; + let protocol = protocol(descriptor); + if protocol == Protocol::HeadlineFirst && newer.is_none() { + return None; + } + + let name = match installed_text { + Some(version) => format!("BaseRT {version}"), + None => "This BaseRT".to_string(), + }; + // The release worth moving to: the newest when it is known to carry the + // protocol, otherwise the first one that does. + let target = match latest { + Some(latest) if *latest >= HEADLINE_FIRST_SINCE => format!("BaseRT {latest}"), + _ if protocol != Protocol::HeadlineFirst => { + format!("BaseRT {HEADLINE_FIRST_SINCE} or newer") + } + _ => "the newer release".to_string(), + }; + let (summary, consequence, plan_note) = match protocol { + Protocol::HeadlineFirst => ( + format!( + "{target} is available (installed: {}).", + installed_text.unwrap_or("unknown") + ), + None, + None, + ), + Protocol::IsolatedWorkloads => ( + format!("{name} predates the current benchmark protocol."), + Some(format!( + "Its runs are signed without the headline-first order. {target} measures PP512 and TG128 first, in a freshly loaded model with a 4K reservation, and reports telemetry from the timed repetitions." + )), + Some("Older BaseRT protocol: no headline-first order"), + ), + Protocol::SharedContext => ( + format!("{name} predates the current benchmark protocol."), + Some(format!( + "Its runs share one context across the sweep and are signed as computearena-throughput-legacy/1, marked not comparable. {target} measures PP512 and TG128 first, in a freshly loaded model with a 4K reservation, and reports telemetry from the timed repetitions." + )), + Some("Older BaseRT protocol: signed as not comparable"), + ), + }; + + let chosen_by_hand = match source { + Source::Override => Some("--runtime-path".to_string()), + Source::Environment(variable) => Some(variable.to_string()), + Source::Managed | Source::Path | Source::KnownLocation => None, + }; + let (action, installable) = match (chosen_by_hand, prebuilt_for_platform) { + (Some(how), true) => ( + format!( + "This harness was chosen with {how}: point that at a newer build, or leave it out and run `computearena basert install`." + ), + false, + ), + (Some(how), false) => ( + format!( + "This harness was chosen with {how}: point that at a build of the newer release ({BASERT_RELEASES})." + ), + false, + ), + (None, true) => ( + format!( + "Update with `computearena basert install`, or the official installer: {BASERT_INSTALL_SCRIPT}" + ), + true, + ), + (None, false) => ( + format!( + "No prebuilt BaseRT is published for this platform; build the harness from the newer release: {BASERT_RELEASES}" + ), + false, + ), + }; + Some(Advice { + summary, + consequence, + action, + installable, + plan_note, + }) +} + +/// Whether `computearena basert install` can fetch a bundle on this machine. +pub(crate) fn prebuilt_for_this_platform() -> bool { + crate::runtimes::asset_rule( + crate::adapters::Runtime::Basert, + std::env::consts::OS, + std::env::consts::ARCH, + ) + .is_ok() +} + +pub(crate) fn print_advice(ui: TerminalUi, advice: &Advice) { + println!("{} {}", ui.warning("!"), ui.strong(&advice.summary)); + if let Some(consequence) = &advice.consequence { + println!(" {}", ui.neutral(consequence)); + } + println!(" {}", ui.neutral(&advice.action)); +} + +/// The release lookup for one command: what the last lookup found, and the +/// one in progress. Nothing here waits for the network. +pub(crate) struct Watch { + latest: Option, + check: Option, +} + +impl Watch { + pub(crate) fn start(paths: &Paths) -> Self { + let (latest, check) = crate::updates::BASERT.start(paths); + Self { latest, check } + } + + #[cfg(test)] + pub(crate) fn known(latest: Option) -> Self { + Self { + latest, + check: None, + } + } + + /// Takes in the lookup's answer if it has arrived; true when it changed + /// what is known. + pub(crate) fn refresh(&mut self) -> bool { + let Some(answer) = self.check.as_ref().and_then(ReleaseCheck::poll) else { + return false; + }; + self.check = None; + match answer { + Some(release) if self.latest.as_ref() != Some(&release) => { + self.latest = Some(release); + true + } + _ => false, + } + } + + pub(crate) fn latest(&self) -> Option<&Version> { + self.latest.as_ref().map(|release| &release.version) + } + + pub(crate) fn advice(&self, descriptor: &Value, source: Source) -> Option { + advice( + descriptor, + self.latest(), + source, + prebuilt_for_this_platform(), + ) + } +} + +/// One `run`: says what is known before the benchmark starts, and afterwards +/// only what the lookup learned in the meantime, so nothing is said twice. +pub(crate) struct RunNotice { + watch: Watch, + descriptor: Option, + source: Source, + said: bool, +} + +impl RunNotice { + /// A harness that cannot be probed says nothing here; the run itself + /// explains what is wrong with it. + pub(crate) fn start(paths: &Paths, harness: &std::path::Path, source: Source) -> Self { + use crate::adapters::Runtime; + Self { + watch: Watch::start(paths), + descriptor: Runtime::Basert.adapter().probe(harness).ok(), + source, + said: false, + } + } + + fn say(&mut self, ui: TerminalUi) { + self.watch.refresh(); + let Some(descriptor) = &self.descriptor else { + return; + }; + if let Some(advice) = self.watch.advice(descriptor, self.source) { + print_advice(ui, &advice); + self.said = true; + } + } + + pub(crate) fn before_run(&mut self, ui: TerminalUi) { + self.say(ui); + } + + /// A benchmark takes minutes, so a lookup that was still running when it + /// started has answered by now. + pub(crate) fn after_run(&mut self, ui: TerminalUi) { + if !self.said { + self.say(ui); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn harness(version: &str, headline: bool, isolated: bool) -> Value { + json!({ + "runtime": {"name": "basert", "version": version}, + "capacity_protocol_schema": "basert-throughput-protocol/2", + "features": { + "headline_context_capacity": headline, + "isolated_workload_contexts": isolated + } + }) + } + + fn version(text: &str) -> Version { + Version::parse(text).unwrap() + } + + #[test] + fn a_current_basert_with_nothing_newer_says_nothing() { + let current = harness("0.2.5", true, true); + assert_eq!(advice(¤t, None, Source::Managed, true), None); + assert_eq!( + advice(¤t, Some(&version("0.2.5")), Source::Managed, true), + None + ); + // An older release on the feed is not an update. + assert_eq!( + advice(¤t, Some(&version("0.2.4")), Source::Managed, true), + None + ); + } + + #[test] + fn a_newer_release_is_offered_with_the_command_that_installs_it() { + let advice = advice( + &harness("0.2.5", true, true), + Some(&version("0.2.6")), + Source::KnownLocation, + true, + ) + .unwrap(); + assert_eq!( + advice.summary, + "BaseRT 0.2.6 is available (installed: 0.2.5)." + ); + assert_eq!(advice.consequence, None); + assert_eq!(advice.plan_note, None); + assert!(advice.installable); + assert!(advice.action.contains("`computearena basert install`")); + assert!(advice.action.contains(BASERT_INSTALL_SCRIPT)); + } + + #[test] + fn an_older_protocol_is_reported_offline_and_names_what_the_report_will_say() { + // 0.2.4: neither isolated contexts nor the headline order. + let offline = advice(&harness("0.2.4", false, false), None, Source::Path, true).unwrap(); + assert_eq!( + offline.summary, + "BaseRT 0.2.4 predates the current benchmark protocol." + ); + let consequence = offline.consequence.as_deref().unwrap(); + assert!(consequence.contains("computearena-throughput-legacy/1")); + assert!(consequence.contains("not comparable")); + assert!(consequence.contains("BaseRT 0.2.5 or newer measures PP512 and TG128 first")); + assert_eq!( + offline.plan_note, + Some("Older BaseRT protocol: signed as not comparable") + ); + assert!(offline.installable); + + // With the feed's answer the advice names the release to move to. + let online = advice( + &harness("0.2.4", false, false), + Some(&version("0.2.6")), + Source::Path, + true, + ) + .unwrap(); + assert!(online + .consequence + .unwrap() + .contains("BaseRT 0.2.6 measures PP512 and TG128 first")); + + // Isolated contexts without the headline order are still comparable. + let isolated = advice(&harness("0.2.4", false, true), None, Source::Path, true).unwrap(); + assert_eq!( + isolated.plan_note, + Some("Older BaseRT protocol: no headline-first order") + ); + let consequence = isolated.consequence.unwrap(); + assert!(consequence.contains("without the headline-first order")); + assert!(!consequence.contains("legacy")); + } + + #[test] + fn the_action_fits_how_the_harness_was_found_and_what_the_platform_offers() { + let old = harness("0.2.4", false, false); + let by_flag = advice(&old, None, Source::Override, true).unwrap(); + assert!(by_flag.action.contains("chosen with --runtime-path")); + assert!(!by_flag.installable); + + let by_variable = advice( + &old, + None, + Source::Environment("COMPUTEARENA_BASERT_HARNESS"), + true, + ) + .unwrap(); + assert!(by_variable + .action + .contains("chosen with COMPUTEARENA_BASERT_HARNESS")); + + let no_bundle = advice(&old, None, Source::Path, false).unwrap(); + assert!(no_bundle.action.contains("No prebuilt BaseRT is published")); + assert!(no_bundle.action.contains(BASERT_RELEASES)); + assert!(!no_bundle.installable); + assert!(!no_bundle.action.contains("basert install")); + } + + #[test] + fn versions_that_cannot_be_compared_never_invent_an_update() { + let unversioned = json!({ + "runtime": {"name": "basert"}, + "capacity_protocol_schema": "basert-throughput-protocol/2", + "features": {"headline_context_capacity": true} + }); + assert_eq!( + advice(&unversioned, Some(&version("9.9.9")), Source::Managed, true), + None + ); + let development = harness("main-abc123", true, true); + assert_eq!( + advice(&development, Some(&version("9.9.9")), Source::Managed, true), + None + ); + // An unversioned harness that lacks the protocol is still told so. + let old = json!({"runtime": {"name": "basert"}, "features": {}}); + let advice = advice(&old, None, Source::Managed, true).unwrap(); + assert_eq!( + advice.summary, + "This BaseRT predates the current benchmark protocol." + ); + + // A harness newer than this client understands is not "older". + let future = json!({ + "runtime": {"name": "basert", "version": "0.9.0"}, + "capacity_protocol_schema": "basert-throughput-protocol/9", + "features": {"headline_context_capacity": true} + }); + assert_eq!(super::advice(&future, None, Source::Managed, true), None); + } + + #[test] + fn a_lookup_that_finishes_later_updates_what_is_known_once() { + let mut watch = Watch::known(Some(crate::updates::release_for_tests("0.2.6"))); + assert_eq!(watch.latest(), Some(&version("0.2.6"))); + // Nothing in progress: nothing changes. + assert!(!watch.refresh()); + let advice = watch + .advice(&harness("0.2.5", true, true), Source::Managed) + .unwrap(); + assert!(advice.summary.contains("BaseRT 0.2.6 is available")); + } +} diff --git a/crates/computearena-cli/src/main.rs b/crates/computearena-cli/src/main.rs index 5195e84..93ccbbf 100644 --- a/crates/computearena-cli/src/main.rs +++ b/crates/computearena-cli/src/main.rs @@ -3,6 +3,7 @@ mod api; use adapters::{BenchmarkRequest, Runtime}; mod auth; mod basert_models; +mod basert_updates; mod benchmark; mod conditioning; mod config; @@ -369,9 +370,21 @@ fn execute( None => return Ok(()), }, }; + let chosen_by_flag = harness.is_some(); let (harness, model) = benchmark::identify_benchmark_paths(runtime, harness, &model, paths)?; benchmark::print_resolved_paths(runtime, &harness)?; + // Before the plan: an older BaseRT decides what the report will + // say, and updating it takes less time than the run does. + let mut basert_notice = if runtime == Runtime::Basert { + let source = runtimes::source_of(runtime, chosen_by_flag, paths)?; + Some(basert_updates::RunNotice::start(paths, &harness, source)) + } else { + None + }; + if let Some(notice) = basert_notice.as_mut() { + notice.before_run(TerminalUi::detect()); + } let Some(cooldown_enabled) = runtime.adapter().confirm( &BenchmarkRequest { model: &model, @@ -400,6 +413,9 @@ fn execute( output, )?; print_submission_hint(paths, api_url, &report)?; + if let Some(notice) = basert_notice.as_mut() { + notice.after_run(TerminalUi::detect()); + } Ok(()) } Action::List { json } => list_reports(paths, json), diff --git a/crates/computearena-cli/src/runtimes.rs b/crates/computearena-cli/src/runtimes.rs index 69c9d09..3876d24 100644 --- a/crates/computearena-cli/src/runtimes.rs +++ b/crates/computearena-cli/src/runtimes.rs @@ -78,6 +78,14 @@ pub(crate) struct Located { pub(crate) source: Source, } +/// How the executable a command is about to run was chosen. +pub(crate) fn source_of(runtime: Runtime, chosen_by_flag: bool, paths: &Paths) -> Result { + if chosen_by_flag { + return Ok(Source::Override); + } + locate(runtime, None, paths).map(|located| located.source) +} + pub(crate) fn executable_on_path(name: &str) -> Option { std::env::var_os("PATH").and_then(|path| { std::env::split_paths(&path) @@ -933,7 +941,7 @@ pub(crate) fn install( path: executable, source: Source::Managed, }; - report_found(ui, runtime, &located)?; + report_found(ui, runtime, &located, paths)?; Ok(Some(located)) } @@ -941,7 +949,12 @@ pub(crate) fn install( /// Print the executable that will run, and check it is usable before anyone /// picks a model. Returns the version when the runtime reports one. -pub(crate) fn report_found(ui: TerminalUi, runtime: Runtime, located: &Located) -> Result<()> { +pub(crate) fn report_found( + ui: TerminalUi, + runtime: Runtime, + located: &Located, + paths: &Paths, +) -> Result<()> { let adapter = runtime.adapter(); let capabilities = adapter.probe(&located.path).with_context(|| { format!( @@ -963,6 +976,14 @@ pub(crate) fn report_found(ui: TerminalUi, runtime: Runtime, located: &Located) ui.muted(located.source.describe()), ); println!(" {}", ui.neutral(compact_path(&located.path))); + // From what the last lookup found; the one started here is for next time, + // so finding a runtime never waits for the network. + if runtime == Runtime::Basert { + let watch = crate::basert_updates::Watch::start(paths); + if let Some(advice) = watch.advice(&capabilities, located.source) { + crate::basert_updates::print_advice(ui, &advice); + } + } Ok(()) } @@ -984,7 +1005,7 @@ pub(crate) fn ensure_runtime( let adapter = runtime.adapter(); loop { let problem = match locate(runtime, override_path.clone(), paths) { - Ok(located) => match report_found(ui, runtime, &located) { + Ok(located) => match report_found(ui, runtime, &located, paths) { Ok(()) => return Ok(RuntimeSetup::Ready(located.path)), Err(error) => format!("{error:#}"), }, diff --git a/crates/computearena-cli/src/tui/app.rs b/crates/computearena-cli/src/tui/app.rs index 5b82e6a..0fcc5cc 100644 --- a/crates/computearena-cli/src/tui/app.rs +++ b/crates/computearena-cli/src/tui/app.rs @@ -62,6 +62,23 @@ impl ReportRow { } } +/// Where a run is decided, what an older BaseRT will sign gets one row. +fn protocol_row(advice: Option<&crate::basert_updates::Advice>) -> Option<(&'static str, String)> { + let advice = advice?; + let note = advice.plan_note?; + Some(( + "Protocol", + format!( + "{note}\n{}", + if advice.installable { + "Update BaseRT from the menu (Esc, then u) for the current one" + } else { + "A newer BaseRT harness signs the current one" + } + ), + )) +} + #[derive(Clone, Copy, PartialEq, Eq)] pub(crate) enum ReportMode { Verify, @@ -206,6 +223,17 @@ pub(crate) struct App { /// offered for submission. pub(crate) pending_submission: Option, update_check: Option, + /// What the runtime in use says it supports and how it was found, kept + /// from the probe that made it usable. + runtime_descriptor: Option<(serde_json::Value, runtimes::Source)>, + /// BaseRT's newest release; looked up the first time BaseRT is entered. + basert_watch: Option, + /// Set by the first `u` on the menu: updating replaces an installation, + /// so it takes a second press. + update_armed: bool, + /// Whether a prebuilt BaseRT exists for this machine, so an update can be + /// installed from here. + prebuilt_basert: bool, } impl App { @@ -235,6 +263,10 @@ impl App { completed_report: Arc::new(Mutex::new(None)), pending_submission: None, update_check, + runtime_descriptor: None, + basert_watch: None, + update_armed: false, + prebuilt_basert: crate::basert_updates::prebuilt_for_this_platform(), }; app.refresh_account(); // The same rule as the printed session: only ask which runtime to use @@ -281,15 +313,24 @@ impl App { /// screen explaining how to obtain it. fn enter_runtime(&mut self) -> Result<()> { let adapter = self.runtime.adapter(); - match runtimes::locate(self.runtime, self.harness_override.clone(), &self.paths) - .and_then(|located| adapter.probe(&located.path).map(|_| located.path)) - { - Ok(path) => { - self.executable = Some(path); + match runtimes::locate(self.runtime, self.harness_override.clone(), &self.paths).and_then( + |located| { + adapter + .probe(&located.path) + .map(|descriptor| (located, descriptor)) + }, + ) { + Ok((located, descriptor)) => { + self.executable = Some(located.path); + self.runtime_descriptor = Some((descriptor, located.source)); + if self.runtime == Runtime::Basert && self.basert_watch.is_none() { + self.basert_watch = Some(crate::basert_updates::Watch::start(&self.paths)); + } self.screens.push(Screen::Menu { cursor: 0 }); } Err(error) => { self.executable = None; + self.runtime_descriptor = None; self.screens.push(Screen::Setup { problem: format!("{error:#}"), instructions: manual_instructions(self.runtime), @@ -300,6 +341,30 @@ impl App { Ok(()) } + /// What is worth saying about the BaseRT in use: a newer release, or a + /// harness that predates the current benchmark protocol. + pub(crate) fn basert_advice(&self) -> Option { + if self.runtime != Runtime::Basert { + return None; + } + let (descriptor, source) = self.runtime_descriptor.as_ref()?; + crate::basert_updates::advice( + descriptor, + self.basert_watch.as_ref().and_then(|watch| watch.latest()), + *source, + self.prebuilt_basert, + ) + } + + /// Whether the menu offers `u`: there is something to gain, and installing + /// the latest release is what gains it. + pub(crate) fn update_offered(&self) -> bool { + matches!(self.screen(), Screen::Menu { .. }) + && self + .basert_advice() + .is_some_and(|advice| advice.installable) + } + pub(crate) fn runtime_label(&self) -> String { match &self.executable { Some(path) => format!("{} · {}", self.runtime.adapter().name(), compact_path(path)), @@ -338,6 +403,9 @@ impl App { changed = true; } } + if let Some(watch) = self.basert_watch.as_mut() { + changed |= watch.refresh(); + } if let Some(pending) = self.pending.as_ref() { match pending.receiver.try_recv() { Ok(loaded) => { @@ -412,7 +480,8 @@ impl App { ) .map(|(_, model)| model)?; let request = self.benchmark_request(&model); - let rows = plan_rows(self.runtime, &request)?; + let mut rows = plan_rows(self.runtime, &request)?; + rows.extend(protocol_row(self.basert_advice().as_ref())); let options = profile_options(self.runtime, &request)?; self.screens.push(Screen::Plan { model, @@ -858,6 +927,12 @@ impl App { return Ok(()); } self.status.clear(); + // Armed only until the next key, whatever that key is. + let update_armed = std::mem::take(&mut self.update_armed); + if matches!(key.code, KeyCode::Char('u')) && self.update_offered() { + self.confirm_or_start_update(update_armed); + return Ok(()); + } match key.code { KeyCode::Char(character) => self.on_char(character)?, KeyCode::Up => self.move_cursor(-1), @@ -874,6 +949,26 @@ impl App { Ok(()) } + /// Updating downloads a release and replaces the installed bundle, so the + /// first `u` says what will happen and the second one does it. + fn confirm_or_start_update(&mut self, armed: bool) { + if armed { + self.start_install(); + return; + } + self.update_armed = true; + let release = match self.basert_watch.as_ref().and_then(|watch| watch.latest()) { + Some(latest) => format!("BaseRT {latest}"), + None => "the latest BaseRT".to_string(), + }; + let directory = runtimes::basert_install_dir() + .map(|directory| compact_path(&directory)) + .unwrap_or_else(|| "its install directory".to_string()); + self.status = format!( + "Press u again to download {release} into {directory}, replacing the BaseRT files there" + ); + } + fn on_char(&mut self, character: char) -> Result<()> { // Typing filters and path entry take precedence over shortcuts. match self.screen_mut() { @@ -1453,6 +1548,10 @@ mod tests { completed_report: Arc::new(Mutex::new(None)), pending_submission: None, update_check: None, + runtime_descriptor: None, + basert_watch: None, + update_armed: false, + prebuilt_basert: true, } } @@ -1480,6 +1579,117 @@ mod tests { } } + fn old_basert() -> serde_json::Value { + serde_json::json!({"runtime": {"name": "basert", "version": "0.2.4"}, "features": {}}) + } + + fn current_basert() -> serde_json::Value { + serde_json::json!({ + "runtime": {"name": "basert", "version": "0.2.5"}, + "capacity_protocol_schema": "basert-throughput-protocol/2", + "features": {"headline_context_capacity": true} + }) + } + + fn press(app: &mut App, character: char) { + use ratatui::crossterm::event::{KeyCode, KeyEvent, KeyModifiers}; + app.on_key(KeyEvent::new(KeyCode::Char(character), KeyModifiers::NONE)) + .unwrap(); + } + + #[test] + fn an_older_basert_is_named_on_the_menu_and_updating_takes_two_presses() { + let dir = tempfile::tempdir().unwrap(); + let mut app = app(dir.path(), None); + app.runtime = Runtime::Basert; + app.screens.push(Screen::Menu { cursor: 0 }); + assert_eq!(app.basert_advice(), None, "nothing is known yet"); + + app.runtime_descriptor = Some((old_basert(), runtimes::Source::KnownLocation)); + let advice = app.basert_advice().unwrap(); + assert_eq!( + advice.summary, + "BaseRT 0.2.4 predates the current benchmark protocol." + ); + let row = protocol_row(Some(&advice)).unwrap(); + assert_eq!(row.0, "Protocol"); + assert!(row + .1 + .starts_with("Older BaseRT protocol: signed as not comparable\n")); + assert!(row.1.lines().all(|line| line.chars().count() <= 64)); + + // The first press only says what a second one would do. + assert!(app.update_offered()); + press(&mut app, 'u'); + assert!(app.update_armed); + assert!(app.job.is_none()); + assert!(app + .status + .starts_with("Press u again to download the latest BaseRT into ")); + assert!(app.status.ends_with("replacing the BaseRT files there")); + // Any other key stands it down. + press(&mut app, 'j'); + assert!(!app.update_armed); + assert!(app.job.is_none()); + + // With the feed's answer the confirmation names the release. + app.basert_watch = Some(crate::basert_updates::Watch::known(Some( + crate::updates::release_for_tests("0.2.6"), + ))); + press(&mut app, 'u'); + assert!(app + .status + .starts_with("Press u again to download BaseRT 0.2.6 into ")); + } + + #[test] + fn updating_is_offered_only_where_it_helps() { + let dir = tempfile::tempdir().unwrap(); + let mut app = app(dir.path(), None); + app.runtime = Runtime::Basert; + app.screens.push(Screen::Menu { cursor: 0 }); + + // A current BaseRT with nothing newer: no notice, and `u` is inert. + app.runtime_descriptor = Some((current_basert(), runtimes::Source::Managed)); + app.basert_watch = Some(crate::basert_updates::Watch::known(Some( + crate::updates::release_for_tests("0.2.5"), + ))); + assert_eq!(app.basert_advice(), None); + assert!(protocol_row(app.basert_advice().as_ref()).is_none()); + press(&mut app, 'u'); + assert!(!app.update_armed); + + // A newer release is an update, but says nothing about the protocol. + app.basert_watch = Some(crate::basert_updates::Watch::known(Some( + crate::updates::release_for_tests("0.2.6"), + ))); + assert!(app.update_offered()); + assert!(protocol_row(app.basert_advice().as_ref()).is_none()); + + // A harness chosen by hand is not replaced by an install. + app.runtime_descriptor = Some((old_basert(), runtimes::Source::Override)); + assert!(app.basert_advice().is_some()); + assert!(!app.update_offered()); + let row = protocol_row(app.basert_advice().as_ref()).unwrap(); + assert!(row + .1 + .ends_with("A newer BaseRT harness signs the current one")); + + // Nor on a platform without a prebuilt BaseRT. + app.runtime_descriptor = Some((old_basert(), runtimes::Source::Path)); + app.prebuilt_basert = false; + assert!(!app.update_offered()); + + // Only on the menu, and never for llama.cpp. + app.prebuilt_basert = true; + assert!(app.update_offered()); + app.screens.push(Screen::Account { cursor: 0 }); + assert!(!app.update_offered()); + app.screens.pop(); + app.runtime = Runtime::LlamaCpp; + assert_eq!(app.basert_advice(), None); + } + #[test] fn a_partial_run_cannot_be_ticked_and_says_why() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/computearena-cli/src/tui/draw.rs b/crates/computearena-cli/src/tui/draw.rs index fffe2f6..52dabb5 100644 --- a/crates/computearena-cli/src/tui/draw.rs +++ b/crates/computearena-cli/src/tui/draw.rs @@ -215,6 +215,9 @@ fn footer(frame: &mut Frame, area: Rect, app: &App) { Screen::HubModels { .. } => "↑/↓ move · Enter list files · Esc back", Screen::HubFiles { .. } => "↑/↓ move · Enter download · Esc back", Screen::BaseRtModels { .. } => "type to filter · ↑/↓ move · Enter download · Esc back", + Screen::Menu { .. } if app.update_offered() => { + "↑/↓ move · Enter select · u update BaseRT · Esc back · Ctrl+C quit" + } _ => "↑/↓ move · Enter select · Esc back · Ctrl+C quit", }; let status = if app.status.is_empty() { @@ -245,7 +248,7 @@ fn body(frame: &mut Frame, area: Rect, app: &mut App) { instructions, cursor, } => setup_screen(frame, area, problem, instructions, *cursor), - Screen::Menu { cursor } => menu_screen(frame, area, *cursor), + Screen::Menu { cursor } => menu_screen(frame, area, *cursor, app.basert_advice()), Screen::Models { rows, filter, @@ -371,13 +374,106 @@ fn setup_screen( render_list(frame, areas[1], "What next", items, cursor); } -fn menu_screen(frame: &mut Frame, area: Rect, cursor: usize) { +/// Greedy word wrapping, done here rather than by the widget so the panel +/// can be given exactly the rows its text needs. +fn wrap_words(text: &str, width: usize) -> Vec { + let width = width.max(8); + let mut lines = Vec::new(); + let mut line = String::new(); + for word in text.split_whitespace() { + let needed = line.chars().count() + usize::from(!line.is_empty()) + word.chars().count(); + if needed > width && !line.is_empty() { + lines.push(std::mem::take(&mut line)); + } + if !line.is_empty() { + line.push(' '); + } + line.push_str(word); + } + if !line.is_empty() { + lines.push(line); + } + lines +} + +/// The notice above the menu: everything when there is room for it and the +/// whole menu, otherwise just what it is and what to press. +fn basert_notice( + advice: &crate::basert_updates::Advice, + width: usize, + rows_to_spare: usize, +) -> Vec> { + let action = if advice.installable { + "Press u to update BaseRT now.".to_string() + } else { + advice.action.clone() + }; + let mut summary = wrap_words(&advice.summary, width.saturating_sub(2)).into_iter(); + let mut lines = vec![Line::from(vec![ + Span::styled("! ", Style::default().fg(danger())), + Span::styled( + summary.next().unwrap_or_default(), + Style::default().add_modifier(Modifier::BOLD), + ), + ])]; + lines.extend(summary.map(|rest| { + Line::from(Span::styled( + format!(" {rest}"), + Style::default().add_modifier(Modifier::BOLD), + )) + })); + let plain = |text: &str| -> Vec> { + wrap_words(text, width) + .into_iter() + .map(|line| Line::from(Span::styled(line, Style::default().fg(neutral())))) + .collect() + }; + let consequence = advice.consequence.as_deref().map(plain).unwrap_or_default(); + let action = plain(&action); + if lines.len() + consequence.len() + action.len() <= rows_to_spare { + lines.extend(consequence); + } + lines.extend(action); + lines +} + +fn menu_screen( + frame: &mut Frame, + area: Rect, + cursor: usize, + advice: Option, +) { let items = MENU_ITEMS .iter() .enumerate() .map(|(index, (label, detail))| item(*label, *detail, index == cursor)) .collect(); - render_list(frame, area, "ComputeArena", items, cursor); + let Some(advice) = advice else { + render_list(frame, area, "ComputeArena", items, cursor); + return; + }; + + // Above the menu rather than in the status line, which the next key + // clears: this stays true until BaseRT is updated. + let whole_menu = MENU_ITEMS.len() * 2 + 2; + let rows_to_spare = usize::from(area.height).saturating_sub(whole_menu + 2); + let lines = basert_notice( + &advice, + usize::from(area.width.saturating_sub(2)), + rows_to_spare, + ); + let height = lines.len() as u16 + 2; + // A terminal too short for even the short form keeps its menu. + if area.height < height + 6 { + render_list(frame, area, "ComputeArena", items, cursor); + return; + } + let areas = Layout::vertical([Constraint::Length(height), Constraint::Min(6)]).split(area); + frame.render_widget( + Paragraph::new(lines).block(focus_panel("BaseRT", false)), + areas[0], + ); + render_list(frame, areas[1], "ComputeArena", items, cursor); } fn models_screen(frame: &mut Frame, area: Rect, rows: &[ModelRow], filter: &str, cursor: usize) { diff --git a/crates/computearena-cli/src/updates.rs b/crates/computearena-cli/src/updates.rs index 487a723..aed3e62 100644 --- a/crates/computearena-cli/src/updates.rs +++ b/crates/computearena-cli/src/updates.rs @@ -1,4 +1,4 @@ -//! A quiet, cached update hint for the interactive client. +//! Quiet, cached release hints: one for the client itself and one for BaseRT. //! //! The GitHub request runs on a worker thread and failures are deliberately //! ignored: running and retaining benchmarks must keep working offline. A @@ -12,12 +12,93 @@ use std::fs; use std::sync::mpsc::{self, Receiver, TryRecvError}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; -const LATEST_RELEASE_API: &str = - "https://api.github.com/repos/basecompute/computearena-cli/releases/latest"; -const UPDATE_CACHE_FILE: &str = "update-check.json"; const UPDATE_CACHE_TTL: Duration = Duration::from_secs(24 * 60 * 60); const UPDATE_HTTP_TIMEOUT: Duration = Duration::from_secs(3); +/// Where the newest release of something is published, and where the answer +/// is kept between sessions. +pub(crate) struct ReleaseFeed { + latest_release_api: &'static str, + /// Names a different endpoint, for tests and mirrors. Only a version is + /// ever read from the answer, so it cannot put words on the screen. + api_override: Option<&'static str>, + cache_file: &'static str, + fallback_url: &'static str, +} + +const COMPUTEARENA: ReleaseFeed = ReleaseFeed { + latest_release_api: "https://api.github.com/repos/basecompute/computearena-cli/releases/latest", + api_override: None, + cache_file: "update-check.json", + fallback_url: COMPUTEARENA_QUICKSTART, +}; + +pub(crate) const BASERT: ReleaseFeed = ReleaseFeed { + latest_release_api: "https://api.github.com/repos/basecompute/baseRT/releases/latest", + api_override: Some("COMPUTEARENA_BASERT_RELEASE_API"), + cache_file: "basert-update-check.json", + fallback_url: crate::runtimes::BASERT_RELEASES, +}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct LatestRelease { + checked_at: u64, + pub(crate) version: Version, + release_url: String, +} + +/// A release lookup running on a worker thread. +pub(crate) struct ReleaseCheck { + receiver: Receiver>, +} + +impl ReleaseCheck { + /// `None` means still running; `Some(None)` means the check completed + /// without an answer (including an offline/network failure). + pub(crate) fn poll(&self) -> Option> { + match self.receiver.try_recv() { + Ok(release) => Some(release), + Err(TryRecvError::Empty) => None, + Err(TryRecvError::Disconnected) => Some(None), + } + } +} + +impl ReleaseFeed { + /// What the last lookup found, and whether it is recent enough to reuse. + fn known(&self, paths: &Paths) -> (Option, bool) { + let cached = read_cache(&paths.root.join(self.cache_file)); + let fresh = cached.as_ref().is_some_and(cache_is_fresh); + (cached, fresh) + } + + /// What is already known, and a lookup in progress when that is missing + /// or more than a day old. Never waits for the network. + pub(crate) fn start(&self, paths: &Paths) -> (Option, Option) { + let (cached, fresh) = self.known(paths); + if fresh { + return (cached, None); + } + + let file = paths.root.join(self.cache_file); + let api = self + .api_override + .and_then(|variable| std::env::var(variable).ok()) + .filter(|url| !url.is_empty()) + .unwrap_or_else(|| self.latest_release_api.to_string()); + let fallback_url = self.fallback_url; + let (sender, receiver) = mpsc::channel(); + std::thread::spawn(move || { + let result = fetch_latest(&api, fallback_url); + if let Some(release) = result.as_ref() { + let _ = write_cache(&file, release); + } + let _ = sender.send(result); + }); + (cached, Some(ReleaseCheck { receiver })) + } +} + #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct UpdateNotice { latest: Version, @@ -32,63 +113,42 @@ impl UpdateNotice { } } -#[derive(Clone, Debug)] -struct CachedRelease { - checked_at: u64, - version: Version, - release_url: String, -} - -pub(crate) struct UpdateCheck { - receiver: Receiver>, -} +pub(crate) struct UpdateCheck(ReleaseCheck); impl UpdateCheck { /// `None` means still running; `Some(None)` means the check completed with /// no update (including an offline/network failure). pub(crate) fn poll(&self) -> Option> { - match self.receiver.try_recv() { - Ok(notice) => Some(notice), - Err(TryRecvError::Empty) => None, - Err(TryRecvError::Disconnected) => Some(None), - } + self.0 + .poll() + .map(|release| release.as_ref().and_then(update_notice)) } } +/// The client's own update hint. pub(crate) fn start(paths: &Paths) -> (Option, Option) { - let file = paths.root.join(UPDATE_CACHE_FILE); - let cached = read_cache(&file); - let initial = cached.as_ref().and_then(update_notice); - if cached.as_ref().is_some_and(cache_is_fresh) { - return (initial, None); - } - - let (sender, receiver) = mpsc::channel(); - std::thread::spawn(move || { - let result = fetch_latest(); - if let Some(release) = result.as_ref() { - let _ = write_cache(&file, release); - } - let _ = sender.send(result.as_ref().and_then(update_notice)); - }); - (initial, Some(UpdateCheck { receiver })) + let (known, check) = COMPUTEARENA.start(paths); + ( + known.as_ref().and_then(update_notice), + check.map(UpdateCheck), + ) } fn current_version() -> Option { Version::parse(env!("CARGO_PKG_VERSION")).ok() } -fn update_notice(release: &CachedRelease) -> Option { +fn update_notice(release: &LatestRelease) -> Option { (release.version > current_version()?).then(|| UpdateNotice { latest: release.version.clone(), }) } -fn cache_is_fresh(release: &CachedRelease) -> bool { +fn cache_is_fresh(release: &LatestRelease) -> bool { unix_seconds().saturating_sub(release.checked_at) < UPDATE_CACHE_TTL.as_secs() } -fn fetch_latest() -> Option { +fn fetch_latest(api: &str, fallback_url: &str) -> Option { let client = reqwest::blocking::Client::builder() .connect_timeout(UPDATE_HTTP_TIMEOUT) .timeout(UPDATE_HTTP_TIMEOUT) @@ -96,7 +156,7 @@ fn fetch_latest() -> Option { .build() .ok()?; let response = client - .get(LATEST_RELEASE_API) + .get(api) .header("Accept", "application/vnd.github+json") .send() .ok()? @@ -105,27 +165,27 @@ fn fetch_latest() -> Option { let body = response.text().ok()?; let value: Value = serde_json::from_str(&body).ok()?; let tag = value.get("tag_name")?.as_str()?.trim_start_matches('v'); - Some(CachedRelease { + Some(LatestRelease { checked_at: unix_seconds(), version: Version::parse(tag).ok()?, release_url: value .get("html_url") .and_then(Value::as_str) - .unwrap_or(COMPUTEARENA_QUICKSTART) + .unwrap_or(fallback_url) .to_string(), }) } -fn read_cache(file: &std::path::Path) -> Option { +fn read_cache(file: &std::path::Path) -> Option { let value: Value = serde_json::from_slice(&fs::read(file).ok()?).ok()?; - Some(CachedRelease { + Some(LatestRelease { checked_at: value.get("checkedAtUnixSeconds")?.as_u64()?, version: Version::parse(value.get("latestVersion")?.as_str()?).ok()?, release_url: value.get("releaseUrl")?.as_str()?.to_string(), }) } -fn write_cache(file: &std::path::Path, release: &CachedRelease) -> anyhow::Result<()> { +fn write_cache(file: &std::path::Path, release: &LatestRelease) -> anyhow::Result<()> { if let Some(parent) = file.parent() { fs::create_dir_all(parent)?; } @@ -147,25 +207,28 @@ fn unix_seconds() -> u64 { .as_secs() } +#[cfg(test)] +pub(crate) fn release_for_tests(version: &str) -> LatestRelease { + LatestRelease { + checked_at: unix_seconds(), + version: Version::parse(version).unwrap(), + release_url: format!("https://example.test/v{version}"), + } +} + #[cfg(test)] mod tests { use super::*; - fn release(version: &str) -> CachedRelease { - CachedRelease { - checked_at: unix_seconds(), - version: Version::parse(version).unwrap(), - release_url: format!("https://example.test/v{version}"), - } - } - #[test] fn only_newer_semantic_versions_create_a_notice() { let current = current_version().unwrap(); - assert!(update_notice(&release(¤t.to_string())).is_none()); + assert!(update_notice(&release_for_tests(¤t.to_string())).is_none()); let newer = Version::new(current.major, current.minor, current.patch + 1); assert_eq!( - update_notice(&release(&newer.to_string())).unwrap().latest, + update_notice(&release_for_tests(&newer.to_string())) + .unwrap() + .latest, newer ); } @@ -173,12 +236,42 @@ mod tests { #[test] fn cache_round_trips() { let directory = tempfile::tempdir().unwrap(); - let file = directory.path().join(UPDATE_CACHE_FILE); - let expected = release("9.8.7"); + let file = directory.path().join(COMPUTEARENA.cache_file); + let expected = release_for_tests("9.8.7"); write_cache(&file, &expected).unwrap(); let actual = read_cache(&file).unwrap(); assert_eq!(actual.version, expected.version); assert_eq!(actual.release_url, expected.release_url); assert!(cache_is_fresh(&actual)); } + + #[test] + fn a_fresh_answer_is_reused_without_a_lookup_and_feeds_do_not_share_it() { + let directory = tempfile::tempdir().unwrap(); + let paths = Paths::resolve(Some(directory.path().to_path_buf())).unwrap(); + fs::create_dir_all(&paths.root).unwrap(); + write_cache( + &paths.root.join(BASERT.cache_file), + &release_for_tests("0.2.5"), + ) + .unwrap(); + + let (known, check) = BASERT.start(&paths); + assert_eq!(known.unwrap().version, Version::new(0, 2, 5)); + assert!(check.is_none(), "a fresh answer must not start a lookup"); + // BaseRT's newest release says nothing about the client's own. + assert!(read_cache(&paths.root.join(COMPUTEARENA.cache_file)).is_none()); + } + + #[test] + fn a_stale_answer_is_still_offered_while_it_is_refreshed() { + let directory = tempfile::tempdir().unwrap(); + let paths = Paths::resolve(Some(directory.path().to_path_buf())).unwrap(); + fs::create_dir_all(&paths.root).unwrap(); + assert_eq!(BASERT.known(&paths), (None, false)); + let mut stale = release_for_tests("0.2.5"); + stale.checked_at -= UPDATE_CACHE_TTL.as_secs() + 1; + write_cache(&paths.root.join(BASERT.cache_file), &stale).unwrap(); + assert_eq!(BASERT.known(&paths), (Some(stale), false)); + } } diff --git a/crates/computearena-cli/tests/adapter_contract.rs b/crates/computearena-cli/tests/adapter_contract.rs index 0c1fc97..e48286b 100644 --- a/crates/computearena-cli/tests/adapter_contract.rs +++ b/crates/computearena-cli/tests/adapter_contract.rs @@ -571,11 +571,58 @@ impl Fixture { .args(["--data-dir"]) .arg(self.dir.path().join("data")) .env("COMPUTEARENA_API_URL", "http://127.0.0.1:1/api/v1") + // No test may ask GitHub which BaseRT is newest: the lookup is + // pointed at a closed port unless a test serves its own answer. + .env( + "COMPUTEARENA_BASERT_RELEASE_API", + "http://127.0.0.1:1/latest", + ) .stdin(Stdio::null()); cmd } fn run(&self, extra: &[&str]) -> Output { + self.run_command(extra).output().unwrap() + } + + /// What the last BaseRT release lookup is remembered to have found. + fn remember_latest_basert(&self, version: &str) { + let data = self.dir.path().join("data"); + fs::create_dir_all(&data).unwrap(); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + fs::write( + data.join("basert-update-check.json"), + serde_json::to_vec(&json!({ + "checkedAtUnixSeconds": now, + "latestVersion": version, + "releaseUrl": "https://example.test/release" + })) + .unwrap(), + ) + .unwrap(); + } + + /// A BaseRT harness that advertises the headline-first protocol. + fn install_headline_capable(&self) { + let mut result = self.result(); + result["params"]["ctx"] = json!(4096); + result["protocol"] = json!({"schema":"basert-throughput-protocol/2","profile":"basert-bench-capacity/1", + "context_isolation":"headline_then_per_prefill","context_capacity_policy":"basert_bench_default", + "model_load_in_timing":false,"execution_layout":"headline_then_prefill_processes", + "execution_order":["pp512","tg128","pp128"], + "prefill":{"128":{"initial_context_tokens":0,"context_capacity_tokens":4096}, + "512":{"initial_context_tokens":0,"context_capacity_tokens":4096}}, + "decode":{"initial_context_tokens":1,"context_capacity_tokens":4096,"seed_prefill_in_timing":false}, + "measurement":{"timed_repetitions":2,"requested_warmup_repetitions":3, + "warmup_policy":"fixed_repetitions","minimum_warmup_s":0, + "telemetry":"disabled","cooldown":false,"timing":"harness_existing_token_operations"}}); + self.install(&result, ""); + } + + fn run_command(&self, extra: &[&str]) -> Command { let mut command = self.command(); command.arg(self.runtime).arg("run").arg(&self.model); if !self.is_default_sweep() { @@ -590,9 +637,8 @@ impl Fixture { command .args(["--reps", "2", "--yes", "--output"]) .arg(&self.report) - .args(extra) - .output() - .unwrap() + .args(extra); + command } fn signed(&self) -> Value { @@ -735,19 +781,8 @@ fn basert_native_same_run_telemetry_is_selected_by_capability_not_version() { #[test] fn headline_capable_basert_signs_new_metadata_without_changing_requested_repetitions() { let f = Fixture::new("basert"); - let mut result = f.result(); - result["params"]["ctx"] = json!(4096); - result["protocol"] = json!({"schema":"basert-throughput-protocol/2","profile":"basert-bench-capacity/1", - "context_isolation":"headline_then_per_prefill","context_capacity_policy":"basert_bench_default", - "model_load_in_timing":false,"execution_layout":"headline_then_prefill_processes", - "execution_order":["pp512","tg128","pp128"], - "prefill":{"128":{"initial_context_tokens":0,"context_capacity_tokens":4096}, - "512":{"initial_context_tokens":0,"context_capacity_tokens":4096}}, - "decode":{"initial_context_tokens":1,"context_capacity_tokens":4096,"seed_prefill_in_timing":false}, - "measurement":{"timed_repetitions":2,"requested_warmup_repetitions":3, - "warmup_policy":"fixed_repetitions","minimum_warmup_s":0, - "telemetry":"disabled","cooldown":false,"timing":"harness_existing_token_operations"}}); - f.install(&result, ""); + f.install_headline_capable(); + let result = f.result(); let signed = f.signed(); assert_eq!( signed["benchmark"]["protocol"]["id"], @@ -1391,6 +1426,218 @@ fn saved_report_lists_mark_partial_runs_as_local_only() { .starts_with("Partial run, not submittable")); } +/// Serves one "latest release" answer the way GitHub does, on a loopback port. +fn release_feed(tag: &str) -> (String, thread::JoinHandle<()>) { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("http://{}/latest", listener.local_addr().unwrap()); + let body = json!({"tag_name": tag, "html_url": "https://example.test/release"}).to_string(); + let handle = thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + let mut request = Vec::new(); + let mut buffer = [0; 1024]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = stream.read(&mut buffer).unwrap(); + assert!(read > 0); + request.extend_from_slice(&buffer[..read]); + } + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ) + .unwrap(); + }); + (url, handle) +} + +/// How the advice ends depends on whether BaseRT publishes a bundle for the +/// machine running the tests; both endings are correct. +fn names_a_way_to_update(printed: &str) -> bool { + printed.contains("Update with `computearena basert install`") + || printed.contains("No prebuilt BaseRT is published for this platform") +} + +#[test] +fn an_older_basert_is_named_before_the_plan_with_what_it_signs_and_how_to_update() { + let f = Fixture::new("basert"); + let output = f.run(&[]); + success(&output); + let printed = text(&output); + let summary = "BaseRT 0.2.4 predates the current benchmark protocol."; + let notice = printed.find(summary).expect(&printed); + assert!( + notice < printed.find("Benchmark plan").unwrap(), + "{printed}" + ); + assert!( + printed.contains("signed as computearena-throughput-legacy/1, marked not comparable"), + "{printed}" + ); + // Offline, the release to move to is the first one with the protocol. + assert!( + printed.contains("BaseRT 0.2.5 or newer measures PP512 and TG128 first"), + "{printed}" + ); + assert!(names_a_way_to_update(&printed), "{printed}"); + // Said once: the end of the run does not repeat it. + assert_eq!(printed.matches(summary).count(), 1, "{printed}"); + // It is advice, not a gate: the report is signed as before. + let report: Value = serde_json::from_slice(&fs::read(&f.report).unwrap()).unwrap(); + assert_eq!( + report["benchmark"]["protocol"]["id"], + "computearena-throughput-legacy/1" + ); + success(&f.verify()); + + // With the newest release known, the advice names it. + let f = Fixture::new("basert"); + f.remember_latest_basert("0.2.6"); + let printed = text(&f.run(&[])); + assert!( + printed.contains("BaseRT 0.2.6 measures PP512 and TG128 first"), + "{printed}" + ); + + // llama.cpp has nothing to do with any of this. + let llama = Fixture::new("llama-cpp"); + llama.remember_latest_basert("9.9.9"); + let printed = text(&llama.run(&[])); + assert!(!printed.contains("BaseRT"), "{printed}"); +} + +#[test] +fn a_current_basert_is_told_about_a_newer_release_and_nothing_else() { + let f = Fixture::new("basert"); + f.install_headline_capable(); + f.remember_latest_basert("0.2.4"); + let output = f.run(&[]); + success(&output); + let printed = text(&output); + assert!(!printed.contains("is available"), "{printed}"); + assert!(!printed.contains("predates"), "{printed}"); + + fs::remove_file(&f.report).unwrap(); + f.remember_latest_basert("9.9.9"); + let output = f.run(&[]); + success(&output); + let printed = text(&output); + let notice = printed + .find("BaseRT 9.9.9 is available (installed: 0.2.4).") + .expect(&printed); + assert!( + notice < printed.find("Benchmark plan").unwrap(), + "{printed}" + ); + assert!(!printed.contains("predates"), "{printed}"); + assert!(names_a_way_to_update(&printed), "{printed}"); + + // A harness named by hand is not something an install would replace. + fs::remove_file(&f.report).unwrap(); + let output = f + .run_command(&["--runtime-path", f.executable.to_str().unwrap()]) + .output() + .unwrap(); + success(&output); + let printed = text(&output); + assert!( + printed.contains("This harness was chosen with --runtime-path"), + "{printed}" + ); +} + +#[test] +fn the_release_lookup_runs_beside_the_benchmark_and_is_remembered() { + let f = Fixture::new("basert"); + f.install_headline_capable(); + let (feed, served) = release_feed("v9.9.9"); + let output = f + .run_command(&[]) + .env("COMPUTEARENA_BASERT_RELEASE_API", &feed) + .output() + .unwrap(); + success(&output); + served.join().unwrap(); + let printed = text(&output); + // Whether the answer arrived before the plan or during the run, it is + // said exactly once. + assert_eq!( + printed + .matches("BaseRT 9.9.9 is available (installed: 0.2.4).") + .count(), + 1, + "{printed}" + ); + let remembered: Value = serde_json::from_slice( + &fs::read(f.dir.path().join("data/basert-update-check.json")).unwrap(), + ) + .unwrap(); + assert_eq!(remembered["latestVersion"], "9.9.9"); + + // The next run answers from that, without a lookup: the feed is gone. + fs::remove_file(&f.report).unwrap(); + let output = f + .run_command(&[]) + .env("COMPUTEARENA_BASERT_RELEASE_API", &feed) + .output() + .unwrap(); + success(&output); + let printed = text(&output); + let notice = printed + .find("BaseRT 9.9.9 is available (installed: 0.2.4).") + .expect(&printed); + assert!( + notice < printed.find("Benchmark plan").unwrap(), + "{printed}" + ); +} + +#[test] +fn an_unreachable_release_feed_never_delays_or_fails_a_run() { + let f = Fixture::new("basert"); + f.install_headline_capable(); + let started = Instant::now(); + let output = f.run(&[]); + success(&output); + assert!(started.elapsed() < Duration::from_secs(20)); + let printed = text(&output); + assert!(!printed.contains("is available"), "{printed}"); + assert!(!f.dir.path().join("data/basert-update-check.json").exists()); +} + +#[test] +fn the_printed_session_names_an_older_basert_when_it_finds_it() { + let f = Fixture::new("basert"); + f.remember_latest_basert("0.2.6"); + let mut child = f + .command() + .arg("basert") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + child.stdin.take().unwrap().write_all(b"6\n").unwrap(); + let output = child.wait_with_output().unwrap(); + success(&output); + let printed = text(&output); + let found = printed.find("Found BaseRT 0.2.4").expect(&printed); + let notice = printed + .find("BaseRT 0.2.4 predates the current benchmark protocol.") + .expect(&printed); + assert!(found < notice, "{printed}"); + assert!( + notice < printed.find("Run benchmarks").unwrap(), + "{printed}" + ); + assert!( + printed.contains("BaseRT 0.2.6 measures PP512 and TG128 first"), + "{printed}" + ); +} + #[test] fn offline_report_cannot_be_uploaded_without_login() { let f = Fixture::full_sweep("basert"); diff --git a/crates/computearena-cli/tests/runtime_flow.rs b/crates/computearena-cli/tests/runtime_flow.rs index 603fd29..8ad3b72 100644 --- a/crates/computearena-cli/tests/runtime_flow.rs +++ b/crates/computearena-cli/tests/runtime_flow.rs @@ -123,6 +123,11 @@ fn basert_uses_the_same_signed_binary_identity_flow() { fs::set_permissions(&executable, fs::Permissions::from_mode(0o755)).unwrap(); let report = dir.path().join("report.json"); let output = Command::new(env!("CARGO_BIN_EXE_computearena")) + // Tests never ask GitHub which BaseRT release is newest. + .env( + "COMPUTEARENA_BASERT_RELEASE_API", + "http://127.0.0.1:1/latest", + ) .args(["basert", "--runtime-path"]) .arg(&executable) .arg("--data-dir") diff --git a/crates/computearena-cli/tests/runtime_setup.rs b/crates/computearena-cli/tests/runtime_setup.rs index 2e03598..4648d2f 100644 --- a/crates/computearena-cli/tests/runtime_setup.rs +++ b/crates/computearena-cli/tests/runtime_setup.rs @@ -90,6 +90,11 @@ impl Sandbox { .env("NO_PROXY", "*") .env("BASERT_INSTALL_DIR", self.path().join("basert-home")) .env("COMPUTEARENA_API_URL", "http://127.0.0.1:1/api/v1") + // Tests never ask GitHub which BaseRT release is newest. + .env( + "COMPUTEARENA_BASERT_RELEASE_API", + "http://127.0.0.1:1/latest", + ) .arg("--data-dir") .arg(self.path().join("data")); cmd diff --git a/docs/testing.md b/docs/testing.md index 3f97a12..802c44b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -46,6 +46,7 @@ They do not run GPU workloads. See [telemetry.md](telemetry.md) for collector co | Run identity | New runs get distinct IDs while preserving the installation key | | Submission | Actual JSON upload body equals the signed report; anonymous submission; public-access notice; informational checksum mismatch/download guidance; duplicate HTTP 200 is successful | | Batch handling | Invalid reports require explicit skip for noninteractive partial uploads; only valid reports sent; all-invalid batches rejected locally | +| BaseRT updates | An older harness is named before the plan, once, with what its report is signed as and how to update; a current harness hears only about a newer release; the release lookup runs beside the benchmark, is remembered for the next run, and an unreachable feed neither delays nor fails a run; a harness chosen by hand is not offered an install; llama.cpp is unaffected. No test contacts GitHub: the lookup is pointed at a closed port or a loopback feed | | Full sweep only | The default run is submittable; a custom `--pp`/`--tg` run is announced as local only in the plan and after the run, listed as `LOCAL ONLY`, and refused at submission with the missing workloads before login or any upload; in a batch it is left local while complete runs upload | | Failure recovery | HTTP 422 allows the next report; HTTP 429/500 stop the queue; saved reports remain available |