Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
cd368a2
chore: bump version to 0.1.1
IsuraManchanayake Sep 10, 2026
33b120e
feat(models): unify acquisition and provenance
IsuraManchanayake Sep 11, 2026
1ee6a1c
feat(models): verify artifact identity before submission
IsuraManchanayake Sep 11, 2026
a4ef1d9
feat(models): explain report trust boundaries
IsuraManchanayake Sep 11, 2026
cf63021
feat(cli): negotiate supported client versions
IsuraManchanayake Sep 11, 2026
89309e0
fix(basert): explain Linux x86-64 availability
IsuraManchanayake Sep 11, 2026
32a50cf
Merge pull request #25 from basecompute/feat/client-version-compatibi…
IsuraManchanayake Sep 11, 2026
da6e847
test(runtimes): account for unsupported BaseRT platforms
IsuraManchanayake Sep 11, 2026
90526db
test(runtime): expect Linux x86 BaseRT guidance
IsuraManchanayake Sep 11, 2026
26470c2
Merge pull request #23 from basecompute/fix/basert-linux-x86-notice
IsuraManchanayake Sep 11, 2026
80fce6f
fix(models): stop inventing artifact paths and let the server settle …
IsuraManchanayake Sep 11, 2026
9beb1b0
Merge pull request #26 from basecompute/fix/basert-artifact-claims
IsuraManchanayake Sep 11, 2026
781e74a
Merge branch 'rc-0.1.1' into feat/model-verification-preflight
IsuraManchanayake Sep 11, 2026
53363b7
Merge pull request #22 from basecompute/feat/model-verification-prefl…
IsuraManchanayake Sep 11, 2026
f2ec60c
docs: add v0.1.1 release notes
IsuraManchanayake Sep 11, 2026
8d8b6df
Merge pull request #27 from basecompute/docs/v0.1.1-release-notes
IsuraManchanayake Sep 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/computearena-cli"]

[workspace.package]
version = "0.1.0"
version = "0.1.1"
edition = "2021"
license = "Apache-2.0"
rust-version = "1.85"
Expand All @@ -14,12 +14,14 @@ base64 = "0.22"
clap = { version = "4", features = ["derive"] }
dialoguer = { version = "0.12", default-features = false, features = ["fuzzy-select"] }
dirs = "5"
percent-encoding = "2"
libc = "0.2"
ed25519-dalek = { version = "2", features = ["rand_core"] }
flate2 = "1"
rand_core = { version = "0.6", features = ["getrandom"] }
ratatui = "0.30"
reqwest = { version = "0.13", default-features = false, features = ["blocking", "rustls"] }
semver = "1"
serde_json = { version = "1", features = ["preserve_order", "float_roundtrip"] }
sha2 = { version = "0.10", features = ["asm"] }
tar = "0.4"
Expand Down
63 changes: 47 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,11 @@ or, from a checkout, `cargo build --release`, which writes
`brew install llama.cpp` or its
[releases](https://github.com/ggml-org/llama.cpp/releases). The client can
also install either one for you (see [Runtimes](#runtimes)).
2. Have a model on disk: a `.base` bundle for BaseRT, for example from
`basert pull Qwen/Qwen3-0.6B`, or a `.gguf` file for llama.cpp. For
llama.cpp the model picker can also search the Hugging Face Hub and
download a GGUF for you, smallest quantization first; set `HF_TOKEN` in
your environment for gated repositories.
2. Choose a model. The BaseRT picker can browse BaseRT's public catalogue and
run its backend-aware `basert pull` flow; the llama.cpp picker can search
Hugging Face and download a GGUF, smallest file first. Existing `.base` and
`.gguf` files can still be entered directly. Set `HF_TOKEN` for gated
Hugging Face repositories.
3. Run `computearena`. In a terminal this opens the full-screen interface:
arrow keys move, Enter selects, Esc goes back, Ctrl+C leaves, and the
wheel or PgUp/PgDn scrolls long output. It asks which
Expand All @@ -123,11 +123,12 @@ computearena llama-cpp run model.gguf # benchmark and save a signed report
computearena list # saved reports (--json for machines)
computearena inspect <run-id-or-path> # print one report
computearena verify <run-id-or-path> # check its signature
computearena identify model.gguf <hf-file-url> # verify a manually downloaded model
computearena login # connect this installation to your account
computearena submit # upload chosen reports
```

`run`, `list`, `inspect`, `verify`, `login`, `logout`, `install`, and `submit`
`run`, `list`, `inspect`, `verify`, `identify`, `login`, `logout`, `install`, and `submit`
work under either runtime selector. Without a selector they act as BaseRT
commands, so scripts written for earlier versions keep working. `--data-dir`,
`--runtime-path` (alias `--harness`), and `--api-url` are accepted anywhere.
Expand Down Expand Up @@ -188,15 +189,20 @@ copy installed the same way. Neither touches shell profiles. `--archive`
unpacks a bundle you already have instead of contacting GitHub. Prebuilt
runtime bundles exist for macOS arm64 and Linux arm64 with CUDA (BaseRT) and
for macOS and Linux CPU or Metal builds (llama.cpp); other platforms and GPU
builds of llama.cpp are installed by hand.
builds of llama.cpp are installed by hand. ComputeArena warns Linux x86-64
users before entering BaseRT because no prebuilt BaseRT runtime is currently
published for that architecture; a compatible manually built harness still works.

### BaseRT

The harness must advertise the `basert-benchmark-harness/1` protocol through
`describe --json`. `COMPUTEARENA_BASERT_HARNESS` is the environment equivalent
of `--runtime-path`. BaseRT 0.2.4 and newer can also start this client with
`basert computearena`, provided `computearena` is beside `basert` or on
`PATH`.
`PATH`. The model picker uses the `basert` command beside the harness, on
`PATH`, or in BaseRT's install directory to list and pull catalogue models.
BaseRT remains responsible for choosing a compatible backend artifact,
downloading split files, conversion, and writing `hub.json` provenance.

### llama.cpp

Expand All @@ -222,9 +228,10 @@ under a `basert/computearena` directory instead; the first run moves that
directory whole, reports, key, sessions, and installed runtimes included,
and says so. Inside it, `reports/` holds one
JSON file per run, `keys/installation.ed25519` is the private signing key
created on first use, `auth.json` holds login sessions, and
`runtimes/` holds llama.cpp builds the client installed. `--output` writes a
report elsewhere instead.
created on first use, `auth.json` holds login sessions, `runtimes/` holds
llama.cpp builds the client installed, and `model-provenance/` holds
content-addressed receipts for models acquired or identified through
ComputeArena. `--output` writes a report elsewhere instead.

Each report is a `computearena-benchmark/1` envelope: a run ID, a timestamp,
the client version, the runtime (name, version, adapter descriptor, and the
Expand All @@ -244,10 +251,28 @@ sign invented numbers. The executable and model hashes identify what was
claimed to run; they do not attest the process, its libraries, or GPU kernels,
and they cannot see a file swapped and restored between the two hashes.

Model identity is deliberately left unresolved. The name and quantization
embedded in the file are kept as display metadata and marked `unverified`. No
filename catalogue or override decides what a file really is, and instruct,
MoE, revision, and fine-tune variants are never assumed equivalent.
Model identity separates three things in `computearena-model/1`: the canonical
Hugging Face model class, the converted artifact repository/revision/path, and
the SHA-256 of the exact local bytes. Quantization is namespaced by format, so
BaseRT Q4 and GGUF Q4_K_M remain distinct variants of a model class. Downloads
are pinned to immutable Hugging Face revisions and checked against published
LFS SHA-256 values when available. BaseRT `hub.json`, standard Hugging Face
cache paths, and GGUF source metadata provide conservative fallback evidence.
Ambiguous merges and files with no evidence stay unresolved rather than being
grouped by filename.

For a file copied or downloaded outside either model picker, bind it to the
exact Hub object before benchmarking:

```sh
computearena identify ./model.gguf \
https://huggingface.co/owner/repository/blob/<revision>/path/to/model.gguf
```

This succeeds only when the local SHA-256 matches the Hub's file object. It
does not let a user assign a model name without evidence. See
[docs/model-identity.md](docs/model-identity.md) for the report fields and
trust limits.

Chip names pass through one normalization before signing, so known aliases
(M5Pro and Apple M5 Pro, or GB10 and NVIDIA GB10) receive one name while the
Expand Down Expand Up @@ -279,7 +304,13 @@ Submitting the same report again succeeds rather than failing.

The client talks to `https://computearena.ai/api/v1`. `--api-url` or
`COMPUTEARENA_API_URL` point it at another deployment, such as a local
development server; sessions are kept per URL.
development server; sessions are kept per URL. Interactive sessions check the latest
stable GitHub release in the background, cache successful checks for 24 hours,
and show a quickstart link only when a newer semantic version exists. Network
failures never block offline benchmarking. API calls include the client version
in both the user agent and x-computearena-client-version header; a server retiring
an old client should return HTTP 426 with client_upgrade_required and may include
minimumClientVersion for an actionable upgrade message.

## Configuration

Expand Down
2 changes: 2 additions & 0 deletions crates/computearena-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,13 @@ base64.workspace = true
clap.workspace = true
dialoguer.workspace = true
dirs.workspace = true
percent-encoding.workspace = true
ed25519-dalek.workspace = true
flate2.workspace = true
rand_core.workspace = true
ratatui.workspace = true
reqwest.workspace = true
semver.workspace = true
serde_json.workspace = true
sha2.workspace = true
tar.workspace = true
Expand Down
4 changes: 2 additions & 2 deletions crates/computearena-cli/src/adapters/basert.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ impl RuntimeAdapter for BaseRtAdapter {
serde_json::from_slice(&output.stdout).context("reading BaseRT benchmark capabilities")
}

fn select_model(&self, _paths: &crate::reports::Paths) -> Result<Option<PathBuf>> {
crate::models::prompt_model_path()
fn select_model(&self, paths: &crate::reports::Paths) -> Result<Option<PathBuf>> {
crate::models::prompt_model_path(paths)
}

fn confirm(&self, r: &BenchmarkRequest<'_>, yes: bool) -> Result<Option<bool>> {
Expand Down
10 changes: 9 additions & 1 deletion crates/computearena-cli/src/adapters/gguf.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,12 @@ pub(crate) fn inspect(path: &Path) -> Result<Value> {
let item = value(&mut file, kind, 0)?;
if matches!(
key.as_str(),
"general.name" | "general.architecture" | "general.file_type"
"general.name"
| "general.architecture"
| "general.file_type"
| "general.repo_url"
| "general.source.repo_url"
| "general.base_model.0.repo_url"
) {
metadata.insert(key, item);
}
Expand Down Expand Up @@ -110,6 +115,9 @@ pub(crate) fn inspect(path: &Path) -> Result<Value> {
"architecture": metadata.get("general.architecture"),
"quantization": quant, "gguf_file_type": file_type,
"format": "gguf", "format_version": version,
"repo_url": metadata.get("general.repo_url"),
"source_repo_url": metadata.get("general.source.repo_url"),
"base_model_repo_url": metadata.get("general.base_model.0.repo_url"),
"file_name": path.file_name().and_then(|s| s.to_str()),
"size_bytes": file.metadata()?.len()
}))
Expand Down
68 changes: 67 additions & 1 deletion crates/computearena-cli/src/api.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,23 @@
use crate::config::HTTP_CONNECT_TIMEOUT;
use crate::config::{COMPUTEARENA_INSTALL_SCRIPT, HTTP_CONNECT_TIMEOUT};
use anyhow::{Context, Result};
use reqwest::blocking::Client;
use reqwest::header::{HeaderMap, HeaderValue};
use serde_json::Value;
use std::time::Duration;

pub(crate) const CLIENT_VERSION_HEADER: &str = "x-computearena-client-version";

pub(crate) fn client(request_timeout: Duration) -> Result<Client> {
let mut headers = HeaderMap::new();
headers.insert(
CLIENT_VERSION_HEADER,
HeaderValue::from_static(env!("CARGO_PKG_VERSION")),
);
Client::builder()
.connect_timeout(HTTP_CONNECT_TIMEOUT)
.timeout(request_timeout)
.user_agent(format!("computearena/{}", env!("CARGO_PKG_VERSION")))
.default_headers(headers)
.build()
.context("building ComputeArena HTTP client")
}
Expand All @@ -28,3 +37,60 @@ pub(crate) fn error_code(body: &str) -> Option<String> {
.as_str()
.map(str::to_owned)
}

/// Turn a structured API failure into an actionable message. The server can
/// retire an old client by returning HTTP 426 (or the matching error code)
/// without making older clients fail with an unexplained generic status.
pub(crate) fn server_error(status: reqwest::StatusCode, body: &str) -> String {
let message =
error_message(body).unwrap_or_else(|| format!("server returned HTTP {}", status.as_u16()));
let upgrade_required = status == reqwest::StatusCode::UPGRADE_REQUIRED
|| matches!(
error_code(body).as_deref(),
Some("client_upgrade_required" | "unsupported_client_version")
);
if !upgrade_required {
return message;
}
let value = serde_json::from_str::<Value>(body).ok();
let minimum = value.as_ref().and_then(|value| {
[
"/error/details/minimumClientVersion",
"/error/minimumClientVersion",
"/minimumClientVersion",
]
.into_iter()
.find_map(|pointer| value.pointer(pointer).and_then(Value::as_str))
});
let requirement = minimum
.map(|version| format!(" The server requires {version} or newer."))
.unwrap_or_default();
format!(
"{message}\nInstalled ComputeArena CLI: {}.{requirement}\nUpdate with: {COMPUTEARENA_INSTALL_SCRIPT}",
env!("CARGO_PKG_VERSION")
)
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn upgrade_errors_include_the_installed_and_required_versions() {
let body = r#"{"error":{"code":"client_upgrade_required","message":"This client is no longer supported","details":{"minimumClientVersion":"0.2.0"}}}"#;
let message = server_error(reqwest::StatusCode::UPGRADE_REQUIRED, body);
assert!(message.contains("This client is no longer supported"));
assert!(message.contains(env!("CARGO_PKG_VERSION")));
assert!(message.contains("0.2.0 or newer"));
assert!(message.contains("computearena.ai/install.sh"));
}

#[test]
fn ordinary_api_errors_are_unchanged() {
let body = r#"{"error":{"code":"bad_request","message":"Invalid report"}}"#;
assert_eq!(
server_error(reqwest::StatusCode::BAD_REQUEST, body),
"Invalid report"
);
}
}
17 changes: 4 additions & 13 deletions crates/computearena-cli/src/auth.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::api::{
client as api_client, error_code as api_error_code, error_message as api_error_message,
client as api_client, error_code as api_error_code, server_error as api_server_error,
};
use crate::config::{
AUTH_HTTP_TIMEOUT, DEFAULT_API_URL, DEFAULT_DEVICE_AUTH_EXPIRES_SECS,
Expand Down Expand Up @@ -71,11 +71,7 @@ pub(crate) fn login(paths: &Paths, api_url: &str) -> Result<()> {
let status = response.status();
let body = response.text().unwrap_or_default();
if !status.is_success() {
bail!(
"{}",
api_error_message(&body)
.unwrap_or_else(|| format!("server returned HTTP {}", status.as_u16()))
);
bail!("{}", api_server_error(status, &body));
}
let device: Value = serde_json::from_str(&body).context("parsing login response")?;
let device_code = required_json_string(&device, "deviceCode")?;
Expand Down Expand Up @@ -132,11 +128,7 @@ pub(crate) fn login(paths: &Paths, api_url: &str) -> Result<()> {
if api_error_code(&body).as_deref() == Some("authorization_pending") {
continue;
}
bail!(
"{}",
api_error_message(&body)
.unwrap_or_else(|| format!("server returned HTTP {}", status.as_u16()))
);
bail!("{}", api_server_error(status, &body));
}
bail!("login code expired; run `computearena login` again")
}
Expand All @@ -161,8 +153,7 @@ fn validate_api_session(api_url: &str, session: &ApiSession) -> Result<Option<St
}
bail!(
"could not validate saved login: {}",
api_error_message(&body)
.unwrap_or_else(|| format!("server returned HTTP {}", status.as_u16()))
api_server_error(status, &body)
)
}

Expand Down
Loading
Loading