From e6a9524a08c26e1da23d8e0a6cf9333655e25747 Mon Sep 17 00:00:00 2001 From: IsuraManchanayake Date: Thu, 17 Sep 2026 16:16:24 +1000 Subject: [PATCH 1/3] docs: add v0.1.2 release notes Curated notes for the 0.1.2 release body: complete runs only, the headline-first order and BaseRT's 4K headline reservation, the unmodified llama-bench path and what it records instead, signed host environment boundaries, and the explanation for previously deleted benchmarks. --- docs/release-notes/0.1.2.md | 125 ++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 docs/release-notes/0.1.2.md diff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.md new file mode 100644 index 0000000..1e1bd3b --- /dev/null +++ b/docs/release-notes/0.1.2.md @@ -0,0 +1,125 @@ +# ComputeArena CLI 0.1.2 + +This release changes how a benchmark runs and which benchmarks can be +published. The headline workloads, PP512 and TG128, now run before the rest of +the prefill sweep, reports sign a description of the host they ran on, and +computearena.ai accepts only complete runs, so every published benchmark can be +compared with every other at every size. It also explains what happens when a +benchmark you deleted on the website is submitted again. + +## Install or upgrade + +On macOS or Linux: + +```sh +curl -LsSf https://computearena.ai/install.sh | sh +``` + +The same command upgrades an existing installation. Saved reports, the +installation signing key, login sessions, model receipts, and installed +runtimes are kept. Release archives and checksums are also available below for +manual installation. + +## Highlights + +- **Only complete runs are published.** A run is submittable when it contains + the full default sweep: every default prefill size, PP128 to PP16384, and + TG128. That is what `run` does when `--pp` and `--tg` are left out. Extra + prefill sizes are fine, and the number of repetitions is not part of the + rule. A run with a custom `--pp` or `--tg` is still a valid signed report + that you can save, inspect, and verify, but it is local only, and the client + says so at every step: in the benchmark plan before the run is confirmed, + with the workloads it will lack; after the run, in place of the submit hint; + as `LOCAL ONLY` in `list` and in the submission pickers, where such reports + are not pre-selected; and at `submit`, where partial runs are listed apart + from invalid reports and refused before login or any upload, while complete + runs in the same batch still go through. computearena.ai enforces the same + rule, so earlier client releases receive the same explanation from the + server. +- **Headline first.** PP512 runs first, then TG128, then the remaining prefill + sizes; a custom sweep without PP512 uses its first size as the headline. + Prefill starts from an empty context and decode starts from one untimed seed + token. Warmup, repetitions, timers, and telemetry are unchanged for both + runtimes. +- **BaseRT reserves 4K for the headline.** A BaseRT harness that advertises + `features.headline_context_capacity` is run with `--headline-first`: the + headline is measured with a 4K context reservation, and the client checks + the returned capacity, history, order, and repetitions before signing. These + reports normalize to `computearena-throughput/3`. Older harnesses keep their + existing invocation and recorded protocol, with no new flags passed to them. +- **Stock llama-bench, differences recorded.** ComputeArena still runs your + unmodified `llama-bench`. It has no independent context reservation option, + so its native PP+TG+depth request is kept and the report records that the 4K + target was not applied; `-d 4096` is never used as a substitute, because it + would add real token history. Standard runs use one process for the + headline, one for decode, and one for the remaining prefill sizes; thermally + controlled runs still use one process per workload. The workload order, + warmup, and context requests are part of the signed report. +- **The host is part of the signed report.** Signed before-and-after + environment boundaries (`computearena-environment/1`) record the OS family, + version and kernel, logical and physical CPU counts, total and available + memory, swap use, macOS memory pressure or Linux PSI, the power or + performance mode, and the available GPU configuration. Slow probes run only + outside the measured process windows, and static macOS GPU details are read + once rather than at every boundary. +- **Deleted benchmarks say so.** A benchmark deleted on the website cannot be + published again from the same saved report. The client says this before a + submission, labels such a report **Failed: previously deleted**, keeps + uploading the other selected reports, and ends with the uploaded, duplicate, + and failed counts and a nonzero exit status. Local files are unchanged. To + publish that result again, run the benchmark again with the same settings + and submit the new report; it counts as a separate benchmark. + +## Reports and compatibility + +Reports saved by 0.1.0 and 0.1.1 remain verifiable. They remain submittable +when they contain the full default sweep, which is what those releases produce +unless `--pp` or `--tg` was passed. Benchmarks already published on +computearena.ai are not changed by this release. + +Matching workload labels do not promise identical conditions across runtimes. +BaseRT and llama.cpp differ in warmup, context reservation, and process layout; +those differences are recorded in each signed report rather than hidden, and +computearena.ai does not split rankings or add a filter because of them. + +## Supported systems + +Prebuilt ComputeArena binaries are available for: + +- macOS on Apple Silicon; +- Linux x86_64 with glibc 2.31 or newer; and +- Linux arm64 with glibc 2.31 or newer. + +Windows and Intel Macs are not supported in this release. ComputeArena does not +bundle an inference runtime or model; install BaseRT or llama.cpp and obtain a +compatible model before benchmarking. Prebuilt BaseRT runtimes exist for macOS +on Apple Silicon and for Linux arm64 with CUDA; on Linux x86-64, use llama.cpp +or a BaseRT benchmark harness you built yourself. + +The macOS binary is not signed with an Apple Developer ID. The release archive +is checksum-verified and carries a Sigstore keyless signature from the release +workflow. Browser-downloaded copies may need their quarantine attribute removed +as described in the README. + +## Trust and limits + +A valid signature shows that a report was not changed after it was signed. It +does not attest that the runtime executed the benchmark honestly or that the +recorded host description is truthful; computearena.ai recomputes every rate +from the signed raw samples and repeats the model identity checks on its own, +because a public client is not a trust boundary. Unless a harness reports its +own same-run telemetry, telemetry is observed from outside the runtime at +one-second intervals, so short peaks can be missed, and a process-window +reading is not a per-token metric. + +Accuracy benchmarks, MLX, vLLM, Windows, and Intel macOS support are not part +of v0.1.2. + +For usage and measurement details, see the +[README](https://github.com/basecompute/computearena-cli#readme), +[docs/benchmark-profiles.md](https://github.com/basecompute/computearena-cli/blob/main/docs/benchmark-profiles.md), +[docs/runtime-adapters.md](https://github.com/basecompute/computearena-cli/blob/main/docs/runtime-adapters.md), +and +[docs/telemetry.md](https://github.com/basecompute/computearena-cli/blob/main/docs/telemetry.md). +Questions and feedback are welcome in the +[ComputeArena Discord](https://discord.gg/CCT24GWhPG). From 882f77dbdc0e00d4004ffb0dffb81a9f70be8283 Mon Sep 17 00:00:00 2001 From: IsuraManchanayake Date: Thu, 17 Sep 2026 16:24:44 +1000 Subject: [PATCH 2/3] docs: name the BaseRT release the 0.1.2 behaviour depends on The headline-first path and same-run telemetry are selected by what the installed harness advertises, and BaseRT 0.2.5 is the first release that advertises them. Say so, say how to upgrade BaseRT, and say that the client does not announce a newer BaseRT by itself. --- docs/release-notes/0.1.2.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.md index 1e1bd3b..49fb47b 100644 --- a/docs/release-notes/0.1.2.md +++ b/docs/release-notes/0.1.2.md @@ -20,6 +20,13 @@ installation signing key, login sessions, model receipts, and installed runtimes are kept. Release archives and checksums are also available below for manual installation. +BaseRT users should also upgrade BaseRT to 0.2.5 or newer, with +`computearena basert install` or the official BaseRT installer. ComputeArena +selects the new BaseRT behaviour below by what the installed harness +advertises, and it does not yet tell you when a newer BaseRT exists: with +BaseRT 0.2.4 it keeps working exactly as before, and the report records that +the older protocol was used. + ## Highlights - **Only complete runs are published.** A run is submittable when it contains @@ -42,11 +49,15 @@ manual installation. token. Warmup, repetitions, timers, and telemetry are unchanged for both runtimes. - **BaseRT reserves 4K for the headline.** A BaseRT harness that advertises - `features.headline_context_capacity` is run with `--headline-first`: the - headline is measured with a 4K context reservation, and the client checks + `features.headline_context_capacity`, which BaseRT 0.2.5 is the first + release to do, is run with `--headline-first`: the headline is measured in a + freshly loaded model with a 4K context reservation, and the client checks the returned capacity, history, order, and repetitions before signing. These - reports normalize to `computearena-throughput/3`. Older harnesses keep their - existing invocation and recorded protocol, with no new flags passed to them. + reports normalize to `computearena-throughput/3`. The same harness reports + power, energy, temperature, and memory from the timed repetitions + themselves, and the client uses that same-run telemetry in place of + observing the process from outside. Older harnesses keep their existing + invocation and recorded protocol, with no new flags passed to them. - **Stock llama-bench, differences recorded.** ComputeArena still runs your unmodified `llama-bench`. It has no independent context reservation option, so its native PP+TG+depth request is kept and the report records that the 4K From ff9a0bf50f15235c53196a09f2a2c6ebaf6fe461 Mon Sep 17 00:00:00 2001 From: IsuraManchanayake Date: Thu, 17 Sep 2026 16:51:49 +1000 Subject: [PATCH 3/3] docs: describe the BaseRT update notice in the 0.1.2 notes The notes said the client does not tell you when a newer BaseRT exists. With #33 it does: before the plan, in the printed session, and above the full-screen menu with an update key. --- docs/release-notes/0.1.2.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.md index 49fb47b..61b3786 100644 --- a/docs/release-notes/0.1.2.md +++ b/docs/release-notes/0.1.2.md @@ -4,8 +4,9 @@ This release changes how a benchmark runs and which benchmarks can be published. The headline workloads, PP512 and TG128, now run before the rest of the prefill sweep, reports sign a description of the host they ran on, and computearena.ai accepts only complete runs, so every published benchmark can be -compared with every other at every size. It also explains what happens when a -benchmark you deleted on the website is submitted again. +compared with every other at every size. It also says when the installed +BaseRT is worth updating, and explains what happens when a benchmark you +deleted on the website is submitted again. ## Install or upgrade @@ -23,9 +24,9 @@ manual installation. BaseRT users should also upgrade BaseRT to 0.2.5 or newer, with `computearena basert install` or the official BaseRT installer. ComputeArena selects the new BaseRT behaviour below by what the installed harness -advertises, and it does not yet tell you when a newer BaseRT exists: with -BaseRT 0.2.4 it keeps working exactly as before, and the report records that -the older protocol was used. +advertises: with BaseRT 0.2.4 it keeps working exactly as before, the report +records that the older protocol was used, and the client now says so before +the run. ## Highlights @@ -58,6 +59,17 @@ the older protocol was used. themselves, and the client uses that same-run telemetry in place of observing the process from outside. Older harnesses keep their existing invocation and recorded protocol, with no new flags passed to them. +- **BaseRT update notices.** ComputeArena says when the BaseRT it found is + worth updating, and never refuses to run an older one. A harness that + predates the headline-first protocol is named before the benchmark plan, + with what its report will be signed as; that needs no network, because the + harness describes itself. A newer BaseRT release is mentioned the same way, + from a background lookup that is remembered for a day and never delays a + run. The full-screen interface shows the notice above its menu and offers + `u` to install the latest release, asking for a second press before it + replaces anything. A harness chosen with `--runtime-path` or + `COMPUTEARENA_BASERT_HARNESS` is left for you to update, and platforms + without a prebuilt BaseRT are pointed at the release to build from. - **Stock llama-bench, differences recorded.** ComputeArena still runs your unmodified `llama-bench`. It has no independent context reservation option, so its native PP+TG+depth request is kept and the report records that the 4K