diff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.md new file mode 100644 index 0000000..61b3786 --- /dev/null +++ b/docs/release-notes/0.1.2.md @@ -0,0 +1,148 @@ +# ComputeArena CLI 0.1.2 + +This release changes how a benchmark runs and which benchmarks can be +published. The headline workloads, PP512 and TG128, now run before the rest of +the prefill sweep, reports sign a description of the host they ran on, and +computearena.ai accepts only complete runs, so every published benchmark can be +compared with every other at every size. It also says when the installed +BaseRT is worth updating, and explains what happens when a benchmark you +deleted on the website is submitted again. + +## Install or upgrade + +On macOS or Linux: + +```sh +curl -LsSf https://computearena.ai/install.sh | sh +``` + +The same command upgrades an existing installation. Saved reports, the +installation signing key, login sessions, model receipts, and installed +runtimes are kept. Release archives and checksums are also available below for +manual installation. + +BaseRT users should also upgrade BaseRT to 0.2.5 or newer, with +`computearena basert install` or the official BaseRT installer. ComputeArena +selects the new BaseRT behaviour below by what the installed harness +advertises: with BaseRT 0.2.4 it keeps working exactly as before, the report +records that the older protocol was used, and the client now says so before +the run. + +## Highlights + +- **Only complete runs are published.** A run is submittable when it contains + the full default sweep: every default prefill size, PP128 to PP16384, and + TG128. That is what `run` does when `--pp` and `--tg` are left out. Extra + prefill sizes are fine, and the number of repetitions is not part of the + rule. A run with a custom `--pp` or `--tg` is still a valid signed report + that you can save, inspect, and verify, but it is local only, and the client + says so at every step: in the benchmark plan before the run is confirmed, + with the workloads it will lack; after the run, in place of the submit hint; + as `LOCAL ONLY` in `list` and in the submission pickers, where such reports + are not pre-selected; and at `submit`, where partial runs are listed apart + from invalid reports and refused before login or any upload, while complete + runs in the same batch still go through. computearena.ai enforces the same + rule, so earlier client releases receive the same explanation from the + server. +- **Headline first.** PP512 runs first, then TG128, then the remaining prefill + sizes; a custom sweep without PP512 uses its first size as the headline. + Prefill starts from an empty context and decode starts from one untimed seed + token. Warmup, repetitions, timers, and telemetry are unchanged for both + runtimes. +- **BaseRT reserves 4K for the headline.** A BaseRT harness that advertises + `features.headline_context_capacity`, which BaseRT 0.2.5 is the first + release to do, is run with `--headline-first`: the headline is measured in a + freshly loaded model with a 4K context reservation, and the client checks + the returned capacity, history, order, and repetitions before signing. These + reports normalize to `computearena-throughput/3`. The same harness reports + power, energy, temperature, and memory from the timed repetitions + themselves, and the client uses that same-run telemetry in place of + observing the process from outside. Older harnesses keep their existing + invocation and recorded protocol, with no new flags passed to them. +- **BaseRT update notices.** ComputeArena says when the BaseRT it found is + worth updating, and never refuses to run an older one. A harness that + predates the headline-first protocol is named before the benchmark plan, + with what its report will be signed as; that needs no network, because the + harness describes itself. A newer BaseRT release is mentioned the same way, + from a background lookup that is remembered for a day and never delays a + run. The full-screen interface shows the notice above its menu and offers + `u` to install the latest release, asking for a second press before it + replaces anything. A harness chosen with `--runtime-path` or + `COMPUTEARENA_BASERT_HARNESS` is left for you to update, and platforms + without a prebuilt BaseRT are pointed at the release to build from. +- **Stock llama-bench, differences recorded.** ComputeArena still runs your + unmodified `llama-bench`. It has no independent context reservation option, + so its native PP+TG+depth request is kept and the report records that the 4K + target was not applied; `-d 4096` is never used as a substitute, because it + would add real token history. Standard runs use one process for the + headline, one for decode, and one for the remaining prefill sizes; thermally + controlled runs still use one process per workload. The workload order, + warmup, and context requests are part of the signed report. +- **The host is part of the signed report.** Signed before-and-after + environment boundaries (`computearena-environment/1`) record the OS family, + version and kernel, logical and physical CPU counts, total and available + memory, swap use, macOS memory pressure or Linux PSI, the power or + performance mode, and the available GPU configuration. Slow probes run only + outside the measured process windows, and static macOS GPU details are read + once rather than at every boundary. +- **Deleted benchmarks say so.** A benchmark deleted on the website cannot be + published again from the same saved report. The client says this before a + submission, labels such a report **Failed: previously deleted**, keeps + uploading the other selected reports, and ends with the uploaded, duplicate, + and failed counts and a nonzero exit status. Local files are unchanged. To + publish that result again, run the benchmark again with the same settings + and submit the new report; it counts as a separate benchmark. + +## Reports and compatibility + +Reports saved by 0.1.0 and 0.1.1 remain verifiable. They remain submittable +when they contain the full default sweep, which is what those releases produce +unless `--pp` or `--tg` was passed. Benchmarks already published on +computearena.ai are not changed by this release. + +Matching workload labels do not promise identical conditions across runtimes. +BaseRT and llama.cpp differ in warmup, context reservation, and process layout; +those differences are recorded in each signed report rather than hidden, and +computearena.ai does not split rankings or add a filter because of them. + +## Supported systems + +Prebuilt ComputeArena binaries are available for: + +- macOS on Apple Silicon; +- Linux x86_64 with glibc 2.31 or newer; and +- Linux arm64 with glibc 2.31 or newer. + +Windows and Intel Macs are not supported in this release. ComputeArena does not +bundle an inference runtime or model; install BaseRT or llama.cpp and obtain a +compatible model before benchmarking. Prebuilt BaseRT runtimes exist for macOS +on Apple Silicon and for Linux arm64 with CUDA; on Linux x86-64, use llama.cpp +or a BaseRT benchmark harness you built yourself. + +The macOS binary is not signed with an Apple Developer ID. The release archive +is checksum-verified and carries a Sigstore keyless signature from the release +workflow. Browser-downloaded copies may need their quarantine attribute removed +as described in the README. + +## Trust and limits + +A valid signature shows that a report was not changed after it was signed. It +does not attest that the runtime executed the benchmark honestly or that the +recorded host description is truthful; computearena.ai recomputes every rate +from the signed raw samples and repeats the model identity checks on its own, +because a public client is not a trust boundary. Unless a harness reports its +own same-run telemetry, telemetry is observed from outside the runtime at +one-second intervals, so short peaks can be missed, and a process-window +reading is not a per-token metric. + +Accuracy benchmarks, MLX, vLLM, Windows, and Intel macOS support are not part +of v0.1.2. + +For usage and measurement details, see the +[README](https://github.com/basecompute/computearena-cli#readme), +[docs/benchmark-profiles.md](https://github.com/basecompute/computearena-cli/blob/main/docs/benchmark-profiles.md), +[docs/runtime-adapters.md](https://github.com/basecompute/computearena-cli/blob/main/docs/runtime-adapters.md), +and +[docs/telemetry.md](https://github.com/basecompute/computearena-cli/blob/main/docs/telemetry.md). +Questions and feedback are welcome in the +[ComputeArena Discord](https://discord.gg/CCT24GWhPG).