From 0f3ddc6198f69326f79918f02eabe3ca98453de9 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 3 Nov 2026 18:21:37 +0000 Subject: [PATCH 1/2] Move the Arrow examples here from the benchx repository Moves prototype/examples/demo-suite, prototype/examples/demo, prototype/examples/archery-local and prototype/demo.sh (now examples/arrow/demo/run.sh) from benchx-dev/benchx at f49d87e, with the contents unchanged. The next commit adapts paths and docs to the new location, so these examples do not run from here yet. The history of these files stays in benchx; its human authors are listed as co-authors. Co-authored-by: Evgeni Burovski Co-authored-by: Maanas Arora Co-authored-by: Rok Mihevc Co-Authored-By: Claude Sonnet 5.5 --- examples/arrow/archery-local/.gitignore | 2 + examples/arrow/archery-local/README.md | 141 ++++++++++++++++++ examples/arrow/archery-local/activate.sh | 8 + examples/arrow/archery-local/benchx.conf.json | 13 ++ examples/arrow/archery-local/provider.sh | 77 ++++++++++ examples/arrow/archery-local/run.sh | 84 +++++++++++ examples/arrow/demo-suite/CMakeLists.txt | 23 +++ examples/arrow/demo-suite/bench.cpp | 44 ++++++ examples/arrow/demo/.gitignore | 2 + examples/arrow/demo/benchx.conf.json | 10 ++ examples/arrow/demo/run.sh | 136 +++++++++++++++++ 11 files changed, 540 insertions(+) create mode 100644 examples/arrow/archery-local/.gitignore create mode 100644 examples/arrow/archery-local/README.md create mode 100644 examples/arrow/archery-local/activate.sh create mode 100644 examples/arrow/archery-local/benchx.conf.json create mode 100755 examples/arrow/archery-local/provider.sh create mode 100644 examples/arrow/archery-local/run.sh create mode 100644 examples/arrow/demo-suite/CMakeLists.txt create mode 100644 examples/arrow/demo-suite/bench.cpp create mode 100644 examples/arrow/demo/.gitignore create mode 100644 examples/arrow/demo/benchx.conf.json create mode 100644 examples/arrow/demo/run.sh diff --git a/examples/arrow/archery-local/.gitignore b/examples/arrow/archery-local/.gitignore new file mode 100644 index 0000000..6acf7e7 --- /dev/null +++ b/examples/arrow/archery-local/.gitignore @@ -0,0 +1,2 @@ +_work/ +results/ diff --git a/examples/arrow/archery-local/README.md b/examples/arrow/archery-local/README.md new file mode 100644 index 0000000..dfbbe50 --- /dev/null +++ b/examples/arrow/archery-local/README.md @@ -0,0 +1,141 @@ +# archery-local: an end-to-end example + +One commit, built two ways (plain and hardened), compared with noise-aware +verdicts, where the comparison says what was built and how. It follows the +Arrow local story, "Measure what a build option costs" +([`docs/user_stories/apache-arrow-local-benchmarking.md`](../../../docs/user_stories/apache-arrow-local-benchmarking.md)), +on a small stand-in suite (`../demo-suite`: a quiet and a noisy Google +Benchmark case, where hardening adds about 3% of work). + +It exercises the target-provider contract of +[`system-decomposition.md`](../../../docs/design/system-decomposition.md) §3.4: + +- **You are the user, and the provider is your build script.** `provider.sh` + configures and builds with CMake, as `archery` or a contributor would. benchx + builds nothing. +- **The provider tells benchx what it built** in a *target description*: the + build directory and its source, plus two pieces of text you own, `how_built` + (the exact commands) and `activation` (what to run to enter the environment, + with the `shell` it is written for). Either may be empty. +- **You activate the environment yourself** (`source activate.sh`). benchx never + runs `activation`. It records the text, as given, in the work order and in + every result's `provenance.info`, beside the allowlisted environment variables + the runner actually inherited. If the two disagree, both stay and nothing + judges. + +## Run it + +You need `cmake`, a C++ compiler, `git` and `python3` on your `PATH`, and network +access the first time (CMake fetches Google Benchmark v1.9.1 unless it is +installed). From the repository root: + +```console +$ cd prototype +$ python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11, or any environment you like +$ pip install -e . +$ examples/archery-local/run.sh +``` + +It takes about two minutes the first time (two CMake builds and a measured +run) and prints every command before running it, as you would type it. The +script runs `bx` and `python` from the active environment. More rounds: +`ROUNDS=10 ./run.sh`. + +Everything it creates is under `_work/` in this directory, including the local +store (`_work/store.parquet`, set by `paths.store` in `benchx.conf.json`), so nothing +outside this directory changes. `rm -rf _work` starts over. + +## What you will see + +1. **Activate.** `source activate.sh` puts a stand-in dependencies directory on + `LD_LIBRARY_PATH`. This is you, not benchx. +2. **`bx target describe @plain`** fails with exit 3 from the provider: nothing is + built yet. `describe` builds nothing, by contract. +3. **`bx target prepare @plain` / `@hardened`** make the provider build. Its CMake + output is the build's own, on stderr; the target description it prints on + stdout is the answer. `how_built` is the very commands the provider ran. +4. **`bx target describe @hardened`** now works. The same document is stored in + the build directory as `.benchx-target.json`, so a long-lived build directory + carries its own description (point `bx compare` at a bare directory and it + reads that file; no provider needed). +5. **`bx compare @plain @hardened ...`** is the workbench's session loop: it + asks the provider to prepare each side (a no-op build now), then alternates + the sides over the rounds, one work order each, delivers the results to the + local store, and compares: + + ``` + run hardening-... profile environments baseline plain contender hardened + plain: how built: cmake -S .../_work/src -B .../_work/build-plain ... (+1 lines) + plain: activation: source .../activate.sh + plain: observed env: LD_LIBRARY_PATH=.../_work/deps/lib + hardened: how built: cmake -S ... -B .../_work/build-hardened ... -DDEMO_HARDENED=ON (+1 lines) + ... + REGRESSED (1) + BM_Quiet/1024 wall-time effect +2.42% noise 0.57% rounds=5 + NO CHANGE DETECTED (1) + BM_Noisy/1024 wall-time effect +3.43% noise 3.04% rounds=5 + ``` + + Compare that with the story's pains: the comparison now describes itself + (what was built, how, in what environment), and the verdict uses each + benchmark's own noise instead of a flat 5%: the quiet case flags a 2.4% + change that is four times its noise, the noisy case does not flag a 3.4% + swing inside its noise. The numbers vary from run to run; with few rounds the + noisy case may flag too. +6. **The work order and a result**, printed: the order's `target` carries + `how_built`, `activation`, `shell` and `provider`; the result's + `provenance.info` carries them, next to `coordinates.environment.system.process.environment` and + `coordinates.subject.configuration` (read by the runner from + `CMakeCache.txt`). +7. **Forget to activate.** The same comparison with `LD_LIBRARY_PATH` removed + from the environment: the declaration still says `source .../activate.sh`, + the observed environment no longer lists `LD_LIBRARY_PATH`, and nothing + refuses: declarations are recorded as given (`benchmark-environments.md` + §3.2), never checked. + +## Try it by hand + +After `./run.sh` (the build directories and the store stay in `_work/`). From +this directory, with the environment still active: + +```console +$ source activate.sh + +$ bx target describe @hardened # what exists, nothing built +$ bx target prepare @hardened # incremental build, then the description +$ bx target describe _work/build-hardened # a bare directory: its own sidecar + +# the story's own knobs: scope by filter, precision by repetitions +$ bx compare @plain @hardened --label build=plain,hardened --filter BM_Quiet --rounds 10 \ + --repetitions 5 --run-key mine # the adapter, suite, project and profile come from benchx.conf.json + +$ bx compare --run mine --label build --baseline plain # read mode +$ bx head -n 4 # the latest stored results +$ bx head --json -n 1 | python3 -m json.tool | less # provenance.info, environment.system +``` + +Edit `provider.sh`'s `hardened)` case or `activate.sh` and run again to see how +each change shows up in the next comparison. Empty text is fine: set +`"activation": ""` (or `null`, or leave it out) in a description and the +runner simply omits it from the results. + +## What is in this directory + +| File | Role | +|---|---| +| `run.sh` | the end-to-end script | +| `provider.sh` | the target provider: your build script, speaking the `describe`/`prepare` contract | +| `benchx.conf.json` | the project file: the provider command, the harness and its suite, the project name, the comparison profile, and where the store and results go | +| `activate.sh` | the environment you activate before calling `bx` | +| `_work/` | everything generated: source copy, build directories, results, store (git-ignored) | + +The documents involved are in `../../../schemas/`: `target-provider-request`, +`target-description`, and `work-order` (its `target` takes the declared text). + +## What this does not show + +The provider here builds one source tree, `WORKSPACE`. Comparing against a tag +or a past revision (`@plain=v1.0`) is the provider's business and is not +implemented in this example; the prototype's own `demo.sh` shows revisions +with two prebuilt worktrees. A declared `build` object in a description is +refused by the prototype's runner (it reads `CMakeCache.txt` instead). diff --git a/examples/arrow/archery-local/activate.sh b/examples/arrow/archery-local/activate.sh new file mode 100644 index 0000000..cd10b8b --- /dev/null +++ b/examples/arrow/archery-local/activate.sh @@ -0,0 +1,8 @@ +# The environment this example's benchmarks are meant to run in. In the Arrow +# story this is where you would load a toolchain or point at the dependencies +# you built; here it only puts a stand-in "deps" directory on the library +# search path. You source it yourself, before invoking bx: benchx never runs +# it, it records the text that says you did (the target description's +# `activation`) next to the environment it actually inherited. +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +export LD_LIBRARY_PATH="$HERE/_work/deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" diff --git a/examples/arrow/archery-local/benchx.conf.json b/examples/arrow/archery-local/benchx.conf.json new file mode 100644 index 0000000..9029306 --- /dev/null +++ b/examples/arrow/archery-local/benchx.conf.json @@ -0,0 +1,13 @@ +{ + "schema_version": "benchx/config/0.1.0", + "project": "archery-local", + "provider": { "command": ["./provider.sh"] }, + "adapter": { + "name": "google-benchmark", + "suite": "demo-bench", + "quantities": ["wall-time"], + "precision": { "repetitions": 5 } + }, + "paths": { "store": "_work/store.parquet", "results": "_work/results" }, + "compare": { "profile": "environments" } +} diff --git a/examples/arrow/archery-local/provider.sh b/examples/arrow/archery-local/provider.sh new file mode 100755 index 0000000..a5226e0 --- /dev/null +++ b/examples/arrow/archery-local/provider.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# The target provider of this example (system-decomposition.md §3.4). +# +# It is the user's own build script, speaking the provider contract: one JSON +# request on stdin (schemas/target-provider-request/0.1.0), one target +# description on stdout (schemas/target-description/0.1.0). Everything else, +# above all the build's own output, goes to stderr. +# +# describe reports a target that exists and builds nothing; exit 3 if none +# prepare configures and builds it, with CMake, as Arrow's contributors do +# (an incremental no-op when it is already built), then reports it +# +# Configurations: `plain`, and `hardened` (-DDEMO_HARDENED=ON), the same +# commit built two ways, like the -DARROW_HARDENING=ON story. +# +# `how_built` is assembled from the very commands this script runs, so the +# text cannot drift from what happened. benchx never runs any of it. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +WORK=${BENCHX_EXAMPLE_WORK:-$HERE/_work} +export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=demo@example.org +export GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=demo@example.org + +request=$(cat) +field() { python3 -c 'import json,sys; print(json.load(sys.stdin).get(sys.argv[1]) or "")' "$1" <<<"$request"; } +operation=$(field operation) +configuration=$(field configuration) +case "$configuration" in + plain) extra=() ;; + hardened) extra=(-DDEMO_HARDENED=ON) ;; + *) echo "provider: unknown configuration '$configuration' (plain or hardened)" >&2; exit 2 ;; +esac +SRC=$WORK/src +BUILD=$WORK/build-$configuration + +describe() { + [[ -f $BUILD/.benchx-target.json ]] || { echo "provider: $configuration is not built; use prepare" >&2; exit 3; } + cat "$BUILD/.benchx-target.json" +} + +if [[ $operation == describe ]]; then describe; exit 0; fi + +HOW="" +step() { # step : record it in how_built, show it, run it (output to stderr) + local line="$*" + HOW+="$line"$'\n' + printf '\033[2m provider$ %s\033[0m\n' "$line" >&2 + "$@" >&2 +} + +if [[ ! -d $SRC/.git ]]; then + mkdir -p "$WORK" + cp -R "$HERE/../demo-suite" "$SRC" + git -C "$SRC" init -q + git -C "$SRC" remote add origin https://example.org/archery-local/demo-suite.git + git -C "$SRC" add -A + git -C "$SRC" commit -qm "demo-suite" +fi +mkdir -p "$WORK/deps/lib" # stand-in for the dependencies activate.sh points at + +step cmake -S "$SRC" -B "$BUILD" -DCMAKE_BUILD_TYPE=Release -DFETCHCONTENT_BASE_DIR="$WORK/_deps" ${extra[@]+"${extra[@]}"} +step cmake --build "$BUILD" -j + +BUILD=$BUILD SRC=$SRC HOW=$HOW CONFIGURATION=$configuration HERE=$HERE python3 - >"$BUILD/.benchx-target.json" <<'PY' +import json, os +print(json.dumps({ + "schema_version": "benchx/target-description/0.1.0", + "target": {"kind": "build", "path": os.environ["BUILD"], + "source": {"uri": "https://example.org/archery-local/demo-suite.git", "type": "git"}, + "source_dir": os.environ["SRC"]}, + "how_built": os.environ["HOW"].rstrip("\n"), + "activation": "source " + os.path.join(os.environ["HERE"], "activate.sh"), + "shell": "bash", + "provider": {"name": "provider.sh", "version": "1"}, +}, indent=1)) +PY +describe # the sidecar just written, which stays with the build directory diff --git a/examples/arrow/archery-local/run.sh b/examples/arrow/archery-local/run.sh new file mode 100644 index 0000000..6389898 --- /dev/null +++ b/examples/arrow/archery-local/run.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# End to end, in the shape of the Arrow local story ("Measure what a build +# option costs"): one commit built two ways, plain and hardened, compared with +# noise-aware verdicts, where the comparison describes itself. +# +# You play the user. The target provider (provider.sh, named in +# benchx.conf.json) is your own build script: it builds with CMake and +# tells benchx what it built, how, and what environment you need. benchx builds +# nothing, and runs none of that text: you activate the environment yourself +# (activate.sh) and benchx records the text beside the environment it inherited. +# +# Every command is printed before it runs, as you would type it. Run it in an +# activated environment that has benchx installed (from prototype/: +# `pip install -e .`, or `uv sync --extra test`). Needs cmake and a C++ +# compiler on PATH, and network access the first time, to fetch Google +# Benchmark unless installed. +# +# ROUNDS=5 alternation rounds (default 5) +# +# benchx.conf.json is this directory's project file, so the local store is kept +# inside it too (_work/store.parquet). Delete _work/ to start over. +set -euo pipefail +cd "$(dirname "$0")" +HERE=$PWD +ROUNDS=${ROUNDS:-5} +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +RUN="hardening-$STAMP" + +for tool in cmake c++ bx python; do + command -v "$tool" >/dev/null || { echo "run.sh: $tool not found on PATH (see README.md)" >&2; exit 1; } +done +bx --help >/dev/null 2>&1 || { echo "run.sh: benchx is not installed in the active environment (see README.md)" >&2; exit 1; } + +step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } +say() { printf '\033[36m$ %s\033[0m\n' "$*" >&2; } +bx() { say bx "$@"; command bx "$@"; } + +step "user: activate the environment (benchx will not do this for you)" +say source activate.sh +source activate.sh +echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" + +step "ask the provider about a target that is not built yet: describe builds nothing" +bx target describe @plain || echo "(exit $?: nothing to describe yet, as expected)" + +step "prepare both configurations: the provider builds (its build output is the dimmed lines)" +bx target prepare @plain > /dev/null +bx target prepare @hardened + +step "describe now works, and the description lives in the build directory too" +bx target describe @hardened | head -3 +say head -3 _work/build-hardened/.benchx-target.json +head -3 _work/build-hardened/.benchx-target.json +echo "(the same document: a build directory carries its own description)" + +step "compare: the workbench asks the provider to prepare each side (a no-op), then alternates them" +bx compare @plain @hardened --label build=plain,hardened --rounds "$ROUNDS" --run-key "$RUN" --out _work/results + +step "the comparison names what was built and how; before, it was two opaque directory paths" +echo "# the target of work order 0001 (round 1, the hardened side), as the runner received it:" +python -c ' +import json, sys +print(json.dumps(json.load(open(sys.argv[1]))["target"], indent=1))' _work/results/orders/order-0001.json + +step "and every result carries it, next to what the runner saw (declared, observed: nothing judges)" +say bx head --json -n 1 '| …' +command bx head --json -n 1 | python -c ' +import json, sys +doc = json.loads(sys.stdin.readline()) +info = doc["provenance"]["info"] +print("provenance.info.how_built :", info["how_built"].replace("\n", "\n" + " " * 29)) +print("provenance.info.activation :", info["activation"]) +print("provenance.info.shell :", info["shell"]) +print("provenance.info.target_provider:", info["target_provider"]) +system = doc["coordinates"]["environment"]["system"] +print("environment.system.process.environment:", system.get("process", {}).get("environment")) +print("coordinates.subject.configuration:", doc["coordinates"]["subject"].get("configuration"))' + +step "forget to activate: the declaration still says what you should have run, the observation says what you did" +say env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build --rounds 3 ... +env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build \ + --label build=plain,hardened --rounds 3 --run-key "$RUN-unactivated" --out _work/results-unactivated + +printf '\nartifacts: %s/_work (results, orders, build directories, the local store)\n' "$HERE" diff --git a/examples/arrow/demo-suite/CMakeLists.txt b/examples/arrow/demo-suite/CMakeLists.txt new file mode 100644 index 0000000..503ff46 --- /dev/null +++ b/examples/arrow/demo-suite/CMakeLists.txt @@ -0,0 +1,23 @@ +cmake_minimum_required(VERSION 3.20) +project(demo CXX) + +option(DEMO_HARDENED "Simulated hardening checks, the one option the environments demo changes" OFF) +set(CMAKE_CXX_STANDARD 17) + +find_package(benchmark QUIET) +if(NOT benchmark_FOUND) + include(FetchContent) + set(BENCHMARK_ENABLE_TESTING OFF CACHE BOOL "" FORCE) + set(BENCHMARK_ENABLE_GTEST_TESTS OFF CACHE BOOL "" FORCE) + FetchContent_Declare(benchmark + GIT_REPOSITORY https://github.com/google/benchmark.git + GIT_TAG v1.9.1 + GIT_SHALLOW TRUE) + FetchContent_MakeAvailable(benchmark) +endif() + +add_executable(demo-bench bench.cpp) +target_link_libraries(demo-bench benchmark::benchmark_main) +if(DEMO_HARDENED) + target_compile_definitions(demo-bench PRIVATE DEMO_HARDENED) +endif() diff --git a/examples/arrow/demo-suite/bench.cpp b/examples/arrow/demo-suite/bench.cpp new file mode 100644 index 0000000..01d5938 --- /dev/null +++ b/examples/arrow/demo-suite/bench.cpp @@ -0,0 +1,44 @@ +// Two benchmarks for the prototype demo: a quiet one and a noisy one. +// +// kWork is what the revisions demo changes between commits; DEMO_HARDENED is +// what the environments demo changes between two builds of one commit. Both +// add a few percent of work, so the expected verdicts are known: the quiet +// benchmark flags the change, the noisy one does not. +#include + +#include +#include + +constexpr int kWork = 100; +#ifdef DEMO_HARDENED +constexpr int kExtra = 3; // simulated cost of hardening checks, in percent +#else +constexpr int kExtra = 0; +#endif + +static double Sum(const std::vector& v, long rounds) { + double s = 0; + for (long r = 0; r < rounds; ++r) { + for (double x : v) s += x; + benchmark::DoNotOptimize(s); + } + return s; +} + +static void BM_Quiet(benchmark::State& state) { + std::vector v(state.range(0), 1.0); + const long rounds = kWork * (100 + kExtra); + for (auto _ : state) benchmark::DoNotOptimize(Sum(v, rounds)); +} +BENCHMARK(BM_Quiet)->Arg(1024); + +// Each repetition draws its own load level, so repetitions disagree by +// around ten percent, as they do for a benchmark at the mercy of its machine. +static void BM_Noisy(benchmark::State& state) { + std::vector v(state.range(0), 1.0); + std::random_device device; + std::uniform_real_distribution level(1.0, 1.25); + const long rounds = static_cast(kWork * (100 + kExtra) * level(device)); + for (auto _ : state) benchmark::DoNotOptimize(Sum(v, rounds)); +} +BENCHMARK(BM_Noisy)->Arg(1024); diff --git a/examples/arrow/demo/.gitignore b/examples/arrow/demo/.gitignore new file mode 100644 index 0000000..13eecef --- /dev/null +++ b/examples/arrow/demo/.gitignore @@ -0,0 +1,2 @@ +store.parquet +results/ diff --git a/examples/arrow/demo/benchx.conf.json b/examples/arrow/demo/benchx.conf.json new file mode 100644 index 0000000..fae92f0 --- /dev/null +++ b/examples/arrow/demo/benchx.conf.json @@ -0,0 +1,10 @@ +{ + "schema_version": "benchx/config/0.1.0", + "adapter": { + "name": "google-benchmark", + "suite": "demo-bench", + "precision": { "repetitions": 5 } + }, + "paths": { "store": "store.parquet", "results": "results" }, + "compare": { "profile": "revisions" } +} diff --git a/examples/arrow/demo/run.sh b/examples/arrow/demo/run.sh new file mode 100644 index 0000000..ed46383 --- /dev/null +++ b/examples/arrow/demo/run.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# End-to-end demonstration of prototype-design.md §6: both profiles, both shapes. +# +# The script plays the user, and in that role the target provider: it prepares +# the environment with its own tools, a git repository with two commits and +# three CMake build directories (benchx builds nothing). Then `bx compare` in +# measure mode does the workbench's part (system-decomposition.md §3.4): it +# alternates the sides over the rounds, assigns round and slot, delivers every +# result to the local store, and compares. +# +# Every command is printed before it runs, as you would type it: `bx` stands +# for `$PYTHON -m benchx.cli`, and $WORK for the scratch directory. +# +# Needs cmake, a C++ compiler, and network access the first time, to fetch +# Google Benchmark unless it is installed. ROUNDS=5 by default. +# +# `bx compare` (measure mode) and `bx run` deliver every result to the one local store, +# examples/demo/store.parquet (paths.store in examples/demo/benchx.conf.json, the demo's +# project file, from whose directory every `bx` command runs), so the store and +# each series' history grow from run to run. `bx head` shows the latest rows +# at any time. Delete the file to start over. +set -euo pipefail +cd "$(dirname "$0")" +PROTOTYPE=$PWD +PYTHON=${PYTHON:-$PROTOTYPE/.venv/bin/python} +ROUNDS=${ROUNDS:-5} +WORK=$(mktemp -d) +DEMO_ROOT=$PROTOTYPE/examples/demo +STORE=$DEMO_ROOT/store.parquet +STAMP=$(date -u +%Y%m%dT%H%M%SZ) # run keys are unique per demo run, so runs accumulate +RUN_REV="rev-adhoc-$STAMP" +RUN_ENV="env-tracked-$STAMP" +export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=demo@example.org +export GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=demo@example.org + +step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } + +# say : print a command as you would type it +quote() { + if [[ $1 =~ ^[A-Za-z0-9_./:=@%+,^-]+$ ]]; then printf '%s' "$1" + else printf "'%s'" "${1//\'/\'\\\'\'}"; fi +} +SAY_SUFFIX="" +say() { # to stderr, so a command's own redirected output never contains it + local line="" arg + for arg in "$@"; do line+=" $(quote "$arg")"; done + line="${line# }${SAY_SUFFIX:+ $SAY_SUFFIX}" + printf '\033[36m$ %s\033[0m\n' "${line//"$WORK"/\$WORK}" >&2 +} +show() { say "$@"; "$@"; } # print, then run +quiet() { say "$@"; "$@" >/dev/null; } # print, then run without its output +bx() { say bx "$@"; (cd "$DEMO_ROOT" && "$PYTHON" -m benchx.cli "$@"); } +into() { # into : run with stdout to a file, printed with its redirect + SAY_SUFFIX="> $1"; local out=$1; shift + "$@" > "$out" + SAY_SUFFIX="" +} + +printf '\033[36m$ WORK=%s\033[0m\n' "$WORK" >&2 + +step "user: a source with two commits, a worktree at each" +show cp -R examples/demo-suite "$WORK/src" +show git -C "$WORK/src" init -q +show git -C "$WORK/src" add -A +show git -C "$WORK/src" commit -qm "base" +BASE=$(git -C "$WORK/src" rev-parse HEAD) +show sed -i.bak 's/kWork = 100/kWork = 102/' "$WORK/src/bench.cpp" +rm "$WORK/src/bench.cpp.bak" +show git -C "$WORK/src" commit -qam "head: 2% more work" +HEAD=$(git -C "$WORK/src" rev-parse HEAD) +show git -C "$WORK/src" worktree add -q --detach "$WORK/wt-base" "$BASE" +show git -C "$WORK/src" worktree add -q --detach "$WORK/wt-head" "$HEAD" +echo "base $BASE" +echo "head $HEAD" + +step "user: three build directories (base, head, head hardened)" +build() { # build [cmake args] + quiet cmake -S "$1" -B "$2" -DCMAKE_BUILD_TYPE=Release -DFETCHCONTENT_BASE_DIR="$WORK/_deps" "${@:3}" + quiet cmake --build "$2" -j +} +build "$WORK/wt-base" "$WORK/build-base" +build "$WORK/wt-head" "$WORK/build-head" +build "$WORK/wt-head" "$WORK/build-hardened" -DDEMO_HARDENED=ON + +step "revisions, ad hoc: no project, measured and compared in one command" +echo "# measure mode: the sides alternate over $ROUNDS rounds, one work order each" +bx compare "$WORK/build-base:$WORK/wt-base" "$WORK/build-head:$WORK/wt-head" \ + --rounds "$ROUNDS" --run-key "$RUN_REV" --out "$WORK/results-rev" # adapter, suite and profile: benchx.conf.json +echo "# the first work order it wrote:" +show cat "$WORK/results-rev/orders/order-0000.json" +echo +echo "# read mode: the same run key, compared again through a throwaway store" +bx compare --run "$RUN_REV" --baseline "${BASE:0:10}" --results "$WORK/results-rev" +echo "# bx compare also delivered these to the local store, as thin results of project local/" + +step "environments, tracked: project demo, into the local store" +bx compare "$WORK/build-head:$WORK/wt-head" "$WORK/build-hardened:$WORK/wt-head" --profile environments \ + --label build=plain,hardened --rounds "$ROUNDS" --project demo \ + --run-key "$RUN_ENV" --out "$WORK/results-env" + +step "re-ingest is a visible no-op" +bx ingest "$WORK/results-env" + +step "a conflicting unit is rejected with its taxonomy code" +echo "# \$WORK/bad.json: a copy of one result, its unit changed from s to ms" +"$PYTHON" - "$WORK" <<'EOF' +import json, pathlib, sys +work = pathlib.Path(sys.argv[1]) +doc = json.loads(next(p for p in sorted((work / "results-env").glob("*_wall-time_*.json"))).read_text()) +doc["ingest_key"] += ":in-ms" +doc["measurement"]["quantity"]["unit"] = "ms" +(work / "bad.json").write_text(json.dumps(doc)) +EOF +bx ingest "$WORK/bad.json" || true + +step "compare from the local store" +bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain + +step "the same comparison through a throwaway store gives the same document" +into "$WORK/a.json" bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain --json +into "$WORK/b.json" bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain --json \ + --results "$WORK/results-env" +show cmp "$WORK/a.json" "$WORK/b.json" && echo "identical" + +step "series and history (a query; history mode is deferred)" +into "$WORK/series.txt" bx series --workload BM_Quiet --quantity wall-time +show head -6 "$WORK/series.txt" +SERIES=$(awk '/demo .*est=benchx\/median/ {print $1; exit}' "$WORK/series.txt") +into "$WORK/history.txt" bx history "$SERIES" +show head -1 "$WORK/history.txt" +show tail -n 4 "$WORK/history.txt" + +step "the store is one Parquet file, and keeps growing" +show ls -l "$STORE" +bx head -n 8 +printf '\ndemo artifacts in %s\n' "$WORK" From 0786e51fe9d5417f7318fbda157377785a95fa51 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 3 Nov 2026 18:29:10 +0000 Subject: [PATCH 2/2] Adapt the Arrow examples to this repository The files came from benchx's prototype/ directory and assumed its layout. - demo/run.sh: runs the active environment's Python (PYTHON overrides it) instead of prototype/.venv, takes its project root from its own directory, and copies the sibling ../demo-suite. The store stays in demo/store.parquet. - archery-local: links to the benchx design documents and schemas point to benchx-dev/benchx on GitHub instead of ../../../; install instructions use an editable install from a benchx checkout; the run.sh header says so too, and the README names demo/run.sh where it named demo.sh. - examples/arrow/README.md describes the three directories and how to run them, and the top-level README lists the new examples. Both run.sh scripts were run from here against benchx f49d87e, with ROUNDS=2 (cmake, make and a C++ compiler on the PATH). They are not executable in the source, so the READMEs use "bash". Co-Authored-By: Claude Sonnet 5.5 --- README.md | 2 + examples/arrow/README.md | 52 ++++++++++++++++++++++++++ examples/arrow/archery-local/README.md | 17 +++++---- examples/arrow/archery-local/run.sh | 4 +- examples/arrow/demo/run.sh | 17 +++++---- 5 files changed, 75 insertions(+), 17 deletions(-) create mode 100644 examples/arrow/README.md diff --git a/README.md b/README.md index 470ad0c..5c844c2 100644 --- a/README.md +++ b/README.md @@ -7,4 +7,6 @@ represent best practices. Currently, the following examples are provided: - `examples/numpy-spin`: out-of-tree benchmarks for NumPy, using `spin` to build NumPy itself; +- `examples/arrow`: Arrow-shaped examples with Google Benchmark and CMake: a walkthrough of both + comparison profiles (`demo`) and one of the target provider contract (`archery-local`). diff --git a/examples/arrow/README.md b/examples/arrow/README.md new file mode 100644 index 0000000..e49853b --- /dev/null +++ b/examples/arrow/README.md @@ -0,0 +1,52 @@ +# Arrow-shaped examples: Google Benchmark and CMake + +Two end-to-end examples in the shape of the Arrow local-benchmarking story, on one +small stand-in suite. They used to live in the benchx repository, as +`prototype/examples/{demo-suite,demo,archery-local}` and `prototype/demo.sh`. + +| Directory | What it is | +|---|---| +| `demo-suite/` | a two-benchmark Google Benchmark suite (`BM_Quiet` and `BM_Noisy`), built with CMake; `-DDEMO_HARDENED=ON` adds about 3% of work | +| `demo/` | `run.sh`, a walkthrough of both comparison profiles that prints every command before running it; the project file (`benchx.conf.json`) and the local store it keeps | +| `archery-local/` | a target provider walkthrough: one commit built plain and hardened, and compared; has its own README | + +## Run them + +You need benchx installed in editable mode from a checkout (it reads its schemas from +there), `cmake`, a C++ compiler, `git` and `python3`, and network access the first time +(CMake fetches Google Benchmark unless it is installed). From the root of this +repository: + +```console +$ git clone https://github.com/benchx-dev/benchx.git +$ python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11 +$ pip install -e benchx/prototype +$ bash examples/arrow/demo/run.sh +$ bash examples/arrow/archery-local/run.sh +``` + +`ROUNDS=2` shortens either one; the default is 5. + +## What `demo/run.sh` shows + +It prepares a git source with two commits and three CMake build directories of +`demo-suite` (base, head, and head with `DEMO_HARDENED=ON`), shows the first work +order of each part, then: + +1. **revisions, ad hoc:** `bx compare` in measure mode alternates base and head over the + rounds with orders that name no project, then read mode compares the same run key + through a throwaway store; +2. **environments, tracked:** `bx compare` alternates plain and hardened, labeled + `build: plain|hardened`, compares from the local store, re-ingests (a no-op), and + rejects a unit conflict; +3. checks that the throwaway store gives the same comparison document as the persistent + one, lists series and history, and ends with `bx head`. + +Every measure-mode `bx compare` delivers to the local store, `demo/store.parquet`, so +each run adds its results to it. Run keys carry a timestamp, so runs accumulate and each +series' history grows. Look at it at any time, or start over: + +```console +$ (cd examples/arrow/demo && python -m benchx.cli head -n 20) +$ rm examples/arrow/demo/store.parquet +``` diff --git a/examples/arrow/archery-local/README.md b/examples/arrow/archery-local/README.md index dfbbe50..89ffcf9 100644 --- a/examples/arrow/archery-local/README.md +++ b/examples/arrow/archery-local/README.md @@ -3,12 +3,12 @@ One commit, built two ways (plain and hardened), compared with noise-aware verdicts, where the comparison says what was built and how. It follows the Arrow local story, "Measure what a build option costs" -([`docs/user_stories/apache-arrow-local-benchmarking.md`](../../../docs/user_stories/apache-arrow-local-benchmarking.md)), +([`docs/user_stories/apache-arrow-local-benchmarking.md`](https://github.com/benchx-dev/benchx/blob/main/docs/user_stories/apache-arrow-local-benchmarking.md) in benchx), on a small stand-in suite (`../demo-suite`: a quiet and a noisy Google Benchmark case, where hardening adds about 3% of work). It exercises the target-provider contract of -[`system-decomposition.md`](../../../docs/design/system-decomposition.md) §3.4: +[`system-decomposition.md`](https://github.com/benchx-dev/benchx/blob/main/docs/design/system-decomposition.md) §3.4: - **You are the user, and the provider is your build script.** `provider.sh` configures and builds with CMake, as `archery` or a contributor would. benchx @@ -27,13 +27,14 @@ It exercises the target-provider contract of You need `cmake`, a C++ compiler, `git` and `python3` on your `PATH`, and network access the first time (CMake fetches Google Benchmark v1.9.1 unless it is -installed). From the repository root: +installed). Install benchx in editable mode from a checkout, which is where it reads +its schemas from, then run the script from the root of this repository: ```console -$ cd prototype +$ git clone https://github.com/benchx-dev/benchx.git $ python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11, or any environment you like -$ pip install -e . -$ examples/archery-local/run.sh +$ pip install -e benchx/prototype +$ bash examples/arrow/archery-local/run.sh ``` It takes about two minutes the first time (two CMake builds and a measured @@ -129,13 +130,13 @@ runner simply omits it from the results. | `activate.sh` | the environment you activate before calling `bx` | | `_work/` | everything generated: source copy, build directories, results, store (git-ignored) | -The documents involved are in `../../../schemas/`: `target-provider-request`, +The documents involved are in [`schemas/`](https://github.com/benchx-dev/benchx/tree/main/schemas) of benchx: `target-provider-request`, `target-description`, and `work-order` (its `target` takes the declared text). ## What this does not show The provider here builds one source tree, `WORKSPACE`. Comparing against a tag or a past revision (`@plain=v1.0`) is the provider's business and is not -implemented in this example; the prototype's own `demo.sh` shows revisions +implemented in this example; the [`demo`](../demo/run.sh) example shows revisions with two prebuilt worktrees. A declared `build` object in a description is refused by the prototype's runner (it reads `CMakeCache.txt` instead). diff --git a/examples/arrow/archery-local/run.sh b/examples/arrow/archery-local/run.sh index 6389898..1873e28 100644 --- a/examples/arrow/archery-local/run.sh +++ b/examples/arrow/archery-local/run.sh @@ -10,8 +10,8 @@ # (activate.sh) and benchx records the text beside the environment it inherited. # # Every command is printed before it runs, as you would type it. Run it in an -# activated environment that has benchx installed (from prototype/: -# `pip install -e .`, or `uv sync --extra test`). Needs cmake and a C++ +# activated environment that has benchx installed, editable, from a benchx +# checkout (`pip install -e /prototype`). Needs cmake and a C++ # compiler on PATH, and network access the first time, to fetch Google # Benchmark unless installed. # diff --git a/examples/arrow/demo/run.sh b/examples/arrow/demo/run.sh index ed46383..689c652 100644 --- a/examples/arrow/demo/run.sh +++ b/examples/arrow/demo/run.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash # End-to-end demonstration of prototype-design.md §6: both profiles, both shapes. +# (The design documents named here are in docs/design/ of benchx-dev/benchx.) # # The script plays the user, and in that role the target provider: it prepares # the environment with its own tools, a git repository with two commits and @@ -11,21 +12,23 @@ # Every command is printed before it runs, as you would type it: `bx` stands # for `$PYTHON -m benchx.cli`, and $WORK for the scratch directory. # -# Needs cmake, a C++ compiler, and network access the first time, to fetch -# Google Benchmark unless it is installed. ROUNDS=5 by default. +# Run it with benchx installed, editable, from a benchx checkout +# (`pip install -e /prototype`) in the active environment: PYTHON is that +# environment's interpreter (default: python). Needs cmake, a C++ compiler, and +# network access the first time, to fetch Google Benchmark unless it is installed. +# ROUNDS=5 by default. # # `bx compare` (measure mode) and `bx run` deliver every result to the one local store, -# examples/demo/store.parquet (paths.store in examples/demo/benchx.conf.json, the demo's +# store.parquet in this directory (paths.store in the benchx.conf.json here, the demo's # project file, from whose directory every `bx` command runs), so the store and # each series' history grow from run to run. `bx head` shows the latest rows # at any time. Delete the file to start over. set -euo pipefail cd "$(dirname "$0")" -PROTOTYPE=$PWD -PYTHON=${PYTHON:-$PROTOTYPE/.venv/bin/python} +DEMO_ROOT=$PWD +PYTHON=${PYTHON:-python} ROUNDS=${ROUNDS:-5} WORK=$(mktemp -d) -DEMO_ROOT=$PROTOTYPE/examples/demo STORE=$DEMO_ROOT/store.parquet STAMP=$(date -u +%Y%m%dT%H%M%SZ) # run keys are unique per demo run, so runs accumulate RUN_REV="rev-adhoc-$STAMP" @@ -59,7 +62,7 @@ into() { # into : run with stdout to a file, printed with its re printf '\033[36m$ WORK=%s\033[0m\n' "$WORK" >&2 step "user: a source with two commits, a worktree at each" -show cp -R examples/demo-suite "$WORK/src" +show cp -R ../demo-suite "$WORK/src" show git -C "$WORK/src" init -q show git -C "$WORK/src" add -A show git -C "$WORK/src" commit -qm "base"