diff --git a/README.md b/README.md index 470ad0c..5c844c2 100644 --- a/README.md +++ b/README.md @@ -7,4 +7,6 @@ represent best practices. Currently, the following examples are provided: - `examples/numpy-spin`: out-of-tree benchmarks for NumPy, using `spin` to build NumPy itself; +- `examples/arrow`: Arrow-shaped examples with Google Benchmark and CMake: a walkthrough of both + comparison profiles (`demo`) and one of the target provider contract (`archery-local`). diff --git a/examples/arrow/README.md b/examples/arrow/README.md new file mode 100644 index 0000000..e49853b --- /dev/null +++ b/examples/arrow/README.md @@ -0,0 +1,52 @@ +# Arrow-shaped examples: Google Benchmark and CMake + +Two end-to-end examples in the shape of the Arrow local-benchmarking story, on one +small stand-in suite. They used to live in the benchx repository, as +`prototype/examples/{demo-suite,demo,archery-local}` and `prototype/demo.sh`. + +| Directory | What it is | +|---|---| +| `demo-suite/` | a two-benchmark Google Benchmark suite (`BM_Quiet` and `BM_Noisy`), built with CMake; `-DDEMO_HARDENED=ON` adds about 3% of work | +| `demo/` | `run.sh`, a walkthrough of both comparison profiles that prints every command before running it; the project file (`benchx.conf.json`) and the local store it keeps | +| `archery-local/` | a target provider walkthrough: one commit built plain and hardened, and compared; has its own README | + +## Run them + +You need benchx installed in editable mode from a checkout (it reads its schemas from +there), `cmake`, a C++ compiler, `git` and `python3`, and network access the first time +(CMake fetches Google Benchmark unless it is installed). From the root of this +repository: + +```console +$ git clone https://github.com/benchx-dev/benchx.git +$ python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11 +$ pip install -e benchx/prototype +$ bash examples/arrow/demo/run.sh +$ bash examples/arrow/archery-local/run.sh +``` + +`ROUNDS=2` shortens either one; the default is 5. + +## What `demo/run.sh` shows + +It prepares a git source with two commits and three CMake build directories of +`demo-suite` (base, head, and head with `DEMO_HARDENED=ON`), shows the first work +order of each part, then: + +1. **revisions, ad hoc:** `bx compare` in measure mode alternates base and head over the + rounds with orders that name no project, then read mode compares the same run key + through a throwaway store; +2. **environments, tracked:** `bx compare` alternates plain and hardened, labeled + `build: plain|hardened`, compares from the local store, re-ingests (a no-op), and + rejects a unit conflict; +3. checks that the throwaway store gives the same comparison document as the persistent + one, lists series and history, and ends with `bx head`. + +Every measure-mode `bx compare` delivers to the local store, `demo/store.parquet`, so +each run adds its results to it. Run keys carry a timestamp, so runs accumulate and each +series' history grows. Look at it at any time, or start over: + +```console +$ (cd examples/arrow/demo && python -m benchx.cli head -n 20) +$ rm examples/arrow/demo/store.parquet +``` diff --git a/examples/arrow/archery-local/.gitignore b/examples/arrow/archery-local/.gitignore new file mode 100644 index 0000000..6acf7e7 --- /dev/null +++ b/examples/arrow/archery-local/.gitignore @@ -0,0 +1,2 @@ +_work/ +results/ diff --git a/examples/arrow/archery-local/README.md b/examples/arrow/archery-local/README.md new file mode 100644 index 0000000..89ffcf9 --- /dev/null +++ b/examples/arrow/archery-local/README.md @@ -0,0 +1,142 @@ +# archery-local: an end-to-end example + +One commit, built two ways (plain and hardened), compared with noise-aware +verdicts, where the comparison says what was built and how. It follows the +Arrow local story, "Measure what a build option costs" +([`docs/user_stories/apache-arrow-local-benchmarking.md`](https://github.com/benchx-dev/benchx/blob/main/docs/user_stories/apache-arrow-local-benchmarking.md) in benchx), +on a small stand-in suite (`../demo-suite`: a quiet and a noisy Google +Benchmark case, where hardening adds about 3% of work). + +It exercises the target-provider contract of +[`system-decomposition.md`](https://github.com/benchx-dev/benchx/blob/main/docs/design/system-decomposition.md) §3.4: + +- **You are the user, and the provider is your build script.** `provider.sh` + configures and builds with CMake, as `archery` or a contributor would. benchx + builds nothing. +- **The provider tells benchx what it built** in a *target description*: the + build directory and its source, plus two pieces of text you own, `how_built` + (the exact commands) and `activation` (what to run to enter the environment, + with the `shell` it is written for). Either may be empty. +- **You activate the environment yourself** (`source activate.sh`). benchx never + runs `activation`. It records the text, as given, in the work order and in + every result's `provenance.info`, beside the allowlisted environment variables + the runner actually inherited. If the two disagree, both stay and nothing + judges. + +## Run it + +You need `cmake`, a C++ compiler, `git` and `python3` on your `PATH`, and network +access the first time (CMake fetches Google Benchmark v1.9.1 unless it is +installed). Install benchx in editable mode from a checkout, which is where it reads +its schemas from, then run the script from the root of this repository: + +```console +$ git clone https://github.com/benchx-dev/benchx.git +$ python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11, or any environment you like +$ pip install -e benchx/prototype +$ bash examples/arrow/archery-local/run.sh +``` + +It takes about two minutes the first time (two CMake builds and a measured +run) and prints every command before running it, as you would type it. The +script runs `bx` and `python` from the active environment. More rounds: +`ROUNDS=10 ./run.sh`. + +Everything it creates is under `_work/` in this directory, including the local +store (`_work/store.parquet`, set by `paths.store` in `benchx.conf.json`), so nothing +outside this directory changes. `rm -rf _work` starts over. + +## What you will see + +1. **Activate.** `source activate.sh` puts a stand-in dependencies directory on + `LD_LIBRARY_PATH`. This is you, not benchx. +2. **`bx target describe @plain`** fails with exit 3 from the provider: nothing is + built yet. `describe` builds nothing, by contract. +3. **`bx target prepare @plain` / `@hardened`** make the provider build. Its CMake + output is the build's own, on stderr; the target description it prints on + stdout is the answer. `how_built` is the very commands the provider ran. +4. **`bx target describe @hardened`** now works. The same document is stored in + the build directory as `.benchx-target.json`, so a long-lived build directory + carries its own description (point `bx compare` at a bare directory and it + reads that file; no provider needed). +5. **`bx compare @plain @hardened ...`** is the workbench's session loop: it + asks the provider to prepare each side (a no-op build now), then alternates + the sides over the rounds, one work order each, delivers the results to the + local store, and compares: + + ``` + run hardening-... profile environments baseline plain contender hardened + plain: how built: cmake -S .../_work/src -B .../_work/build-plain ... (+1 lines) + plain: activation: source .../activate.sh + plain: observed env: LD_LIBRARY_PATH=.../_work/deps/lib + hardened: how built: cmake -S ... -B .../_work/build-hardened ... -DDEMO_HARDENED=ON (+1 lines) + ... + REGRESSED (1) + BM_Quiet/1024 wall-time effect +2.42% noise 0.57% rounds=5 + NO CHANGE DETECTED (1) + BM_Noisy/1024 wall-time effect +3.43% noise 3.04% rounds=5 + ``` + + Compare that with the story's pains: the comparison now describes itself + (what was built, how, in what environment), and the verdict uses each + benchmark's own noise instead of a flat 5%: the quiet case flags a 2.4% + change that is four times its noise, the noisy case does not flag a 3.4% + swing inside its noise. The numbers vary from run to run; with few rounds the + noisy case may flag too. +6. **The work order and a result**, printed: the order's `target` carries + `how_built`, `activation`, `shell` and `provider`; the result's + `provenance.info` carries them, next to `coordinates.environment.system.process.environment` and + `coordinates.subject.configuration` (read by the runner from + `CMakeCache.txt`). +7. **Forget to activate.** The same comparison with `LD_LIBRARY_PATH` removed + from the environment: the declaration still says `source .../activate.sh`, + the observed environment no longer lists `LD_LIBRARY_PATH`, and nothing + refuses: declarations are recorded as given (`benchmark-environments.md` + §3.2), never checked. + +## Try it by hand + +After `./run.sh` (the build directories and the store stay in `_work/`). From +this directory, with the environment still active: + +```console +$ source activate.sh + +$ bx target describe @hardened # what exists, nothing built +$ bx target prepare @hardened # incremental build, then the description +$ bx target describe _work/build-hardened # a bare directory: its own sidecar + +# the story's own knobs: scope by filter, precision by repetitions +$ bx compare @plain @hardened --label build=plain,hardened --filter BM_Quiet --rounds 10 \ + --repetitions 5 --run-key mine # the adapter, suite, project and profile come from benchx.conf.json + +$ bx compare --run mine --label build --baseline plain # read mode +$ bx head -n 4 # the latest stored results +$ bx head --json -n 1 | python3 -m json.tool | less # provenance.info, environment.system +``` + +Edit `provider.sh`'s `hardened)` case or `activate.sh` and run again to see how +each change shows up in the next comparison. Empty text is fine: set +`"activation": ""` (or `null`, or leave it out) in a description and the +runner simply omits it from the results. + +## What is in this directory + +| File | Role | +|---|---| +| `run.sh` | the end-to-end script | +| `provider.sh` | the target provider: your build script, speaking the `describe`/`prepare` contract | +| `benchx.conf.json` | the project file: the provider command, the harness and its suite, the project name, the comparison profile, and where the store and results go | +| `activate.sh` | the environment you activate before calling `bx` | +| `_work/` | everything generated: source copy, build directories, results, store (git-ignored) | + +The documents involved are in [`schemas/`](https://github.com/benchx-dev/benchx/tree/main/schemas) of benchx: `target-provider-request`, +`target-description`, and `work-order` (its `target` takes the declared text). + +## What this does not show + +The provider here builds one source tree, `WORKSPACE`. Comparing against a tag +or a past revision (`@plain=v1.0`) is the provider's business and is not +implemented in this example; the [`demo`](../demo/run.sh) example shows revisions +with two prebuilt worktrees. A declared `build` object in a description is +refused by the prototype's runner (it reads `CMakeCache.txt` instead). diff --git a/examples/arrow/archery-local/activate.sh b/examples/arrow/archery-local/activate.sh new file mode 100644 index 0000000..cd10b8b --- /dev/null +++ b/examples/arrow/archery-local/activate.sh @@ -0,0 +1,8 @@ +# The environment this example's benchmarks are meant to run in. In the Arrow +# story this is where you would load a toolchain or point at the dependencies +# you built; here it only puts a stand-in "deps" directory on the library +# search path. You source it yourself, before invoking bx: benchx never runs +# it, it records the text that says you did (the target description's +# `activation`) next to the environment it actually inherited. +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +export LD_LIBRARY_PATH="$HERE/_work/deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" diff --git a/examples/arrow/archery-local/benchx.conf.json b/examples/arrow/archery-local/benchx.conf.json new file mode 100644 index 0000000..9029306 --- /dev/null +++ b/examples/arrow/archery-local/benchx.conf.json @@ -0,0 +1,13 @@ +{ + "schema_version": "benchx/config/0.1.0", + "project": "archery-local", + "provider": { "command": ["./provider.sh"] }, + "adapter": { + "name": "google-benchmark", + "suite": "demo-bench", + "quantities": ["wall-time"], + "precision": { "repetitions": 5 } + }, + "paths": { "store": "_work/store.parquet", "results": "_work/results" }, + "compare": { "profile": "environments" } +} diff --git a/examples/arrow/archery-local/provider.sh b/examples/arrow/archery-local/provider.sh new file mode 100755 index 0000000..a5226e0 --- /dev/null +++ b/examples/arrow/archery-local/provider.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# The target provider of this example (system-decomposition.md §3.4). +# +# It is the user's own build script, speaking the provider contract: one JSON +# request on stdin (schemas/target-provider-request/0.1.0), one target +# description on stdout (schemas/target-description/0.1.0). Everything else, +# above all the build's own output, goes to stderr. +# +# describe reports a target that exists and builds nothing; exit 3 if none +# prepare configures and builds it, with CMake, as Arrow's contributors do +# (an incremental no-op when it is already built), then reports it +# +# Configurations: `plain`, and `hardened` (-DDEMO_HARDENED=ON), the same +# commit built two ways, like the -DARROW_HARDENING=ON story. +# +# `how_built` is assembled from the very commands this script runs, so the +# text cannot drift from what happened. benchx never runs any of it. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +WORK=${BENCHX_EXAMPLE_WORK:-$HERE/_work} +export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=demo@example.org +export GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=demo@example.org + +request=$(cat) +field() { python3 -c 'import json,sys; print(json.load(sys.stdin).get(sys.argv[1]) or "")' "$1" <<<"$request"; } +operation=$(field operation) +configuration=$(field configuration) +case "$configuration" in + plain) extra=() ;; + hardened) extra=(-DDEMO_HARDENED=ON) ;; + *) echo "provider: unknown configuration '$configuration' (plain or hardened)" >&2; exit 2 ;; +esac +SRC=$WORK/src +BUILD=$WORK/build-$configuration + +describe() { + [[ -f $BUILD/.benchx-target.json ]] || { echo "provider: $configuration is not built; use prepare" >&2; exit 3; } + cat "$BUILD/.benchx-target.json" +} + +if [[ $operation == describe ]]; then describe; exit 0; fi + +HOW="" +step() { # step : record it in how_built, show it, run it (output to stderr) + local line="$*" + HOW+="$line"$'\n' + printf '\033[2m provider$ %s\033[0m\n' "$line" >&2 + "$@" >&2 +} + +if [[ ! -d $SRC/.git ]]; then + mkdir -p "$WORK" + cp -R "$HERE/../demo-suite" "$SRC" + git -C "$SRC" init -q + git -C "$SRC" remote add origin https://example.org/archery-local/demo-suite.git + git -C "$SRC" add -A + git -C "$SRC" commit -qm "demo-suite" +fi +mkdir -p "$WORK/deps/lib" # stand-in for the dependencies activate.sh points at + +step cmake -S "$SRC" -B "$BUILD" -DCMAKE_BUILD_TYPE=Release -DFETCHCONTENT_BASE_DIR="$WORK/_deps" ${extra[@]+"${extra[@]}"} +step cmake --build "$BUILD" -j + +BUILD=$BUILD SRC=$SRC HOW=$HOW CONFIGURATION=$configuration HERE=$HERE python3 - >"$BUILD/.benchx-target.json" <<'PY' +import json, os +print(json.dumps({ + "schema_version": "benchx/target-description/0.1.0", + "target": {"kind": "build", "path": os.environ["BUILD"], + "source": {"uri": "https://example.org/archery-local/demo-suite.git", "type": "git"}, + "source_dir": os.environ["SRC"]}, + "how_built": os.environ["HOW"].rstrip("\n"), + "activation": "source " + os.path.join(os.environ["HERE"], "activate.sh"), + "shell": "bash", + "provider": {"name": "provider.sh", "version": "1"}, +}, indent=1)) +PY +describe # the sidecar just written, which stays with the build directory diff --git a/examples/arrow/archery-local/run.sh b/examples/arrow/archery-local/run.sh new file mode 100644 index 0000000..1873e28 --- /dev/null +++ b/examples/arrow/archery-local/run.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# End to end, in the shape of the Arrow local story ("Measure what a build +# option costs"): one commit built two ways, plain and hardened, compared with +# noise-aware verdicts, where the comparison describes itself. +# +# You play the user. The target provider (provider.sh, named in +# benchx.conf.json) is your own build script: it builds with CMake and +# tells benchx what it built, how, and what environment you need. benchx builds +# nothing, and runs none of that text: you activate the environment yourself +# (activate.sh) and benchx records the text beside the environment it inherited. +# +# Every command is printed before it runs, as you would type it. Run it in an +# activated environment that has benchx installed, editable, from a benchx +# checkout (`pip install -e /prototype`). Needs cmake and a C++ +# compiler on PATH, and network access the first time, to fetch Google +# Benchmark unless installed. +# +# ROUNDS=5 alternation rounds (default 5) +# +# benchx.conf.json is this directory's project file, so the local store is kept +# inside it too (_work/store.parquet). Delete _work/ to start over. +set -euo pipefail +cd "$(dirname "$0")" +HERE=$PWD +ROUNDS=${ROUNDS:-5} +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +RUN="hardening-$STAMP" + +for tool in cmake c++ bx python; do + command -v "$tool" >/dev/null || { echo "run.sh: $tool not found on PATH (see README.md)" >&2; exit 1; } +done +bx --help >/dev/null 2>&1 || { echo "run.sh: benchx is not installed in the active environment (see README.md)" >&2; exit 1; } + +step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } +say() { printf '\033[36m$ %s\033[0m\n' "$*" >&2; } +bx() { say bx "$@"; command bx "$@"; } + +step "user: activate the environment (benchx will not do this for you)" +say source activate.sh +source activate.sh +echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" + +step "ask the provider about a target that is not built yet: describe builds nothing" +bx target describe @plain || echo "(exit $?: nothing to describe yet, as expected)" + +step "prepare both configurations: the provider builds (its build output is the dimmed lines)" +bx target prepare @plain > /dev/null +bx target prepare @hardened + +step "describe now works, and the description lives in the build directory too" +bx target describe @hardened | head -3 +say head -3 _work/build-hardened/.benchx-target.json +head -3 _work/build-hardened/.benchx-target.json +echo "(the same document: a build directory carries its own description)" + +step "compare: the workbench asks the provider to prepare each side (a no-op), then alternates them" +bx compare @plain @hardened --label build=plain,hardened --rounds "$ROUNDS" --run-key "$RUN" --out _work/results + +step "the comparison names what was built and how; before, it was two opaque directory paths" +echo "# the target of work order 0001 (round 1, the hardened side), as the runner received it:" +python -c ' +import json, sys +print(json.dumps(json.load(open(sys.argv[1]))["target"], indent=1))' _work/results/orders/order-0001.json + +step "and every result carries it, next to what the runner saw (declared, observed: nothing judges)" +say bx head --json -n 1 '| …' +command bx head --json -n 1 | python -c ' +import json, sys +doc = json.loads(sys.stdin.readline()) +info = doc["provenance"]["info"] +print("provenance.info.how_built :", info["how_built"].replace("\n", "\n" + " " * 29)) +print("provenance.info.activation :", info["activation"]) +print("provenance.info.shell :", info["shell"]) +print("provenance.info.target_provider:", info["target_provider"]) +system = doc["coordinates"]["environment"]["system"] +print("environment.system.process.environment:", system.get("process", {}).get("environment")) +print("coordinates.subject.configuration:", doc["coordinates"]["subject"].get("configuration"))' + +step "forget to activate: the declaration still says what you should have run, the observation says what you did" +say env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build --rounds 3 ... +env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build \ + --label build=plain,hardened --rounds 3 --run-key "$RUN-unactivated" --out _work/results-unactivated + +printf '\nartifacts: %s/_work (results, orders, build directories, the local store)\n' "$HERE" diff --git a/examples/arrow/demo-suite/CMakeLists.txt b/examples/arrow/demo-suite/CMakeLists.txt new file mode 100644 index 0000000..503ff46 --- /dev/null +++ b/examples/arrow/demo-suite/CMakeLists.txt @@ -0,0 +1,23 @@ +cmake_minimum_required(VERSION 3.20) +project(demo CXX) + +option(DEMO_HARDENED "Simulated hardening checks, the one option the environments demo changes" OFF) +set(CMAKE_CXX_STANDARD 17) + +find_package(benchmark QUIET) +if(NOT benchmark_FOUND) + include(FetchContent) + set(BENCHMARK_ENABLE_TESTING OFF CACHE BOOL "" FORCE) + set(BENCHMARK_ENABLE_GTEST_TESTS OFF CACHE BOOL "" FORCE) + FetchContent_Declare(benchmark + GIT_REPOSITORY https://github.com/google/benchmark.git + GIT_TAG v1.9.1 + GIT_SHALLOW TRUE) + FetchContent_MakeAvailable(benchmark) +endif() + +add_executable(demo-bench bench.cpp) +target_link_libraries(demo-bench benchmark::benchmark_main) +if(DEMO_HARDENED) + target_compile_definitions(demo-bench PRIVATE DEMO_HARDENED) +endif() diff --git a/examples/arrow/demo-suite/bench.cpp b/examples/arrow/demo-suite/bench.cpp new file mode 100644 index 0000000..01d5938 --- /dev/null +++ b/examples/arrow/demo-suite/bench.cpp @@ -0,0 +1,44 @@ +// Two benchmarks for the prototype demo: a quiet one and a noisy one. +// +// kWork is what the revisions demo changes between commits; DEMO_HARDENED is +// what the environments demo changes between two builds of one commit. Both +// add a few percent of work, so the expected verdicts are known: the quiet +// benchmark flags the change, the noisy one does not. +#include + +#include +#include + +constexpr int kWork = 100; +#ifdef DEMO_HARDENED +constexpr int kExtra = 3; // simulated cost of hardening checks, in percent +#else +constexpr int kExtra = 0; +#endif + +static double Sum(const std::vector& v, long rounds) { + double s = 0; + for (long r = 0; r < rounds; ++r) { + for (double x : v) s += x; + benchmark::DoNotOptimize(s); + } + return s; +} + +static void BM_Quiet(benchmark::State& state) { + std::vector v(state.range(0), 1.0); + const long rounds = kWork * (100 + kExtra); + for (auto _ : state) benchmark::DoNotOptimize(Sum(v, rounds)); +} +BENCHMARK(BM_Quiet)->Arg(1024); + +// Each repetition draws its own load level, so repetitions disagree by +// around ten percent, as they do for a benchmark at the mercy of its machine. +static void BM_Noisy(benchmark::State& state) { + std::vector v(state.range(0), 1.0); + std::random_device device; + std::uniform_real_distribution level(1.0, 1.25); + const long rounds = static_cast(kWork * (100 + kExtra) * level(device)); + for (auto _ : state) benchmark::DoNotOptimize(Sum(v, rounds)); +} +BENCHMARK(BM_Noisy)->Arg(1024); diff --git a/examples/arrow/demo/.gitignore b/examples/arrow/demo/.gitignore new file mode 100644 index 0000000..13eecef --- /dev/null +++ b/examples/arrow/demo/.gitignore @@ -0,0 +1,2 @@ +store.parquet +results/ diff --git a/examples/arrow/demo/benchx.conf.json b/examples/arrow/demo/benchx.conf.json new file mode 100644 index 0000000..fae92f0 --- /dev/null +++ b/examples/arrow/demo/benchx.conf.json @@ -0,0 +1,10 @@ +{ + "schema_version": "benchx/config/0.1.0", + "adapter": { + "name": "google-benchmark", + "suite": "demo-bench", + "precision": { "repetitions": 5 } + }, + "paths": { "store": "store.parquet", "results": "results" }, + "compare": { "profile": "revisions" } +} diff --git a/examples/arrow/demo/run.sh b/examples/arrow/demo/run.sh new file mode 100644 index 0000000..689c652 --- /dev/null +++ b/examples/arrow/demo/run.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# End-to-end demonstration of prototype-design.md §6: both profiles, both shapes. +# (The design documents named here are in docs/design/ of benchx-dev/benchx.) +# +# The script plays the user, and in that role the target provider: it prepares +# the environment with its own tools, a git repository with two commits and +# three CMake build directories (benchx builds nothing). Then `bx compare` in +# measure mode does the workbench's part (system-decomposition.md §3.4): it +# alternates the sides over the rounds, assigns round and slot, delivers every +# result to the local store, and compares. +# +# Every command is printed before it runs, as you would type it: `bx` stands +# for `$PYTHON -m benchx.cli`, and $WORK for the scratch directory. +# +# Run it with benchx installed, editable, from a benchx checkout +# (`pip install -e /prototype`) in the active environment: PYTHON is that +# environment's interpreter (default: python). Needs cmake, a C++ compiler, and +# network access the first time, to fetch Google Benchmark unless it is installed. +# ROUNDS=5 by default. +# +# `bx compare` (measure mode) and `bx run` deliver every result to the one local store, +# store.parquet in this directory (paths.store in the benchx.conf.json here, the demo's +# project file, from whose directory every `bx` command runs), so the store and +# each series' history grow from run to run. `bx head` shows the latest rows +# at any time. Delete the file to start over. +set -euo pipefail +cd "$(dirname "$0")" +DEMO_ROOT=$PWD +PYTHON=${PYTHON:-python} +ROUNDS=${ROUNDS:-5} +WORK=$(mktemp -d) +STORE=$DEMO_ROOT/store.parquet +STAMP=$(date -u +%Y%m%dT%H%M%SZ) # run keys are unique per demo run, so runs accumulate +RUN_REV="rev-adhoc-$STAMP" +RUN_ENV="env-tracked-$STAMP" +export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=demo@example.org +export GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=demo@example.org + +step() { printf '\n\033[1m== %s\033[0m\n' "$*"; } + +# say : print a command as you would type it +quote() { + if [[ $1 =~ ^[A-Za-z0-9_./:=@%+,^-]+$ ]]; then printf '%s' "$1" + else printf "'%s'" "${1//\'/\'\\\'\'}"; fi +} +SAY_SUFFIX="" +say() { # to stderr, so a command's own redirected output never contains it + local line="" arg + for arg in "$@"; do line+=" $(quote "$arg")"; done + line="${line# }${SAY_SUFFIX:+ $SAY_SUFFIX}" + printf '\033[36m$ %s\033[0m\n' "${line//"$WORK"/\$WORK}" >&2 +} +show() { say "$@"; "$@"; } # print, then run +quiet() { say "$@"; "$@" >/dev/null; } # print, then run without its output +bx() { say bx "$@"; (cd "$DEMO_ROOT" && "$PYTHON" -m benchx.cli "$@"); } +into() { # into : run with stdout to a file, printed with its redirect + SAY_SUFFIX="> $1"; local out=$1; shift + "$@" > "$out" + SAY_SUFFIX="" +} + +printf '\033[36m$ WORK=%s\033[0m\n' "$WORK" >&2 + +step "user: a source with two commits, a worktree at each" +show cp -R ../demo-suite "$WORK/src" +show git -C "$WORK/src" init -q +show git -C "$WORK/src" add -A +show git -C "$WORK/src" commit -qm "base" +BASE=$(git -C "$WORK/src" rev-parse HEAD) +show sed -i.bak 's/kWork = 100/kWork = 102/' "$WORK/src/bench.cpp" +rm "$WORK/src/bench.cpp.bak" +show git -C "$WORK/src" commit -qam "head: 2% more work" +HEAD=$(git -C "$WORK/src" rev-parse HEAD) +show git -C "$WORK/src" worktree add -q --detach "$WORK/wt-base" "$BASE" +show git -C "$WORK/src" worktree add -q --detach "$WORK/wt-head" "$HEAD" +echo "base $BASE" +echo "head $HEAD" + +step "user: three build directories (base, head, head hardened)" +build() { # build [cmake args] + quiet cmake -S "$1" -B "$2" -DCMAKE_BUILD_TYPE=Release -DFETCHCONTENT_BASE_DIR="$WORK/_deps" "${@:3}" + quiet cmake --build "$2" -j +} +build "$WORK/wt-base" "$WORK/build-base" +build "$WORK/wt-head" "$WORK/build-head" +build "$WORK/wt-head" "$WORK/build-hardened" -DDEMO_HARDENED=ON + +step "revisions, ad hoc: no project, measured and compared in one command" +echo "# measure mode: the sides alternate over $ROUNDS rounds, one work order each" +bx compare "$WORK/build-base:$WORK/wt-base" "$WORK/build-head:$WORK/wt-head" \ + --rounds "$ROUNDS" --run-key "$RUN_REV" --out "$WORK/results-rev" # adapter, suite and profile: benchx.conf.json +echo "# the first work order it wrote:" +show cat "$WORK/results-rev/orders/order-0000.json" +echo +echo "# read mode: the same run key, compared again through a throwaway store" +bx compare --run "$RUN_REV" --baseline "${BASE:0:10}" --results "$WORK/results-rev" +echo "# bx compare also delivered these to the local store, as thin results of project local/" + +step "environments, tracked: project demo, into the local store" +bx compare "$WORK/build-head:$WORK/wt-head" "$WORK/build-hardened:$WORK/wt-head" --profile environments \ + --label build=plain,hardened --rounds "$ROUNDS" --project demo \ + --run-key "$RUN_ENV" --out "$WORK/results-env" + +step "re-ingest is a visible no-op" +bx ingest "$WORK/results-env" + +step "a conflicting unit is rejected with its taxonomy code" +echo "# \$WORK/bad.json: a copy of one result, its unit changed from s to ms" +"$PYTHON" - "$WORK" <<'EOF' +import json, pathlib, sys +work = pathlib.Path(sys.argv[1]) +doc = json.loads(next(p for p in sorted((work / "results-env").glob("*_wall-time_*.json"))).read_text()) +doc["ingest_key"] += ":in-ms" +doc["measurement"]["quantity"]["unit"] = "ms" +(work / "bad.json").write_text(json.dumps(doc)) +EOF +bx ingest "$WORK/bad.json" || true + +step "compare from the local store" +bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain + +step "the same comparison through a throwaway store gives the same document" +into "$WORK/a.json" bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain --json +into "$WORK/b.json" bx compare --run "$RUN_ENV" --profile environments --label build --baseline plain --json \ + --results "$WORK/results-env" +show cmp "$WORK/a.json" "$WORK/b.json" && echo "identical" + +step "series and history (a query; history mode is deferred)" +into "$WORK/series.txt" bx series --workload BM_Quiet --quantity wall-time +show head -6 "$WORK/series.txt" +SERIES=$(awk '/demo .*est=benchx\/median/ {print $1; exit}' "$WORK/series.txt") +into "$WORK/history.txt" bx history "$SERIES" +show head -1 "$WORK/history.txt" +show tail -n 4 "$WORK/history.txt" + +step "the store is one Parquet file, and keeps growing" +show ls -l "$STORE" +bx head -n 8 +printf '\ndemo artifacts in %s\n' "$WORK"