ENH: Add Arrow/archery examples

README.md

@@ -7,4 +7,6 @@ represent best practices.
 Currently, the following examples are provided:
 
 - `examples/numpy-spin`: out-of-tree benchmarks for NumPy, using `spin` to build NumPy itself;
+- `examples/arrow`: Arrow-shaped examples with Google Benchmark and CMake: a walkthrough of both
+  comparison profiles (`demo`) and one of the target provider contract (`archery-local`).
 

examples/arrow/README.md

@@ -0,0 +1,52 @@
+# Arrow-shaped examples: Google Benchmark and CMake
+
+Two end-to-end examples in the shape of the Arrow local-benchmarking story, on one
+small stand-in suite. They used to live in the benchx repository, as
+`prototype/examples/{demo-suite,demo,archery-local}` and `prototype/demo.sh`.
+
+| Directory | What it is |
+|---|---|
+| `demo-suite/` | a two-benchmark Google Benchmark suite (`BM_Quiet` and `BM_Noisy`), built with CMake; `-DDEMO_HARDENED=ON` adds about 3% of work |
+| `demo/` | `run.sh`, a walkthrough of both comparison profiles that prints every command before running it; the project file (`benchx.conf.json`) and the local store it keeps |
+| `archery-local/` | a target provider walkthrough: one commit built plain and hardened, and compared; has its own README |
+
+## Run them
+
+You need benchx installed in editable mode from a checkout (it reads its schemas from
+there), `cmake`, a C++ compiler, `git` and `python3`, and network access the first time
+(CMake fetches Google Benchmark unless it is installed). From the root of this
+repository:
+
+```console
+$ git clone https://github.com/benchx-dev/benchx.git
+$ python3 -m venv .venv && . .venv/bin/activate    # Python >= 3.11
+$ pip install -e benchx/prototype
+$ bash examples/arrow/demo/run.sh
+$ bash examples/arrow/archery-local/run.sh
+```
+
+`ROUNDS=2` shortens either one; the default is 5.
+
+## What `demo/run.sh` shows
+
+It prepares a git source with two commits and three CMake build directories of
+`demo-suite` (base, head, and head with `DEMO_HARDENED=ON`), shows the first work
+order of each part, then:
+
+1. **revisions, ad hoc:** `bx compare` in measure mode alternates base and head over the
+   rounds with orders that name no project, then read mode compares the same run key
+   through a throwaway store;
+2. **environments, tracked:** `bx compare` alternates plain and hardened, labeled
+   `build: plain|hardened`, compares from the local store, re-ingests (a no-op), and
+   rejects a unit conflict;
+3. checks that the throwaway store gives the same comparison document as the persistent
+   one, lists series and history, and ends with `bx head`.
+
+Every measure-mode `bx compare` delivers to the local store, `demo/store.parquet`, so
+each run adds its results to it. Run keys carry a timestamp, so runs accumulate and each
+series' history grows. Look at it at any time, or start over:
+
+```console
+$ (cd examples/arrow/demo && python -m benchx.cli head -n 20)
+$ rm examples/arrow/demo/store.parquet
+```

examples/arrow/archery-local/.gitignore

@@ -0,0 +1,2 @@
+_work/
+results/

examples/arrow/archery-local/README.md

@@ -0,0 +1,142 @@
+# archery-local: an end-to-end example
+
+One commit, built two ways (plain and hardened), compared with noise-aware
+verdicts, where the comparison says what was built and how. It follows the
+Arrow local story, "Measure what a build option costs"
+([`docs/user_stories/apache-arrow-local-benchmarking.md`](https://github.com/benchx-dev/benchx/blob/main/docs/user_stories/apache-arrow-local-benchmarking.md) in benchx),
+on a small stand-in suite (`../demo-suite`: a quiet and a noisy Google
+Benchmark case, where hardening adds about 3% of work).
+
+It exercises the target-provider contract of
+[`system-decomposition.md`](https://github.com/benchx-dev/benchx/blob/main/docs/design/system-decomposition.md) §3.4:
+
+- **You are the user, and the provider is your build script.** `provider.sh`
+  configures and builds with CMake, as `archery` or a contributor would. benchx
+  builds nothing.
+- **The provider tells benchx what it built** in a *target description*: the
+  build directory and its source, plus two pieces of text you own, `how_built`
+  (the exact commands) and `activation` (what to run to enter the environment,
+  with the `shell` it is written for). Either may be empty.
+- **You activate the environment yourself** (`source activate.sh`). benchx never
+  runs `activation`. It records the text, as given, in the work order and in
+  every result's `provenance.info`, beside the allowlisted environment variables
+  the runner actually inherited. If the two disagree, both stay and nothing
+  judges.
+
+## Run it
+
+You need `cmake`, a C++ compiler, `git` and `python3` on your `PATH`, and network
+access the first time (CMake fetches Google Benchmark v1.9.1 unless it is
+installed). Install benchx in editable mode from a checkout, which is where it reads
+its schemas from, then run the script from the root of this repository:
+
+```console
+$ git clone https://github.com/benchx-dev/benchx.git
+$ python3 -m venv .venv && . .venv/bin/activate    # Python >= 3.11, or any environment you like
+$ pip install -e benchx/prototype
+$ bash examples/arrow/archery-local/run.sh
+```
+
+It takes about two minutes the first time (two CMake builds and a measured
+run) and prints every command before running it, as you would type it. The
+script runs `bx` and `python` from the active environment. More rounds:
+`ROUNDS=10 ./run.sh`.
+
+Everything it creates is under `_work/` in this directory, including the local
+store (`_work/store.parquet`, set by `paths.store` in `benchx.conf.json`), so nothing
+outside this directory changes. `rm -rf _work` starts over.
+
+## What you will see
+
+1. **Activate.** `source activate.sh` puts a stand-in dependencies directory on
+   `LD_LIBRARY_PATH`. This is you, not benchx.
+2. **`bx target describe @plain`** fails with exit 3 from the provider: nothing is
+   built yet. `describe` builds nothing, by contract.
+3. **`bx target prepare @plain` / `@hardened`** make the provider build. Its CMake
+   output is the build's own, on stderr; the target description it prints on
+   stdout is the answer. `how_built` is the very commands the provider ran.
+4. **`bx target describe @hardened`** now works. The same document is stored in
+   the build directory as `.benchx-target.json`, so a long-lived build directory
+   carries its own description (point `bx compare` at a bare directory and it
+   reads that file; no provider needed).
+5. **`bx compare @plain @hardened ...`** is the workbench's session loop: it
+   asks the provider to prepare each side (a no-op build now), then alternates
+   the sides over the rounds, one work order each, delivers the results to the
+   local store, and compares:
+
+   ```
+   run hardening-...  profile environments  baseline plain  contender hardened
+     plain: how built: cmake -S .../_work/src -B .../_work/build-plain ... (+1 lines)
+     plain: activation: source .../activate.sh
+     plain: observed env: LD_LIBRARY_PATH=.../_work/deps/lib
+     hardened: how built: cmake -S ... -B .../_work/build-hardened ... -DDEMO_HARDENED=ON (+1 lines)
+     ...
+   REGRESSED (1)
+     BM_Quiet/1024   wall-time  effect +2.42%  noise 0.57%  rounds=5
+   NO CHANGE DETECTED (1)
+     BM_Noisy/1024   wall-time  effect +3.43%  noise 3.04%  rounds=5
+   ```
+
+   Compare that with the story's pains: the comparison now describes itself
+   (what was built, how, in what environment), and the verdict uses each
+   benchmark's own noise instead of a flat 5%: the quiet case flags a 2.4%
+   change that is four times its noise, the noisy case does not flag a 3.4%
+   swing inside its noise. The numbers vary from run to run; with few rounds the
+   noisy case may flag too.
+6. **The work order and a result**, printed: the order's `target` carries
+   `how_built`, `activation`, `shell` and `provider`; the result's
+   `provenance.info` carries them, next to `coordinates.environment.system.process.environment` and
+   `coordinates.subject.configuration` (read by the runner from
+   `CMakeCache.txt`).
+7. **Forget to activate.** The same comparison with `LD_LIBRARY_PATH` removed
+   from the environment: the declaration still says `source .../activate.sh`,
+   the observed environment no longer lists `LD_LIBRARY_PATH`, and nothing
+   refuses: declarations are recorded as given (`benchmark-environments.md`
+   §3.2), never checked.
+
+## Try it by hand
+
+After `./run.sh` (the build directories and the store stay in `_work/`). From
+this directory, with the environment still active:
+
+```console
+$ source activate.sh
+
+$ bx target describe @hardened                    # what exists, nothing built
+$ bx target prepare  @hardened                    # incremental build, then the description
+$ bx target describe _work/build-hardened         # a bare directory: its own sidecar
+
+# the story's own knobs: scope by filter, precision by repetitions
+$ bx compare @plain @hardened --label build=plain,hardened --filter BM_Quiet --rounds 10 \
+      --repetitions 5 --run-key mine        # the adapter, suite, project and profile come from benchx.conf.json
+
+$ bx compare --run mine --label build --baseline plain   # read mode
+$ bx head -n 4                                    # the latest stored results
+$ bx head --json -n 1 | python3 -m json.tool | less   # provenance.info, environment.system
+```
+
+Edit `provider.sh`'s `hardened)` case or `activate.sh` and run again to see how
+each change shows up in the next comparison. Empty text is fine: set
+`"activation": ""` (or `null`, or leave it out) in a description and the
+runner simply omits it from the results.
+
+## What is in this directory
+
+| File | Role |
+|---|---|
+| `run.sh` | the end-to-end script |
+| `provider.sh` | the target provider: your build script, speaking the `describe`/`prepare` contract |
+| `benchx.conf.json` | the project file: the provider command, the harness and its suite, the project name, the comparison profile, and where the store and results go |
+| `activate.sh` | the environment you activate before calling `bx` |
+| `_work/` | everything generated: source copy, build directories, results, store (git-ignored) |
+
+The documents involved are in [`schemas/`](https://github.com/benchx-dev/benchx/tree/main/schemas) of benchx: `target-provider-request`,
+`target-description`, and `work-order` (its `target` takes the declared text).
+
+## What this does not show
+
+The provider here builds one source tree, `WORKSPACE`. Comparing against a tag
+or a past revision (`@plain=v1.0`) is the provider's business and is not
+implemented in this example; the [`demo`](../demo/run.sh) example shows revisions
+with two prebuilt worktrees. A declared `build` object in a description is
+refused by the prototype's runner (it reads `CMakeCache.txt` instead).

examples/arrow/archery-local/activate.sh

@@ -0,0 +1,8 @@
+# The environment this example's benchmarks are meant to run in. In the Arrow
+# story this is where you would load a toolchain or point at the dependencies
+# you built; here it only puts a stand-in "deps" directory on the library
+# search path. You source it yourself, before invoking bx: benchx never runs
+# it, it records the text that says you did (the target description's
+# `activation`) next to the environment it actually inherited.
+HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+export LD_LIBRARY_PATH="$HERE/_work/deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

examples/arrow/archery-local/benchx.conf.json

@@ -0,0 +1,13 @@
+{
+  "schema_version": "benchx/config/0.1.0",
+  "project": "archery-local",
+  "provider": { "command": ["./provider.sh"] },
+  "adapter": {
+    "name": "google-benchmark",
+    "suite": "demo-bench",
+    "quantities": ["wall-time"],
+    "precision": { "repetitions": 5 }
+  },
+  "paths": { "store": "_work/store.parquet", "results": "_work/results" },
+  "compare": { "profile": "environments" }
+}

examples/arrow/archery-local/provider.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# The target provider of this example (system-decomposition.md §3.4).
+#
+# It is the user's own build script, speaking the provider contract: one JSON
+# request on stdin (schemas/target-provider-request/0.1.0), one target
+# description on stdout (schemas/target-description/0.1.0). Everything else,
+# above all the build's own output, goes to stderr.
+#
+#   describe  reports a target that exists and builds nothing; exit 3 if none
+#   prepare   configures and builds it, with CMake, as Arrow's contributors do
+#             (an incremental no-op when it is already built), then reports it
+#
+# Configurations: `plain`, and `hardened` (-DDEMO_HARDENED=ON), the same
+# commit built two ways, like the -DARROW_HARDENING=ON story.
+#
+# `how_built` is assembled from the very commands this script runs, so the
+# text cannot drift from what happened. benchx never runs any of it.
+set -euo pipefail
+HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+WORK=${BENCHX_EXAMPLE_WORK:-$HERE/_work}
+export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=demo@example.org
+export GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=demo@example.org
+
+request=$(cat)
+field() { python3 -c 'import json,sys; print(json.load(sys.stdin).get(sys.argv[1]) or "")' "$1" <<<"$request"; }
+operation=$(field operation)
+configuration=$(field configuration)
+case "$configuration" in
+  plain)    extra=() ;;
+  hardened) extra=(-DDEMO_HARDENED=ON) ;;
+  *) echo "provider: unknown configuration '$configuration' (plain or hardened)" >&2; exit 2 ;;
+esac
+SRC=$WORK/src
+BUILD=$WORK/build-$configuration
+
+describe() {
+  [[ -f $BUILD/.benchx-target.json ]] || { echo "provider: $configuration is not built; use prepare" >&2; exit 3; }
+  cat "$BUILD/.benchx-target.json"
+}
+
+if [[ $operation == describe ]]; then describe; exit 0; fi
+
+HOW=""
+step() {  # step <command...>: record it in how_built, show it, run it (output to stderr)
+  local line="$*"
+  HOW+="$line"$'\n'
+  printf '\033[2m  provider$ %s\033[0m\n' "$line" >&2
+  "$@" >&2
+}
+
+if [[ ! -d $SRC/.git ]]; then
+  mkdir -p "$WORK"
+  cp -R "$HERE/../demo-suite" "$SRC"
+  git -C "$SRC" init -q
+  git -C "$SRC" remote add origin https://example.org/archery-local/demo-suite.git
+  git -C "$SRC" add -A
+  git -C "$SRC" commit -qm "demo-suite"
+fi
+mkdir -p "$WORK/deps/lib"   # stand-in for the dependencies activate.sh points at
+
+step cmake -S "$SRC" -B "$BUILD" -DCMAKE_BUILD_TYPE=Release -DFETCHCONTENT_BASE_DIR="$WORK/_deps" ${extra[@]+"${extra[@]}"}
+step cmake --build "$BUILD" -j
+
+BUILD=$BUILD SRC=$SRC HOW=$HOW CONFIGURATION=$configuration HERE=$HERE python3 - >"$BUILD/.benchx-target.json" <<'PY'
+import json, os
+print(json.dumps({
+    "schema_version": "benchx/target-description/0.1.0",
+    "target": {"kind": "build", "path": os.environ["BUILD"],
+               "source": {"uri": "https://example.org/archery-local/demo-suite.git", "type": "git"},
+               "source_dir": os.environ["SRC"]},
+    "how_built": os.environ["HOW"].rstrip("\n"),
+    "activation": "source " + os.path.join(os.environ["HERE"], "activate.sh"),
+    "shell": "bash",
+    "provider": {"name": "provider.sh", "version": "1"},
+}, indent=1))
+PY
+describe   # the sidecar just written, which stays with the build directory

examples/arrow/archery-local/run.sh

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+# End to end, in the shape of the Arrow local story ("Measure what a build
+# option costs"): one commit built two ways, plain and hardened, compared with
+# noise-aware verdicts, where the comparison describes itself.
+#
+# You play the user. The target provider (provider.sh, named in
+# benchx.conf.json) is your own build script: it builds with CMake and
+# tells benchx what it built, how, and what environment you need. benchx builds
+# nothing, and runs none of that text: you activate the environment yourself
+# (activate.sh) and benchx records the text beside the environment it inherited.
+#
+# Every command is printed before it runs, as you would type it. Run it in an
+# activated environment that has benchx installed, editable, from a benchx
+# checkout (`pip install -e <checkout>/prototype`). Needs cmake and a C++
+# compiler on PATH, and network access the first time, to fetch Google
+# Benchmark unless installed.
+#
+#   ROUNDS=5  alternation rounds (default 5)
+#
+# benchx.conf.json is this directory's project file, so the local store is kept
+# inside it too (_work/store.parquet). Delete _work/ to start over.
+set -euo pipefail
+cd "$(dirname "$0")"
+HERE=$PWD
+ROUNDS=${ROUNDS:-5}
+STAMP=$(date -u +%Y%m%dT%H%M%SZ)
+RUN="hardening-$STAMP"
+
+for tool in cmake c++ bx python; do
+  command -v "$tool" >/dev/null || { echo "run.sh: $tool not found on PATH (see README.md)" >&2; exit 1; }
+done
+bx --help >/dev/null 2>&1 || { echo "run.sh: benchx is not installed in the active environment (see README.md)" >&2; exit 1; }
+
+step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
+say() { printf '\033[36m$ %s\033[0m\n' "$*" >&2; }
+bx() { say bx "$@"; command bx "$@"; }
+
+step "user: activate the environment (benchx will not do this for you)"
+say source activate.sh
+source activate.sh
+echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
+
+step "ask the provider about a target that is not built yet: describe builds nothing"
+bx target describe @plain || echo "(exit $?: nothing to describe yet, as expected)"
+
+step "prepare both configurations: the provider builds (its build output is the dimmed lines)"
+bx target prepare @plain > /dev/null
+bx target prepare @hardened
+
+step "describe now works, and the description lives in the build directory too"
+bx target describe @hardened | head -3
+say head -3 _work/build-hardened/.benchx-target.json
+head -3 _work/build-hardened/.benchx-target.json
+echo "(the same document: a build directory carries its own description)"
+
+step "compare: the workbench asks the provider to prepare each side (a no-op), then alternates them"
+bx compare @plain @hardened --label build=plain,hardened --rounds "$ROUNDS" --run-key "$RUN" --out _work/results
+
+step "the comparison names what was built and how; before, it was two opaque directory paths"
+echo "# the target of work order 0001 (round 1, the hardened side), as the runner received it:"
+python -c '
+import json, sys
+print(json.dumps(json.load(open(sys.argv[1]))["target"], indent=1))' _work/results/orders/order-0001.json
+
+step "and every result carries it, next to what the runner saw (declared, observed: nothing judges)"
+say bx head --json -n 1 '| …'
+command bx head --json -n 1 | python -c '
+import json, sys
+doc = json.loads(sys.stdin.readline())
+info = doc["provenance"]["info"]
+print("provenance.info.how_built  :", info["how_built"].replace("\n", "\n" + " " * 29))
+print("provenance.info.activation :", info["activation"])
+print("provenance.info.shell      :", info["shell"])
+print("provenance.info.target_provider:", info["target_provider"])
+system = doc["coordinates"]["environment"]["system"]
+print("environment.system.process.environment:", system.get("process", {}).get("environment"))
+print("coordinates.subject.configuration:", doc["coordinates"]["subject"].get("configuration"))'
+
+step "forget to activate: the declaration still says what you should have run, the observation says what you did"
+say env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build --rounds 3 ...
+env -u LD_LIBRARY_PATH bx compare @plain @hardened --no-build \
+  --label build=plain,hardened --rounds 3 --run-key "$RUN-unactivated" --out _work/results-unactivated
+
+printf '\nartifacts: %s/_work (results, orders, build directories, the local store)\n' "$HERE"