Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,17 @@ jobs:
if: runner.os != 'Windows'
shell: bash
run: uv run --extra test --python ${{ matrix.python-version }} bash examples/archery-local/run.sh

# The end-to-end demo of both profiles: builds a small CMake project three
# ways, then measures and compares it. Needs cmake and a C++ compiler
# (preinstalled on the Linux and macOS runners) and network access to fetch
# Google Benchmark. ROUNDS is lowered from the default of 5 to keep the job
# short; it changes how many samples are taken, not what the script exercises.
# Run directly, the way the README shows it, so a lost executable bit
# fails here too.
- name: Run the archery-compare example
if: runner.os != 'Windows'
shell: bash
env:
ROUNDS: "2"
run: uv run --extra test --python ${{ matrix.python-version }} examples/archery-compare/run.sh
2 changes: 1 addition & 1 deletion docs/design/prototype-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,7 @@ re-ingested to show idempotency, and compared again with `--results`; that
document must equal the one from the local store (criterion 8).

Every `bx run` delivers to the local store, so both shapes' results land in
the demo project's store (`prototype/examples/demo/store.parquet`), the ad hoc ones as thin results of project
the demo project's store (`prototype/examples/archery-compare/store.parquet`), the ad hoc ones as thin results of project
`local/<hostname>`. Run keys carry a timestamp, so successive demo runs add
rows rather than colliding on `ingest_key`. The store and each series'
history grow from run to run, and the demo ends with `bx head`.
Expand Down
12 changes: 6 additions & 6 deletions prototype/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,10 +174,10 @@ every result is pyperf. Results also record the installed project's version
```console
$ uv venv && uv pip install -e '.[test]' # editable: schemas are read from ../schemas
$ .venv/bin/python -m pytest tests # 344 tests, fake Google Benchmark binary, no compiler
$ ./demo.sh # real builds; needs cmake, a C++ compiler, network once
$ examples/archery-compare/run.sh # real builds; needs cmake, a C++ compiler, network once
```

`demo.sh` prints every command before running it, as you would type it
`examples/archery-compare/run.sh` prints every command before running it, as you would type it
(`bx` for `.venv/bin/python -m benchx.cli`, `$WORK` for its scratch
directory), so it doubles as a walkthrough. It prepares a git source with two
commits and three CMake build directories of `examples/demo-suite` (base,
Expand All @@ -194,14 +194,14 @@ part, then:
persistent one, lists series and history, and ends with `bx head`.

Every `bx run` and every measure-mode `bx compare` delivers to the local store, so each demo run adds its
results, ad hoc and tracked, to `examples/demo/store.parquet` (the demo's project root, set
results, ad hoc and tracked, to `examples/archery-compare/store.parquet` (the demo's project root, set
in its `benchx.conf.json`). Run keys carry a
timestamp, so runs accumulate and each series' history grows. Look at it any
time:

```console
$ (cd examples/demo && ../../.venv/bin/python -m benchx.cli head -n 20)
$ rm examples/demo/store.parquet # start over
$ (cd examples/archery-compare && ../../.venv/bin/python -m benchx.cli head -n 20)
$ rm examples/archery-compare/store.parquet # start over
```

Both changes add two to three percent of work. The quiet benchmark flags them
Expand Down Expand Up @@ -237,7 +237,7 @@ benchx/
cli.py bx (compare has a read mode and a measure mode)
../schemas/ the result and work-order schemas, read in place ($BENCHX_SCHEMAS overrides)
examples/demo-suite a two-benchmark Google Benchmark suite for the demo
examples/demo the demo's project root: its benchx.conf.json, and its persistent store
examples/archery-compare the demo: run.sh, its project root's benchx.conf.json, and its persistent store
examples/pyperf-bench Python functions under `bx bench` (own README)
examples/archery-local an end-to-end, Arrow-local-shaped example with a target provider (own README)
tests/ test_prototype.py: the success criteria and the runner and store
Expand Down
15 changes: 8 additions & 7 deletions prototype/demo.sh → prototype/examples/archery-compare/run.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,18 @@
# Google Benchmark unless it is installed. ROUNDS=5 by default.
#
# `bx compare` (measure mode) and `bx run` deliver every result to the one local store,
# examples/demo/store.parquet (paths.store in examples/demo/benchx.conf.json, the demo's
# project file, from whose directory every `bx` command runs), so the store and
# each series' history grow from run to run. `bx head` shows the latest rows
# at any time. Delete the file to start over.
# examples/archery-compare/store.parquet (paths.store in
# examples/archery-compare/benchx.conf.json, the demo's project file, from whose
# directory every `bx` command runs), so the store and each series' history grow
# from run to run. `bx head` shows the latest rows at any time. Delete the file
# to start over.
set -euo pipefail
cd "$(dirname "$0")"
PROTOTYPE=$PWD
DEMO_ROOT=$PWD
PROTOTYPE=$(cd ../.. && pwd)
PYTHON=${PYTHON:-$PROTOTYPE/.venv/bin/python}
ROUNDS=${ROUNDS:-5}
WORK=$(mktemp -d)
DEMO_ROOT=$PROTOTYPE/examples/demo
STORE=$DEMO_ROOT/store.parquet
STAMP=$(date -u +%Y%m%dT%H%M%SZ) # run keys are unique per demo run, so runs accumulate
RUN_REV="rev-adhoc-$STAMP"
Expand Down Expand Up @@ -59,7 +60,7 @@ into() { # into <file> <bx ...>: run with stdout to a file, printed with its re
printf '\033[36m$ WORK=%s\033[0m\n' "$WORK" >&2

step "user: a source with two commits, a worktree at each"
show cp -R examples/demo-suite "$WORK/src"
show cp -R ../demo-suite "$WORK/src"
show git -C "$WORK/src" init -q
show git -C "$WORK/src" add -A
show git -C "$WORK/src" commit -qm "base"
Expand Down
7 changes: 4 additions & 3 deletions prototype/examples/archery-local/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ The documents involved are in `../../../schemas/`: `target-provider-request`,

The provider here builds one source tree, `WORKSPACE`. Comparing against a tag
or a past revision (`@plain=v1.0`) is the provider's business and is not
implemented in this example; the prototype's own `demo.sh` shows revisions
with two prebuilt worktrees. A declared `build` object in a description is
refused by the prototype's runner (it reads `CMakeCache.txt` instead).
implemented in this example; the prototype's own
`examples/archery-compare/run.sh` shows revisions with two prebuilt worktrees.
A declared `build` object in a description is refused by the prototype's
runner (it reads `CMakeCache.txt` instead).
Loading