CI: make pyperf-suite example executable

.github/workflows/e2e.yml

@@ -1,7 +1,16 @@
-name: E2E pyperf-suite
+name: E2E
 
+# One job per example, each calling the reusable workflow. The `paths` filter below applies
+# to the whole workflow, so extend it when an example is added.
 on:
   push:
+    branches: [main]
+    paths:
+      - "examples/pyperf-suite/**"
+      - "scripts/execute_markdown.py"
+      - ".github/workflows/e2e.yml"
+      - ".github/workflows/reusable-example.yml"
+
   pull_request:
   workflow_dispatch:
 
@@ -10,8 +19,9 @@ permissions:
 
 jobs:
   pyperf-suite:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Bootstrap
-        run: echo "Workflow registered successfully"
-
+    uses: ./.github/workflows/reusable-example.yml
+    with:
+      example: pyperf-suite
+      benchx-source: pypi
+      examples-repository: ${{ github.repository }}
+      examples-ref: ${{ github.sha }}

.github/workflows/reusable-example.yml

@@ -0,0 +1,153 @@
+name: Reusable example E2E
+
+# Executes examples/<example>/README.md with scripts/execute_markdown.py and uploads
+# three artifacts: logs-<example>, docs-<example> (the generated page) and results-<example>
+# (the example's out/ directory).
+#
+#   benchx-source: pypi      the released BenchX; used by the benchx_examples own CI
+#   benchx-source: checkout  the calling repository's checkout, which must be benchx;
+#                            used by benchx's CI
+#
+# The `example` input generalizes paths and artifact names, not the environment: the
+# install steps below fit examples that need only benchx, pyperf and the runner. Add
+# example-specific setup here, or a specialized workflow, as more examples are migrated.
+
+on:
+  workflow_call:
+    inputs:
+      example:
+        description: Directory name under examples/
+        required: true
+        type: string
+      benchx-source:
+        description: "Where BenchX comes from: pypi or checkout"
+        required: true
+        type: string
+      examples-ref:
+        description: Branch, tag or commit of benchx_examples to run
+        required: true
+        type: string
+      examples-repository:
+        description: >-
+          Repository to take the examples from. The benchx_examples own workflow passes
+          github.repository, so that runs in a fork use the fork's commits.
+        required: false
+        type: string
+        default: benchx-dev/benchx_examples
+
+permissions:
+  contents: read
+
+jobs:
+  example:
+    name: BenchX from ${{ inputs.benchx-source }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    env:
+      EXAMPLE: ${{ inputs.example }}
+      BENCHX_SOURCE: ${{ inputs.benchx-source }}
+
+    steps:
+      - name: Validate inputs
+        run: |
+          case "$BENCHX_SOURCE" in
+            pypi|checkout) ;;
+            *) echo "Invalid benchx-source" >&2; exit 1 ;;
+          esac
+          case "$EXAMPLE" in
+            ""|*[!a-zA-Z0-9_-]*) echo "Invalid example name" >&2; exit 1 ;;
+          esac
+
+      - name: Check out benchx_examples
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          repository: ${{ inputs.examples-repository }}
+          ref: ${{ inputs.examples-ref }}
+          path: benchx_examples
+          persist-credentials: false
+
+      # No `repository`: this is the calling repository, which is benchx in this mode.
+      - name: Check out benchx
+        if: inputs.benchx-source == 'checkout'
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          path: benchx
+          persist-credentials: false
+
+      # Only the checkout mode has a lock file to key the cache on.
+      - uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2  # v10.3.3
+        with:
+          enable-cache: ${{ inputs.benchx-source == 'checkout' }}
+          cache-dependency-glob: benchx/uv.lock
+
+      # The example scripts run plain `python` and `bx`, and the benchmark workers use the
+      # same `python`, so the environment goes on the PATH.
+      - name: Install BenchX from PyPI
+        if: inputs.benchx-source == 'pypi'
+        run: |
+          uv python install 3.13
+          uv venv --python 3.13 "$RUNNER_TEMP/venv"
+          uv pip install --python "$RUNNER_TEMP/venv/bin/python" benchx pyperf markdown-it-py
+          echo "$RUNNER_TEMP/venv/bin" >> "$GITHUB_PATH"
+
+      # An editable install, which is where BenchX reads its schemas from. The `test`
+      # extra brings pyperf.
+      - name: Install BenchX from the checkout
+        if: inputs.benchx-source == 'checkout'
+        working-directory: benchx
+        run: |
+          uv python install 3.13
+          uv sync --extra test --python 3.13
+          uv pip install --python .venv/bin/python markdown-it-py
+          echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
+
+      # `benchx` should be under site-packages for pypi and under benchx/src for checkout.
+      - name: Show what is under test
+        run: |
+          echo "python: $(python --version) ($(command -v python))"
+          echo "bx: $(command -v bx)"
+          echo "benchx: $(python -c 'import benchx; print(benchx.__file__)')"
+          echo "pyperf: $(python -c 'import pyperf; print(pyperf.__version__)')"
+          if [ -d benchx ]; then
+            echo "benchx checkout: $(git -C benchx rev-parse --short HEAD)"
+          fi
+          echo "benchx_examples: $(git -C benchx_examples rev-parse --short HEAD)"
+
+      # Run from the root of the examples checkout, so that annotations carry a path
+      # relative to the repository. The generated page and the logs go outside it: BenchX
+      # records the state of the whole examples tree, and new files in it could make
+      # later results dirty.
+      - name: Execute example
+        working-directory: benchx_examples
+        run: |
+          python scripts/execute_markdown.py \
+            --source "examples/$EXAMPLE/README.md" \
+            --output "../generated/$EXAMPLE/$EXAMPLE.md" \
+            --logs "../logs/$EXAMPLE"
+
+      - name: Upload logs
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9  # v7.0.2
+        with:
+          name: logs-${{ inputs.example }}
+          path: logs/${{ inputs.example }}/
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Upload generated documentation
+        if: success()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9  # v7.0.2
+        with:
+          name: docs-${{ inputs.example }}
+          path: generated/${{ inputs.example }}/
+          if-no-files-found: error
+          retention-days: 14
+
+      - name: Upload benchmark results
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9  # v7.0.2
+        with:
+          name: results-${{ inputs.example }}
+          path: benchx_examples/examples/${{ inputs.example }}/out/
+          if-no-files-found: ignore
+          retention-days: 14

.gitignore

@@ -0,0 +1,2 @@
+generated/
+logs/

examples/pyperf-suite/README.md

@@ -4,24 +4,68 @@ A runnable example of a benchmark suite. `bench_suite.py` holds five small
 functions; `bx bench` runs each one under [pyperf](https://pyperf.readthedocs.io),
 writes benchx result files, delivers them to a local store, and prints a table.
 
-## Run it
+## First, install benchx
+
+We install `benchx` with `pyperf`
+
+```
+$ python3 -m venv .venv && . .venv/bin/activate    # Python >= 3.11, or any environment you like
+$ pip install pyperf benchx
+```
+
+## Benchmarks
+
+In the simplest case, benchmarks are plain python functions. Here is an example suite
+of several small microbenchmarks (here we only show one function out of five in the suite):
+
+```sh
+head -20 bench_suite.py
+```
+
+```text
+"""Example benchmark suite: plain Python functions, one case each.
+
+`bx bench bench_suite.py` runs every function named bench_* or benchmark_*.
+Name functions on the command line to run only those:
+    bx bench bench_suite.py bench_sorted bench_join
+
+Each function is the timed body; pyperf calls it many times. Do your setup
+at module level (it runs once per worker process, outside the timing).
+"""
+
+import os
+
+DATA = list(range(2000, 0, -1))
+WORDS = [f"word{i}" for i in range(500)]
+
+
+def bench_sorted():
+    sorted(DATA)
+```
+
+Additionally, this directory contains two additional files: `benchx.conf.json`
+is an optional configuration file, and `run.sh` is a helper script which demonstrates
+basic CLI commands.
+
+## Run the benchmarks
+
+The main CLI entry point for the `benchx` package is the `bx` executable, which
+supports several subcommands. The first subcommand is `bx bench` which accepts the
+name of the suite or a selection of benchmarks, and runs it: to run the whole suite,
+use `$ bx bench bench_suite.py`. Here we demonstrate several `bx` subcommands, in a
+shell script (see below for specific steps the script):
 
-Install benchx in editable mode from a checkout (it reads its schemas from there), with
-pyperf, then run the script from the root of this repository:
 
 ```sh
-git clone https://github.com/benchx-dev/benchx.git
-python3 -m venv .venv && . .venv/bin/activate    # Python >= 3.11, or any environment you like
-pip install -e benchx/prototype pyperf
-bash examples/pyperf-suite/run.sh
+bash ./run.sh
 ```
 
-The script runs `bx` and `python` from the active environment, so benchx and pyperf must
-be installed there; the workers use that same `python`. Extra arguments go to
+The script runs `bx` and `python` from the active environment, so `benchx` and `pyperf`
+must be installed there; the workers use that same `python`. Extra arguments go to
 `bx bench` in step 1 and win over the config file, e.g. `./run.sh --processes 20`.
-It takes about 20 seconds. Nothing outside this directory changes: the local store is
-`out/store.parquet` (`paths.store` in `benchx.conf.json`), and `out/` is deleted at the start
-of each run.
+On a particular machine, the run takes about 20 seconds.
+Nothing outside the root directory changes: the local store is `out/store.parquet`
+(`paths.store` in `benchx.conf.json`).
 
 ## What goes in
 
@@ -103,7 +147,8 @@ bench_join         3.585 us  3.543 us ± 272.4 ns  3.236 us .. 3.997 us  15  suc
 underneath. A `[warning]` after the status means the result carries a quality warning,
 such as `env-mismatch`.
 
-Files, under `out/all/` and `out/env/`:
+Files, under `out/all/`, `out/env/` and `out/again/` (one directory for each of the first
+three `bx bench` calls in `run.sh`):
 
 - `<run-key>_<case>_wall-time.json`: one benchx result per case (schema `measurement-result`;
   with `-e` the name also carries the percent-encoded parameters, as in
@@ -120,8 +165,9 @@ Files, under `out/all/` and `out/env/`:
   script that ran the function; `.worker-env.json`: the environment a worker really had;
   `.stdout` and `.stderr`.
 
-The local store, `out/home/store.parquet`, holds the results. `run.sh` reads it back
-(your numbers and fingerprints will differ):
+The local store, `out/store.parquet`, holds the results. `run.sh` reads it back, and
+keeps what `bx series` and `bx history` print in `out/series.txt`, `out/join-series.txt`
+and `out/history.txt` (your numbers and fingerprints will differ):
 
 ```
 $ bx series --quantity wall-time