Skip to content

add infrastructre for executable examples, add results to the web page #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ev-br wants to merge 4 commits into
base: main
Choose a base branch
from
+405 −88
Open

add infrastructre for executable examples, add results to the web page #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ec9105b
WIP: add reusable workflows for CI
Nov 14, 2026
aa565d8
CI: add the Examples page to the docs, populate it from the reusable …
Nov 14, 2026
07e124c
MAINT: make local doc builds the same as CI
Nov 14, 2026
c488381
ENH: add the site builder script
Nov 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 87 additions & 7 deletions .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
name: docs

# Builds the documentation site and deploys it to GitHub Pages.
#
# What triggers a deployment (job `deploy`):
# - a push to main that changes one of the paths below, usually the merge of a pull request;
# - a manual run on main: Actions > docs > Run workflow, branch main.
# A pull request, and a manual run on another branch, only build the site.
#
# The Examples page shows pages that the e2e workflow (e2e.yml) generated from the examples.
# Each build takes, for every example, the newest docs-<example> artifact of a run of e2e.yml
# on main. A change to an example, or to what e2e produces, therefore reaches the site with the
# next deployment, not by itself. To deploy after such a change, see the steps at
# `workflow_dispatch` below.

on:
push:
branches: [main]
Expand All @@ -19,6 +32,14 @@ on:
- "zensical.toml"
- "README.md"
- ".github/workflows/docs.yml"
# To deploy the site again without a change under the paths above, for example after an
# example changed in benchx_examples:
# 1. Run "e2e examples" by hand on main (Actions > e2e examples > Run workflow), and wait
# until its job for the example has finished: a build takes the example pages from the
# newest e2e runs on main, and a run that is still going has no page to give yet.
# 2. Run this workflow by hand on main (Actions > docs > Run workflow).
# A run on main deploys, like a push to main does. A run on another branch only builds.
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -30,23 +51,82 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
# actions: read, to find and download the example pages (the first two steps).
permissions:
contents: read
actions: read
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3

# The pages of the examples are made by the e2e workflow, which runs on every push to
# main: the job of each example uploads a docs-<example> artifact. For each example, take
# the newest artifact from a run of e2e.yml on main that was not started by a pull request
# (a pull request from a fork whose branch is called main must not get in). Whether the
# other jobs of that run passed does not matter: an example uploads its page only when
# it succeeded. The run for the push being built is usually still going, so the page
# is typically the one from the push before. Artifacts expire after 14 days; an example
# without an artifact is left out of the site, with a warning.
- name: Download the example pages from the latest e2e runs on main
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
declare -A found
count=0
runs=$(gh run list --repo "$GITHUB_REPOSITORY" --workflow e2e.yml --branch main \
--limit 20 --json databaseId,event \
--jq '.[] | select(.event != "pull_request") | .databaseId' || true)
while IFS= read -r run_id; do
[ -n "$run_id" ] || continue
names=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$run_id/artifacts?per_page=100" \
--jq '.artifacts[] | select(.expired | not) | .name' || true)
while IFS= read -r name; do
[[ "$name" =~ ^docs-[A-Za-z0-9_-]+$ ]] || continue
[ -z "${found[$name]:-}" ] || continue
if gh run download "$run_id" --repo "$GITHUB_REPOSITORY" \
--name "$name" --dir "generated/$name"; then
found[$name]="$run_id"
count=$((count + 1))
echo "$name: from e2e run $run_id"
fi
done <<< "$names"
done <<< "$runs"
if [ "$count" -eq 0 ]; then
echo "::warning::No example pages: no unexpired docs-* artifact in the last 20 e2e runs on main. The site is built without examples."
fi

- uses: astral-sh/setup-uv@v5
# One directory per example, docs/examples/<example>/, with the page <example>.md and
# its assets/. The Examples page lists whatever is there (tools/gen_docs.py).
- name: Assemble the example pages into docs/examples/
run: |
set -euo pipefail
for artifact in generated/docs-*; do
[ -d "$artifact" ] || continue
example="${artifact##*/docs-}"
case "$example" in
""|*[!a-zA-Z0-9_-]*) echo "Invalid example name in $artifact" >&2; exit 1 ;;
esac
mkdir -p "docs/examples/$example"
cp -R "$artifact/." "docs/examples/$example/"
done
if [ -d docs/examples ]; then find docs/examples -type f | sort; fi

- name: Generate derived pages (landing page, benchx page, schema reference)
- name: Generate derived pages (landing page, benchx page, schema reference, Examples page)
run: uv run --python 3.12 --with-requirements docs/requirements.txt python tools/gen_docs.py

- name: Build site
run: uv run --python 3.12 --with-requirements docs/requirements.txt zensical build --strict --clean

- uses: actions/upload-pages-artifact@v3
# Since v4 this leaves hidden files (dotfiles) out of the artifact; the site has none.
# If it ever needs one, pass `include-hidden-files: true` (an input from v5 on).
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: site

deploy:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main'
needs: build
runs-on: ubuntu-latest
permissions:
Expand All @@ -56,7 +136,7 @@ jobs:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- uses: actions/configure-pages@v5
- uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0

- id: deployment
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1
123 changes: 50 additions & 73 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ name: e2e examples
# arrow builds a small Google Benchmark suite with CMake and compares the builds:
# examples/arrow/demo/run.sh
# examples/arrow/archery-local/run.sh
# pyperf-suite runs a suite of Python functions with `bx bench`:
# pyperf-suite runs a suite of Python functions with `bx bench`, by executing the
# example's README through benchx_examples' reusable workflow (see the job):
# examples/pyperf-suite/run.sh
# matmul-params runs one parametrized benchmark, numpy and torch at three sizes, with
# `bx bench`; the quick version from the example's README:
Expand All @@ -20,9 +21,17 @@ name: e2e examples
# Nothing is checked. The job fails only if a step fails (a clone, a build, a script that
# crashes); the scripts exit 0 even when a revision fails, so read the printed output.
#
# Runs on every pull request, and on demand. A run on demand can take another
# benchx_examples branch, tag or commit (examples_ref), which a pull request here cannot:
# the examples live in another repository.
# Runs on every pull request, on every push to main, and on demand. A run on demand can take
# another benchx_examples branch, tag or commit (examples_ref), which a pull request here
# cannot: the examples live in another repository.
#
# The runs on main are what the documentation site shows of the examples: docs.yml takes, for
# each example, the docs-<example> artifact of the newest run on main (see the comment there).
# This workflow deploys nothing itself. To put a change to an example on the site: run this
# workflow by hand on main (Actions > e2e examples > Run workflow), wait until the job of
# the example has finished, then run docs.yml by hand on main (Actions > docs > Run workflow).
# A merge that changes the docs paths deploys the site by itself, with the pages of the runs
# that had finished by then.
#
# In the numpy-spin job the benchmarks are pinned to two cores (BENCH_CPUS). num_threads.sh
# should use 1, 2 and 4 threads, so that the last of them oversubscribes the two cores. The
Expand All @@ -35,6 +44,8 @@ on:
description: "benchx_examples branch, tag or commit to run"
default: main
pull_request:
push:
branches: [main]

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -56,12 +67,12 @@ jobs:
CCACHE_MAXSIZE: 500M
steps:
- name: Check out benchx
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: benchx

- name: Clone benchx_examples
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: benchx-dev/benchx_examples
ref: ${{ inputs.examples_ref || 'main' }}
Expand All @@ -81,7 +92,7 @@ jobs:
# The key is new for every run, because a cache cannot be overwritten; a run starts
# from the most recent one. It is saved at the end, also when a later step fails.
- name: Restore the ccache cache
uses: actions/cache/restore@v4
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ runner.os }}-${{ github.run_id }}-${{ github.run_attempt }}
Expand All @@ -90,7 +101,7 @@ jobs:
- name: Zero the ccache statistics
run: ccache --zero-stats

- uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true
cache-dependency-glob: benchx/uv.lock
Expand Down Expand Up @@ -164,14 +175,14 @@ jobs:

- name: Save the ccache cache
if: always()
uses: actions/cache/save@v4
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ runner.os }}-${{ github.run_id }}-${{ github.run_attempt }}

- name: Upload the output
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: numpy-spin-output
path: |
Expand All @@ -186,18 +197,18 @@ jobs:
timeout-minutes: 45
steps:
- name: Check out benchx
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: benchx

- name: Clone benchx_examples
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: benchx-dev/benchx_examples
ref: ${{ inputs.examples_ref || 'main' }}
path: benchx_examples

- uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true
cache-dependency-glob: benchx/uv.lock
Expand Down Expand Up @@ -232,7 +243,7 @@ jobs:

- name: Upload the output
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: arrow-output
path: |
Expand All @@ -241,58 +252,24 @@ jobs:
if-no-files-found: warn
retention-days: 14

# The example's README is executed by the reusable workflow of benchx_examples, which
# installs this checkout (benchx-source: checkout), runs examples/pyperf-suite/run.sh
# through the README, and uploads three artifacts: logs-pyperf-suite, results-pyperf-suite
# (the example's out/) and docs-pyperf-suite (the generated page, which docs.yml publishes).
# It prints the same "what is under test" lines as the other jobs.
#
# TEMPORARY: benchx_examples PR #16 (branch ci_exec_md of ev-br/benchx_examples) is not
# merged yet, so the workflow and the examples both come from that branch, and the
# `examples_ref` input is not used here. After the merge, use instead:
# uses: benchx-dev/benchx_examples/.github/workflows/reusable-example.yml@main
# with: examples-repository omitted, examples-ref: ${{ inputs.examples_ref || 'main' }}
pyperf-suite:
name: pyperf-suite (run.sh)
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Check out benchx
uses: actions/checkout@v4
with:
path: benchx

- name: Clone benchx_examples
uses: actions/checkout@v4
with:
repository: benchx-dev/benchx_examples
ref: ${{ inputs.examples_ref || 'main' }}
path: benchx_examples

- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: benchx/uv.lock

# The script runs plain `python` and `bx`, and the workers need pyperf, which the
# `test` extra brings. benchx is installed in editable mode, which is where it reads
# its schemas from.
- name: Install benchx
working-directory: benchx
run: |
uv python install 3.13
uv sync --extra test --python 3.13
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"

- name: Show what is under test
run: |
echo "python: $(python --version) ($(command -v python))"
echo "bx: $(command -v bx)"
echo "pyperf: $(python -c 'import pyperf; print(pyperf.__version__)')"
echo "benchx: $(git -C benchx rev-parse --short HEAD)"
echo "benchx_examples: $(git -C benchx_examples rev-parse --short HEAD)"

- name: Run the pyperf suite
working-directory: benchx_examples/examples/pyperf-suite
run: bash ./run.sh

- name: Upload the output
if: always()
uses: actions/upload-artifact@v4
with:
name: pyperf-suite-output
path: benchx_examples/examples/pyperf-suite/out
if-no-files-found: warn
retention-days: 14
uses: ev-br/benchx_examples/.github/workflows/reusable-example.yml@ci_exec_md
with:
example: pyperf-suite
benchx-source: checkout
examples-repository: ev-br/benchx_examples
examples-ref: ci_exec_md

matmul-params:
name: matmul-params (quick version)
Expand All @@ -303,18 +280,18 @@ jobs:
BENCHX_HOME: ${{ github.workspace }}/benchx_examples/examples/matmul-params/out/home
steps:
- name: Check out benchx
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: benchx

- name: Clone benchx_examples
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: benchx-dev/benchx_examples
ref: ${{ inputs.examples_ref || 'main' }}
path: benchx_examples

- uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true
cache-dependency-glob: benchx/uv.lock
Expand Down Expand Up @@ -354,7 +331,7 @@ jobs:

- name: Upload the output
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: matmul-params-output
path: benchx_examples/examples/matmul-params/out
Expand All @@ -367,18 +344,18 @@ jobs:
timeout-minutes: 30
steps:
- name: Check out benchx
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: benchx

- name: Clone benchx_examples
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: benchx-dev/benchx_examples
ref: ${{ inputs.examples_ref || 'main' }}
path: benchx_examples

- uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true
cache-dependency-glob: benchx/uv.lock
Expand Down Expand Up @@ -410,7 +387,7 @@ jobs:

- name: Upload the output
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: numpy-venvs-output
path: benchx_examples/examples/numpy-venvs/out
Expand Down
Loading
Loading