README.md

Binding Performance Benchmarks

This suite compares wrappers generated by prik and NumPy's f2py on the same machine. It measures both runtime call and NumPy-array overhead and clean, end-to-end build time.

The existing default-interface cohort remains unchanged. A separate direct-entrypoint cohort uses sources/direct_kernels.f90 for a no-op, scalar function, and scalar subroutine. PRIK must call each bind(C) label without a generated user-procedure adapter. f2py retains its Python C/API binding while using --no-wrap-functions --skip-empty-wrappers; those flags suppress unneeded generated Fortran procedure wrappers/files, not the Python binding. An untimed preflight checks correctness, records the physical generated and compiled sources, and verifies that each direct Python binding object refers to the three user bind(C) labels while the native object and linked extension define those labels. Object discovery uses those symbol relationships rather than backend-specific filenames because Meson retains source suffixes in object names while other f2py backends do not. The selected binding and native object paths are recorded in the preflight report. These checks finish before either cohort enters a timer.

Every scalar case receives the same pre-created numpy.float64 inputs and produces the same numerical value. The timed calls retain each tool's natural public result: PRIK returns the contract's exact numpy.float64, while f2py returns a built-in float. The preflight checks those classes explicitly and the runtime metadata records the result contract; the benchmark does not hide the difference behind an untimed or Python-level normalization shim.

The default prik wrapper and the f2py wrappers measured here keep the GIL held, so the suite reports one like-for-like comparison of their normal generated interfaces.

Build timings treat each tool as a black box. A timed sample starts with an empty output directory and includes source processing, wrapper generation, native and generated-source compilation, and linking. Import verification runs after the timer stops. Both tools use their normal dependency-aware compiler concurrency. The workloads are:

• the one-source, five-procedure module used by the runtime suite; and
• the repository's authoritative 155-source Reference BLAS project under examples/blas/native/, with all 155 routines required in each generated extension.

One untimed warm-up precedes four measured clean builds of each workload by default. Tool order alternates between rounds. Set PRIK_BUILD_BENCHMARK_RUNS or PRIK_BUILD_BENCHMARK_WARMUPS to change those counts for local investigation.

Every workload is measured with two compiler profiles:

• development: -O0; and
• optimized: -O3 -march=native -mtune=native.

Runtime-call measurements continue to use only the optimized profile.

PRIK_BENCHMARK_FIRST=prik selects the first clean-build tool. Set it to f2py to reverse the first round; later build rounds alternate automatically.

For each runtime group, measurements use an A/B/B/A sequence in the same job: PRIK then f2py, immediately followed by f2py then PRIK. Each order receives half of the reduced worker budget, and the final per-tool suites merge both passes before comparison. The four order-specific JSON files remain in results/ so the balance is auditable. This keeps the same 13 workloads while targeting about 20 minutes on the pinned CI benchmark runner instead of doubling the previous 30-minute job.

Run the complete correctness check and rigorous benchmark with:

bash run.sh

The script rebuilds both runtime extensions, runs both clean-build profiles, and applies each profile consistently to the native Fortran source, generated Fortran wrapper, and generated C binding. Runtime extensions use the optimized profile. Runtime cases use latency, medium, and bulk sampling budgets: the short and noisier cases use more pyperf worker processes and values, while expensive matrix cases use fewer. Each case group runs adjacent PRIK/f2py measurements in both orders. The order passes are merged into the same final per-tool JSON suites, so comparison and publication commands remain unchanged. The script retains f2py's generated sources under build/f2py for local inspection.

Direct-entrypoint results are written separately as prik-direct.json/f2py-direct.json, while the ordinary-Fortran PRIK control is prik-adapted.json. Their clean-build counterparts end in -build.json. Metadata identifies the direct_c_abi or generated_fortran_adapter route, f2py wrapper mode, natural result class, compiler flags, process order, affinity protocol, and inspected source membership. These files are published as separate tables and are never merged into the default cohort or its geometric mean.

The three-route runtime control uses a forward/reverse sequence: PRIK-direct, f2py-direct, PRIK-adapted, followed by PRIK-adapted, f2py-direct, PRIK-direct. Clean-build rounds use the same reversal. This balances every pair instead of always measuring the adapted control last.

To compare existing results without rebuilding:

python3 -m pyperf compare_to \
    results/f2py.json \
    results/prik.json \
    --table

Compare the clean build timings with:

python3 -m pyperf compare_to \
    results/f2py-build.json \
    results/prik-build.json \
    --table

Compare the direct-entrypoint cohort with:

python3 -m pyperf compare_to results/f2py-direct.json results/prik-direct.json --table
python3 -m pyperf compare_to results/prik-adapted.json results/prik-direct.json --table
python3 -m pyperf compare_to results/f2py-direct-build.json results/prik-direct-build.json --table
python3 -m pyperf compare_to results/prik-adapted-build.json results/prik-direct-build.json --table

Results are machine-specific. Compare files produced in the same run; CPU, compiler, Python, and NumPy differences can otherwise dominate small timings. The generated build directories, extensions, and result files are local artifacts rather than repository sources.

Publish a Documentation Snapshot

After a complete paired run on the maintained benchmark runner, refresh the generated sections of the public Performance page and its chart with:

python3 tools/generate_performance_docs.py

Run this command from the repository root. It reads the runtime and build-time pyperf pairs, checks that each pair contains the same benchmarks and compatible platform metadata, records the host operating-system distribution and compiler, and updates only the marked normal, direct, adapter-control, and build result sections in docs/user/performance.md plus the normal runtime and clean-build comparison SVGs in docs/user/assets/. Explanatory prose and the reproduction instructions remain hand-maintained. Do not publish a local sample or revise the default cohort's geometric-mean population. The generator keeps the direct-entrypoint and adapter-control results in their own published sections and requires compatible maintained-run metadata across every cohort.

The Documentation workflow performs the same generation after successful correctness checks and rigorous measurements on pushes to main. The benchmark job uses GitHub's ubuntu-24.04-arm runner, whose standard hosted pool is based on Microsoft Cobalt 100 processors, and verifies that the allocated machine is ARM64 with the expected Neoverse N2/Cobalt 100 CPU part before measuring. The documentation build and deployment remain separate x86-64 jobs. The workflow keeps the raw pyperf files as an artifact and overlays the generated snapshot only in the website build; it does not create a result commit. It also uploads the complete untimed direct-entrypoint preflight directory as a separate artifact so the pinned generated C sources, binding/native objects, linked extensions, and symbol report remain inspectable alongside the published timing tables.

The publication environment pins Python 3.12, NumPy/f2py 2.5.1, pyperf 2.10.0, Meson 1.11.2, Ninja 1.13.0, GNU Fortran 13, and the ARM64 runner label. Update those inputs through a reviewed change so published runs remain comparable.