[ev-br]

Prototyped when working on benchx-dev/benchx#96 : tweak the collectors/providers to be more careful about checking numpy facts.

Claude-written explainer:

The problem. In a revisions comparison, each side must import its own numpy build. Two
layouts went wrong without any warning:

• Wrong target and one PYTHONPATH: the provider reports the build root while one
  PYTHONPATH is exported, so both sides measure the same build.
• Same-version numpy elsewhere: in numpy-spin, a numpy of the same version sits
  outside $BUILD, so the version check passes.

numpy-revisions could not simply print the numpy version from its collector, because
benchx matches collector facts between the two sides, including a digest of the whole
output. A version differs by design, so sides-agree would fail.

The change. The collector now prints a boolean, imported_from_target: does the
imported numpy's file lie under the target directory (the collector's working
directory)? A correct setup gives true on both sides, so sides-agree has nothing to
object to. A wrong one gives false, and the run is refused. Per file, in both
examples:

File: numpy_facts.py
Change: Computes the boolean, prints it in the facts and in the summary that is stored

with each result, and writes a stderr line when it is false. In numpy-revisions it
already put the target first on sys.path, as the worker does, and still does. In
numpy-spin it adds no sys.path entry, so it genuinely tests that PYTHONPATH leads
into the build.
────────────────────────────────────────
File: provider.py
Change: Removes the boolean from how_built, since it is an observation and not a build

fact, and refuses a target where it is false. In numpy-spin the message names the
path numpy was actually imported from.
────────────────────────────────────────

File: benchx.conf.json
Change: Adds "expect": {"imported_from_target": true} to the collector. It keeps
verify: how_built and on_fail: refuse.
────────────────────────────────────────
File: READMEs
Change: A note in numpy-revisions on what the check does and doesn't do, plus two
edited table cells. One sentence in numpy-spin.

What I tested. I used real spin builds of numpy v2.4.0 and v2.4.3, from clean
checkouts of the branch:

• compare.sh, numpy-revisions: the unchanged and changed examples both exit 0 with NO
  CHANGE DETECTED. Both sides stored the same digest.
• num_threads.sh, numpy-spin: 3 runs stored, all success.
• track.sh, numpy-spin: exit 0, with three real revisions built and benchmarked; the
  deliberate v2.30 fails as intended.
• Build-root layout: with the provider patched to report the build root and one
  exported PYTHONPATH, the run is refused before measuring.
• Numpy copy elsewhere: the original code benchmarked it silently; the changed
  provider refuses it.

Limits

• Emulated, not observed. The collector is a separate process. It cannot see the
  benchmark's real sys.path, so a suite directory shadowing the package would go
  unseen.
• True by construction in numpy-revisions. With the package directory as target, the
  check always passes there. It only matters for providers that report the build root,
  as numpy-spin does.
• Needs collectors (open PR #71). No benchx change was needed.

Small things to know

• Cosmetic nit. The docstring edit in numpy-revisions/provider.py left line 24 at 130
  columns, because I joined my sentence onto the original. It is harmless, and I can
  re-wrap and amend if you want.
• Stale sample output. The numpy-revisions README's sample output was not regenerated.
  With the UC-02 branch of benchx the unit prints as bench_matmul
  [OPENBLAS_NUM_THREADS=1].