ENH: record a suite outside the subject's checkout as its own benchmark source by ev-br · Pull Request #70 · benchx-dev/benchx

ev-br · 2026-10-20T16:36:34Z

One more piece from my dogfooding numpy benchmarks: the subject is numpy, and benchmarks are out-of-tree. For the moment, they live in a branch of benchx/prototype/examples, and are uncommitted. With this PR, the relevant parts of the results JSON read

 "project": "numpy",
 "source": {
  "uri": "https://github.com/numpy/numpy.git",
  "type": "git"
 },
 "revision": {
  "key": "6f721564ad94da4ba2138d2b216cd43c38ff172d"    # this is the numpy git hash
 },
 "benchmark": {
  "source": {
   "uri": "ssh://git@github.com/~/ev-br/benchx.git",
   "type": "git"
  },
  "revision": {
   "key": "5cb9142e4b4a62c98562aa0188d6428b0043c9f4"     # this is the benchx git hash
  }

and

 "provenance": {
  "subject_dirty": "clean",     # correct, a clean checkout of numpy
  "benchmark_dirty": "dirty",   # correct, benchx is dirty because the benchmark itself is not committed

A pyperf suite that lies outside the subject's source tree is the benchmark's own code. Its identity is the suite file, not its directory: in a git checkout the source is that repository and its HEAD, with the file's blob id as the tree (clean when it equals HEAD's version); outside git the source is the file:// URI of its directory, with no revision, the blob id as the tree, and dirty.

The work order carries benchmark (its revision is now optional), and the runner finds the suite's identity itself and refuses an order that disagrees. A Google Benchmark suite is a binary built from the subject, so it is the subject's own, as before.

closes #48

ev-br · 2026-10-21T13:50:56Z

Pushed a demo to its own repository: https://github.com/ev-br/benchx_user

ev-br · 2026-10-29T12:34:27Z

Mind taking a quick look @MaanasArora ? This is a small addition to allow out-of-tree benchmark suites: basically, the goal is to be able to record the benchmarking suite (local or a repository) separately from the project being benchmarked.

MaanasArora

Sorry for the delay @ev-br, this looks great thanks! It's a good idea to do this, for both tracked and ad-hoc use cases.

My main comment is that we correctly identify unversioned benchmarks as dirty now, so revisions doesn't seem to work if they're outside. I think it'll be a (different) small change in compare.py though, so happy for it to be follow-up!

MaanasArora · 2026-11-01T01:29:41Z

 ### 3.3 Benchmark suite location (addendum)

-§3's field table and §3.1 name only the subject's target. Nothing in the original draft says where the *benchmark* suite's own code lives, yet every non-thin result requires `benchmark.source` and `benchmark.revision` (`benchmark-result-schema.md` §4.1, §5.2) alongside `provenance.benchmark_dirty`/`benchmark_tree`. This is a gap in the original design, closed here: an optional `benchmark` object, `{source, revision}`, mirroring `sourceRef` and a bare revision key the way `target.kind: revision` does. When present, the runner records that revision the same way it records `target`'s, from the benchmark checkout it is pointed at; it does not fetch it. When absent, the benchmarks are assumed to live in the same checkout as `target`, and the runner reuses `target`'s resolved source, revision, dirty flag, and tree for `benchmark.*` and `provenance.benchmark_dirty`/`benchmark_tree` — the common case where a project's benchmarks live alongside its own code. A benchmark suite with no version control at all (an installed, unversioned directory) is not covered by this addendum and remains open.
+§3's field table and §3.1 name only the subject's target. Nothing in the original draft says where the *benchmark* suite's own code lives, yet every non-thin result requires `benchmark.source` and `benchmark.revision` (`benchmark-result-schema.md` §4.1, §5.2) alongside `provenance.benchmark_dirty`/`benchmark_tree`. This is a gap in the original design, closed here: an optional `benchmark` object, `{source, revision}`, mirroring `sourceRef` and a bare revision key the way `target.kind: revision` does. When present, the runner records that revision the same way it records `target`'s, from the benchmark checkout it is pointed at; it does not fetch it. When absent, the benchmarks are assumed to live in the same checkout as `target`, and the runner reuses `target`'s resolved source, revision, dirty flag, and tree for `benchmark.*` and `provenance.benchmark_dirty`/`benchmark_tree` — the common case where a project's benchmarks live alongside its own code. A benchmark suite with no version control is covered for a single file: `benchmark.revision` is then absent, `benchmark.source` is the `file://` URI of the file's directory (`type: directory`), `provenance.benchmark_tree` is the file's git blob id (`git hash-object FILE`), and `benchmark_dirty` is `dirty`. The file, not its directory, is the unit hashed, so what it imports is not covered. A suite inside another git repository takes that repository's source and HEAD as its revision, with the file's own blob id as the tree and `clean` when the file equals HEAD's version. The prototype applies this to suites that are source files (pyperf); a suite that is a built binary (Google Benchmark) is the subject's own. The runner finds the suite's identity itself, whatever the order says, and refuses an order whose `benchmark` disagrees with it. A `file://` source names a place on one machine, so such a series does not follow the suite to another machine or folder (the prototype's identity keeps the benchmark's source URI).


Nit, but seems Claude is memorializing an "original draft", if we don't need it maybe we can retire its memory here :)

MaanasArora · 2026-11-01T01:31:57Z


 NAME = "pyperf"
+# The suite is a Python source file, which may live outside the subject's checkout.
+SUITE_IS_SOURCE = True


Just a note, we would also need this for asv after rebase.

MaanasArora · 2026-11-01T02:09:20Z

                "ended_at": _timestamp(started + duration),
                "subject_dirty": source["dirty"],
-                "benchmark_dirty": source["dirty"],
+                "benchmark_dirty": benchmark["dirty"] if benchmark else source["dirty"],


I think right now compare.py matches for clean trees, so I think if we have an unversioned benchmark file we can't run it against revisions. I think it might just be a missing update from #83? Either way, happy to do that in follow-up too if it's tricky!

Agreed it's gh-83. It touches multiple places, so I'd prefer to get to it once in all adapters in gh-83 to keep the scope manageable.

MaanasArora · 2026-11-01T02:16:39Z

+    suite = Path(suite_file).resolve()
+    if suite.is_relative_to(Path(subject_source).resolve()):
+        return None
+    tree = blob_id(suite)


My Claude noticed this could be too strict about line-endings (on Windows for example where the git default is CRLF). It suggests we use git hash-object instead of the blob, but I haven't looked too much into it, maybe we can look at it down the line!

Oh let's punt on Windows for the moment? There are multiple places which assume POSIX, let's turn windows testing and make it run in one go separately.

…rk source A pyperf suite that lies outside the subject's source tree is the benchmark's own code. Its identity is the suite file, not its directory: in a git checkout the source is that repository and its HEAD, with the file's blob id as the tree (clean when it equals HEAD's version); outside git the source is the file:// URI of its directory, with no revision, the blob id as the tree, and dirty. The work order carries `benchmark` (its revision is now optional), and the runner finds the suite's identity itself and refuses an order that disagrees. A Google Benchmark suite is a binary built from the subject, so it is the subject's own, as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ev-br · 2026-11-01T11:09:20Z

Thanks for the review @MaanasArora ! Merged this per your approval, with fixes to Claude's memorializations and setting the out-of-tree flag for asv suites.
Windows and dirty benchmarks for revisions require multiple changes, so I did not include either of them here. The latter is gh-80.

MaanasArora previously approved these changes Nov 1, 2026

View reviewed changes

ev-br dismissed MaanasArora’s stale review via 748857a November 1, 2026 10:46

ev-br force-pushed the out-of-tree-benchmarks branch from cff9c6e to 748857a Compare November 1, 2026 10:46

TST: fix a test after the rebase

ea3c306

ev-br merged commit 5f17bba into benchx-dev:main Nov 1, 2026
6 checks passed

Navigation Menu

Repository navigation

ENH: record a suite outside the subject's checkout as its own benchmark source - #70

ENH: record a suite outside the subject's checkout as its own benchmark source#70
ev-br merged 2 commits into
benchx-dev:mainfrom
ev-br:out-of-tree-benchmarks

ev-br commented Oct 20, 2026 •

edited

Loading

Uh oh!

ev-br commented Oct 21, 2026

Uh oh!

ev-br commented Oct 29, 2026

Uh oh!

MaanasArora left a comment

Uh oh!

MaanasArora Nov 1, 2026

Uh oh!

MaanasArora Nov 1, 2026

Uh oh!

MaanasArora Nov 1, 2026

Uh oh!

ev-br Nov 1, 2026

Uh oh!

MaanasArora Nov 1, 2026

Uh oh!

ev-br Nov 1, 2026

Uh oh!

Uh oh!

ev-br commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ev-br commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ev-br commented Oct 21, 2026

Uh oh!

ev-br commented Oct 29, 2026

Uh oh!

MaanasArora left a comment

Choose a reason for hiding this comment

Uh oh!

MaanasArora Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

MaanasArora Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

MaanasArora Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

ev-br Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

MaanasArora Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

ev-br Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ev-br commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ev-br commented Oct 20, 2026 •

edited

Loading