diff --git a/docs/design/prototype-design.md b/docs/design/prototype-design.md index 52a8481..d40dfed 100644 --- a/docs/design/prototype-design.md +++ b/docs/design/prototype-design.md @@ -277,7 +277,10 @@ landing where its §3.4 puts it: neither locates a checkout the runner refuses the order, because a result must state its revision (`runner.md` §2), and `runner-schema.md` §5.1 refuses an order with no revision to record. The benchmark code is the subject's own - checkout, so the benchmark revision, dirty flag, and tree are the subject's + checkout, so the benchmark revision, dirty flag, and tree are the subject's, + unless the harness's suite is a source file (pyperf) outside that checkout: then + the order carries `benchmark`, and the result records the suite file's own + source, revision (when under git) and blob id as `benchmark_tree` (`runner-schema.md` §3.3). - **Build configuration**, from a built-in plugin that reads `CMakeCache.txt` in the build directory: build type, compiler and version, diff --git a/docs/design/runner-schema.md b/docs/design/runner-schema.md index 907e0b4..19563e9 100644 --- a/docs/design/runner-schema.md +++ b/docs/design/runner-schema.md @@ -89,7 +89,7 @@ Where results are delivered (store URL, local file) is an invocation argument, n ### 3.3 Benchmark suite location (addendum) -§3's field table and §3.1 name only the subject's target. Nothing in the original draft says where the *benchmark* suite's own code lives, yet every non-thin result requires `benchmark.source` and `benchmark.revision` (`benchmark-result-schema.md` §4.1, §5.2) alongside `provenance.benchmark_dirty`/`benchmark_tree`. This is a gap in the original design, closed here: an optional `benchmark` object, `{source, revision}`, mirroring `sourceRef` and a bare revision key the way `target.kind: revision` does. When present, the runner records that revision the same way it records `target`'s, from the benchmark checkout it is pointed at; it does not fetch it. When absent, the benchmarks are assumed to live in the same checkout as `target`, and the runner reuses `target`'s resolved source, revision, dirty flag, and tree for `benchmark.*` and `provenance.benchmark_dirty`/`benchmark_tree` — the common case where a project's benchmarks live alongside its own code. A benchmark suite with no version control at all (an installed, unversioned directory) is not covered by this addendum and remains open. +§3's field table and §3.1 name only the subject's target, yet every non-thin result requires `benchmark.source` and `benchmark.revision` (`benchmark-result-schema.md` §4.1, §5.2) alongside `provenance.benchmark_dirty`/`benchmark_tree`. The order therefore says where the *benchmark* suite's own code lives with an optional `benchmark` object, `{source, revision}`, mirroring `sourceRef` and a bare revision key the way `target.kind: revision` does. When present, the runner records that revision the same way it records `target`'s, from the benchmark checkout it is pointed at; it does not fetch it. When absent, the benchmarks are assumed to live in the same checkout as `target`, and the runner reuses `target`'s resolved source, revision, dirty flag, and tree for `benchmark.*` and `provenance.benchmark_dirty`/`benchmark_tree` — the common case where a project's benchmarks live alongside its own code. A benchmark suite with no version control is covered for a single file: `benchmark.revision` is then absent, `benchmark.source` is the `file://` URI of the file's directory (`type: directory`), `provenance.benchmark_tree` is the file's git blob id (`git hash-object FILE`), and `benchmark_dirty` is `dirty`. The file, not its directory, is the unit hashed, so what it imports is not covered. A suite inside another git repository takes that repository's source and HEAD as its revision, with the file's own blob id as the tree and `clean` when the file equals HEAD's version. The prototype applies this to suites that are source files (pyperf); a suite that is a built binary (Google Benchmark) is the subject's own. The runner finds the suite's identity itself, whatever the order says, and refuses an order whose `benchmark` disagrees with it. A `file://` source names a place on one machine, so such a series does not follow the suite to another machine or folder (the prototype's identity keeps the benchmark's source URI). ## 4. Environment policy @@ -203,7 +203,7 @@ Every `plan` entry ends as exactly one result, using the five statuses the resul Reasons are drawn from a closed list of codes, optionally followed by a hyphen and a detail, so dashboards can group failures by prefix without parsing free text. -**Flagged issue (found while fixing the token-regex bug below).** The result schema's `measurement.reason` pattern is `^[a-z0-9]+([.-][a-z0-9]+)*$` (`schemas/measurement-result/0.1.0/schema.json`) — lowercase alphanumeric segments joined only by `.` or `-`. Two things above violated it and are corrected in this revision: the codes used `_` (now `-`, matching `harness-adapter.md`'s own `harness.no-output`-style reasons), and `policy_unsatisfied:` used `:` as a code/detail separator, which the pattern permits nowhere — no delimiter was ever reserved for it. The fix joins code and detail with `-` instead (`policy-unsatisfied-`), which validates, but weakens the original intent: §7's "closed list of codes, optionally followed by a detail, so dashboards can group failures without parsing free text" assumed an unambiguous boundary between the fixed code and the open check name. Joined by the same character multi-word codes already use, that boundary survives only for a human who knows the closed code list, not for an automated grouping-by-prefix reader. `` must itself be hyphen-safe for the composed reason to validate, which is also why `gpu.clocks_locked` in §4.1 is corrected to `gpu.clocks-locked`. A structured alternative — keeping `policy-unsatisfied` as the whole `reason` and carrying `` in a separate field (a new `constraint`-adjacent field, or `provenance.info`) instead of concatenating it — would preserve the separation properly; that's a result-schema change and is left open rather than decided here. +**Reason format.** The result schema's `measurement.reason` pattern is `^[a-z0-9]+([.-][a-z0-9]+)*$` (`schemas/measurement-result/0.1.0/schema.json`): lowercase alphanumeric segments joined only by `.` or `-`. Codes therefore join words with `-` (matching `harness-adapter.md`'s `harness.no-output`-style reasons), and a code and its detail are joined with `-` too (`policy-unsatisfied-`), since the pattern permits no other separator, such as `:`. `` must itself be hyphen-safe for the composed reason to validate, which is why the check is named `gpu.clocks-locked` in §4.1. The cost is that the boundary between the fixed code and the open check name is not unambiguous: joined by the same character multi-word codes use, it is visible only to a human who knows the closed code list, not to an automated grouping-by-prefix reader. A structured alternative — keeping `policy-unsatisfied` as the whole `reason` and carrying `` in a separate field (a new `constraint`-adjacent field, or `provenance.info`) instead of concatenating it — would preserve the separation properly; that's a result-schema change and is left open. ## 7. Versioning and hashing diff --git a/prototype/benchx/adapters/asv.py b/prototype/benchx/adapters/asv.py index 9100dc8..7573010 100644 --- a/prototype/benchx/adapters/asv.py +++ b/prototype/benchx/adapters/asv.py @@ -63,6 +63,7 @@ NAME = "asv" CONTEXT_KEY = "asv" WORKLOAD_PARAMETERS = False +SUITE_IS_SOURCE = True # the suite is Python source, as pyperf's PRODUCER = {"name": "benchx/asv-adapter", "version": "0.2.0", "mapping_version": "asv-on-pyperf/v2"} QUANTITIES = pyperf.QUANTITIES ENV_ALLOWLIST = pyperf.ENV_ALLOWLIST diff --git a/prototype/benchx/adapters/gbench.py b/prototype/benchx/adapters/gbench.py index 586ec3d..61efdef 100644 --- a/prototype/benchx/adapters/gbench.py +++ b/prototype/benchx/adapters/gbench.py @@ -15,6 +15,8 @@ from .base import Unsupported, run_bounded NAME = "google-benchmark" +# The suite is a binary built from the subject's source, so its benchmarks are the subject's own. +SUITE_IS_SOURCE = False CONTEXT_KEY = "google_benchmark" WORKLOAD_PARAMETERS = False PRODUCER = {"name": "benchx/gbench-adapter", "version": "0.1.0", "mapping_version": "gbench-to-benchx/v1"} diff --git a/prototype/benchx/adapters/pyperf.py b/prototype/benchx/adapters/pyperf.py index 9912451..1e05ce6 100644 --- a/prototype/benchx/adapters/pyperf.py +++ b/prototype/benchx/adapters/pyperf.py @@ -26,6 +26,8 @@ from .base import Unsupported, run_bounded NAME = "pyperf" +# The suite is a Python source file, which may live outside the subject's checkout. +SUITE_IS_SOURCE = True CONTEXT_KEY = "pyperf" WORKLOAD_PARAMETERS = False PRODUCER = {"name": "benchx/pyperf-adapter", "version": "0.1.0", "mapping_version": "pyperf-to-benchx/v1"} diff --git a/prototype/benchx/orders.py b/prototype/benchx/orders.py index 43d6990..b3972f0 100644 --- a/prototype/benchx/orders.py +++ b/prototype/benchx/orders.py @@ -27,7 +27,7 @@ def default_run_key(prefix: str) -> str: def make_order(*, suite: dict, target: dict, precision: dict, run_key: str, quantity: str = "wall-time", case_s: float = CASE_TIMEOUT_S, order_s: float = ORDER_TIMEOUT_S, project=None, labels=None, - environment_variables=None, round_=None, slot=None) -> dict: + environment_variables=None, round_=None, slot=None, benchmark=None) -> dict: """One resolved work order. Optional fields that are None or empty are left out. Raises OrderError when the result is not a valid work order. @@ -57,6 +57,8 @@ def make_order(*, suite: dict, target: dict, precision: dict, run_key: str, quan order["labels"] = dict(labels) if project: order["project"] = project + if benchmark: + order["benchmark"] = benchmark errors = core.check(order, "work-order") if errors: raise OrderError("; ".join(str(error) for error in errors)) diff --git a/prototype/benchx/runner.py b/prototype/benchx/runner.py index 00d408a..244e0cc 100644 --- a/prototype/benchx/runner.py +++ b/prototype/benchx/runner.py @@ -88,8 +88,6 @@ def check(order: dict) -> Iterator[Refusal]: yield Refusal(f"no adapter for {suite['adapter']!r}") if order["target"]["kind"] not in ("build", "working_tree"): yield Refusal(f"only build and working_tree targets are supported; {order['target']['kind']!r} is deferred") - if "benchmark" in order: - yield Refusal("only benchmarks in the subject's own checkout are supported") if "include" in suite or "exclude" in suite: yield Refusal("only a single suites[].filter is supported, not include/exclude") if adapter is not None: @@ -108,6 +106,18 @@ def check(order: dict) -> Iterator[Refusal]: yield from e.problems +def _benchmark_mismatches(stated: dict | None, found: dict | None): + """Refusals when the order's `benchmark` is not what the suite file turns out to be.""" + if stated is None: + return + if found is None: + yield Refusal("the order names a benchmark source, but the suite is in the subject's own checkout") + elif (stated["source"]["uri"], stated.get("revision")) != (found["uri"], found.get("revision")): + yield Refusal(f"the order names benchmark {stated['source']['uri']}" + f"{'@' + stated['revision'] if 'revision' in stated else ''}, " + f"but the suite is in {found['uri']}{'@' + found['revision'] if 'revision' in found else ''}") + + def run(order_path, out_dir) -> dict: """Execute one order; write results and the order into out_dir.""" inspection = core.inspect(Path(order_path), "work-order") @@ -139,6 +149,14 @@ def run(order_path, out_dir) -> dict: if source is None: # runner.md §2: a result that cannot state its revision is a defect. refusals.append(Refusal("cannot locate the checkout the target was built from; set target.source_dir")) + benchmark = None + if source is not None and getattr(adapter, "SUITE_IS_SOURCE", False): + # The suite's own code, when it is not in the subject's checkout (runner-schema.md §3.3): + # found as it is, whatever the order says; an order that says otherwise is refused. + benchmark = snapshot.benchmark_identity(target_path / order["suites"][0]["suite"], source["path"]) + refusals += _benchmark_mismatches(order.get("benchmark"), benchmark) + elif "benchmark" in order: + refusals.append(Refusal(f"{adapter.NAME} suites are the subject's own; the order names a benchmark source")) if refusals: raise Refused(refusals) @@ -228,13 +246,15 @@ def run(order_path, out_dir) -> dict: "started_at": _timestamp(started), "ended_at": _timestamp(started + duration), "subject_dirty": source["dirty"], - "benchmark_dirty": source["dirty"], + "benchmark_dirty": benchmark["dirty"] if benchmark else source["dirty"], "runner": RUNNER, **({} if condition else {"artifacts": artifacts}), "info": info, } if "tree" in source: provenance["subject_tree"] = provenance["benchmark_tree"] = source["tree"] + if benchmark: + provenance["benchmark_tree"] = benchmark["tree"] if "labels" in order: provenance["labels"] = order["labels"] workload = {"name": workload_name} @@ -252,10 +272,13 @@ def run(order_path, out_dir) -> dict: **({"project": order["project"]} if "project" in order else {}), "source": order["target"]["source"], "revision": {"key": source["revision"]}, - "benchmark": {"source": order["target"]["source"], - # An unversioned tree is identified by its hash in provenance, not by a revision. - **({} if source["revision"].startswith("directory-") - else {"revision": {"key": source["revision"]}})}, + "benchmark": ({"source": {"uri": benchmark["uri"], "type": benchmark["type"]}, + **({"revision": {"key": benchmark["revision"]}} if "revision" in benchmark else {})} + if benchmark else + {"source": order["target"]["source"], + # An unversioned tree is identified by its hash in provenance, not by a revision. + **({} if source["revision"].startswith("directory-") + else {"revision": {"key": source["revision"]}})}), "coordinates": { "workload": workload, "subject": subject, diff --git a/prototype/benchx/session.py b/prototype/benchx/session.py index c264adc..a52188b 100644 --- a/prototype/benchx/session.py +++ b/prototype/benchx/session.py @@ -100,9 +100,23 @@ def _side(spec: str, suite: str, operation: str, harness=None, environment_varia if not identity: raise SessionError(f"cannot identify the working tree {source}") return {"kind": kind, "path": path, "source": identity["path"], "identity": identity, "uri": uri, + "benchmark": _benchmark(harness, path, suite, identity["path"]), "text": target_mod.text(description), "labels": dict((description or {}).get("labels", {}))} +def _benchmark(harness, path: Path, suite: str, subject_source: str) -> dict | None: + """The order's `benchmark` entry when the suite's code is not in the subject's + checkout (see snapshot.benchmark_identity), else None. A suite that is a built + binary is the subject's own.""" + if not adapters.get(harness.name).SUITE_IS_SOURCE: + return None + found = snapshot.benchmark_identity(path / suite, subject_source) + if found is None: + return None + return {"source": {"uri": found["uri"], "type": found["type"]}, + **({"revision": found["revision"]} if "revision" in found else {})} + + def _order(side: dict, *, harness, suite, filter, quantity, precision, environment_variables, project, run_key, source_uri, round_=None, slot=None, labels=None, case_s=orders.CASE_TIMEOUT_S, order_s=orders.ORDER_TIMEOUT_S) -> dict: @@ -113,7 +127,8 @@ def _order(side: dict, *, harness, suite, filter, quantity, precision, environme return orders.make_order(suite=harness.suite_entry(suite, filter), target=target, precision=precision, quantity=quantity, run_key=run_key, project=project, labels={**side["labels"], **(labels or {})}, case_s=case_s, order_s=order_s, - environment_variables=environment_variables, round_=round_, slot=slot) + environment_variables=environment_variables, round_=round_, slot=slot, + benchmark=side.get("benchmark")) except orders.OrderError as e: raise SessionError(f"work order{'' if round_ is None else f' for round {round_}'}: {e}") from None @@ -215,6 +230,8 @@ def compare_targets(baseline: str, contender: str, *, profile: str, suite: str, raise SessionError(f"{name} tree {side['source']} is {side['identity']['dirty']}; " "a revisions comparison needs clean trees (UC-03 §10)") source_uri = _source_uri(sides, source_uri) + if len({json.dumps((side.get("benchmark") or {}).get("source")) for side in sides}) != 1: + raise SessionError("the sides' suites come from different sources; give --suite as one absolute path") run_key = run_key or orders.default_run_key("compare") out = Path(out) if out else Path("results") / run_key.replace("/", "_") diff --git a/prototype/benchx/snapshot.py b/prototype/benchx/snapshot.py index 3bcca14..f75395c 100644 --- a/prototype/benchx/snapshot.py +++ b/prototype/benchx/snapshot.py @@ -144,6 +144,39 @@ def source_uri(source_dir) -> str: return Path(source_dir, common).resolve().parent.as_uri() if common else Path(source_dir).resolve().as_uri() +def blob_id(path) -> str | None: + """The git blob id of a file: what `git hash-object FILE` prints.""" + try: + content = Path(path).read_bytes() + except OSError: + return None + return hashlib.sha1(b"blob %d\0" % len(content) + content).hexdigest() + + +def benchmark_identity(suite_file, subject_source) -> dict | None: + """Where the suite's own code lives, when that is not the subject's checkout: + None when the file is inside `subject_source`. + + The file is the unit, not its directory (runner-schema.md §3.3). Its tree id + is its blob id. In a git checkout the source is that repository, with its + HEAD as the revision, and the file is `clean` when it equals HEAD's version + and `dirty` otherwise, including when HEAD has none. Outside git the source + is the file's directory as a `file://` URI, with no revision, and the file is + `dirty`: nothing says it was ever committed. + """ + suite = Path(suite_file).resolve() + if suite.is_relative_to(Path(subject_source).resolve()): + return None + tree = blob_id(suite) + top = _run(["git", "rev-parse", "--show-toplevel"], cwd=suite.parent) + head = top and _run(["git", "rev-parse", "HEAD"], cwd=top) + if head: + committed = _run(["git", "rev-parse", f"HEAD:{suite.relative_to(Path(top).resolve()).as_posix()}"], cwd=top) + return {"uri": source_uri(top), "type": "git", "revision": head, "tree": tree, + "dirty": "clean" if committed == tree else "dirty"} + return {"uri": suite.parent.as_uri(), "type": "directory", "tree": tree, "dirty": "dirty"} + + _SKIPPED_DIRS = {".git", ".benchx", "__pycache__"} diff --git a/prototype/tests/test_benchmark_source.py b/prototype/tests/test_benchmark_source.py new file mode 100644 index 0000000..f07882a --- /dev/null +++ b/prototype/tests/test_benchmark_source.py @@ -0,0 +1,149 @@ +"""A suite outside the subject's checkout is its own source, identified by its file (runner-schema.md §3.3).""" + +import json +import textwrap +from pathlib import Path + +import pytest + +pytest.importorskip("pyperf") + +from benchx import cli, core, runner, snapshot +from benchx.store import Store +from conftest import git + +SUITE = textwrap.dedent(''' + def bench_a(): + sum(range(100)) +''') +RESULTS = Path("results") # the project root is the test directory +FAST = ["--processes", "2", "--values", "2", "--loops", "20"] + + +@pytest.fixture +def subject(tmp): + """The thing measured: a git checkout with no suite of its own.""" + path = tmp / "subject" + path.mkdir() + git(path, "init", "-q") + git(path, "remote", "add", "origin", "https://example.org/subject.git") + (path / "lib.py").write_text("X = 1\n") + git(path, "add", "-A") + git(path, "commit", "-qm", "one") + return path + + +@pytest.fixture +def loose_suite(tmp): + """A suite file in a folder under no version control.""" + path = tmp / "loose" + path.mkdir() + (path / "bench_x.py").write_text(SUITE) + return path / "bench_x.py" + + +@pytest.fixture +def versioned_suite(tmp): + path = tmp / "suites" + path.mkdir() + git(path, "init", "-q") + git(path, "remote", "add", "origin", "https://example.org/suites.git") + (path / "bench_x.py").write_text(SUITE) + git(path, "add", "-A") + git(path, "commit", "-qm", "suite") + return path / "bench_x.py" + + +def bench(subject, suite, run_key, *extra): + assert cli.main(["bench", str(subject), "--adapter", "pyperf", "--suite", str(suite), *FAST, + "--run-key", run_key, *extra]) == cli.EXIT_OK + return json.loads(next(p for p in sorted((RESULTS / run_key / "orders").glob("*.json"))).read_text()) + + +def result(run_key): + path = next(p for p in sorted((RESULTS / run_key).glob("*.json")) if not p.name.startswith("workorder-")) + document = json.loads(path.read_text()) + assert core.check(document, "measurement-result") == [] + return document + + +def test_a_suite_inside_the_subject_is_the_subjects_own(subject): + assert snapshot.benchmark_identity(subject / "lib.py", subject) is None + + +def test_an_unversioned_suite_is_identified_by_its_file(subject, loose_suite): + found = snapshot.benchmark_identity(loose_suite, subject) + assert found == {"uri": loose_suite.parent.as_uri(), "type": "directory", "tree": snapshot.blob_id(loose_suite), + "dirty": "dirty"} + assert found["tree"] == git(loose_suite.parent, "hash-object", loose_suite.name) # git's own blob id + + +def test_a_versioned_suite_is_clean_until_it_is_edited(subject, versioned_suite): + clean = snapshot.benchmark_identity(versioned_suite, subject) + assert (clean["uri"], clean["type"], clean["dirty"]) == ("https://example.org/suites.git", "git", "clean") + assert clean["revision"] == git(versioned_suite.parent, "rev-parse", "HEAD") + versioned_suite.write_text(SUITE + "\n# edited\n") + edited = snapshot.benchmark_identity(versioned_suite, subject) + assert (edited["revision"], edited["dirty"]) == (clean["revision"], "dirty") + assert edited["tree"] != clean["tree"] + + +def test_a_new_file_in_a_repository_is_dirty(subject, versioned_suite): + new = versioned_suite.parent / "bench_new.py" + new.write_text(SUITE) + assert snapshot.benchmark_identity(new, subject)["dirty"] == "dirty" + + +def test_an_unversioned_suite_is_recorded_without_a_revision(subject, loose_suite): + order = bench(subject, loose_suite, "u1") + assert order["benchmark"] == {"source": {"uri": loose_suite.parent.as_uri(), "type": "directory"}} + assert core.check(order, "work-order") == [] + document = result("u1") + assert document["benchmark"] == {"source": {"uri": loose_suite.parent.as_uri(), "type": "directory"}} + assert document["source"]["uri"] == "https://example.org/subject.git" # the subject is unchanged + assert document["revision"]["key"] == git(subject, "rev-parse", "HEAD") + provenance = document["provenance"] + assert (provenance["benchmark_dirty"], provenance["benchmark_tree"]) == ("dirty", snapshot.blob_id(loose_suite)) + assert provenance["subject_dirty"] == "clean" and provenance["subject_tree"] != provenance["benchmark_tree"] + + +def test_a_versioned_suite_is_recorded_with_its_own_revision(subject, versioned_suite): + bench(subject, versioned_suite, "v1") + document = result("v1") + assert document["benchmark"]["source"]["uri"] == "https://example.org/suites.git" + assert document["benchmark"]["revision"]["key"] == git(versioned_suite.parent, "rev-parse", "HEAD") + assert document["provenance"]["benchmark_dirty"] == "clean" + assert document["revision"]["key"] == git(subject, "rev-parse", "HEAD") + + +def test_editing_the_suite_changes_its_tree_but_not_the_series(subject, loose_suite): + bench(subject, loose_suite, "e1") + before = result("e1") + loose_suite.write_text(SUITE + "\n# edited\n") + bench(subject, loose_suite, "e2") + after = result("e2") + assert before["provenance"]["benchmark_tree"] != after["provenance"]["benchmark_tree"] + # Each run also records the load average beside the case, a condition measurement and a series of its own. + series = Store().series(quantity="wall-time") + assert len(series) == 1 + assert len(Store().history(series[0]["fingerprint"])) == 2 # both runs are points of it + + +def test_a_suite_inside_the_subject_is_recorded_as_before(subject): + (subject / "bench_in.py").write_text(SUITE) + git(subject, "add", "-A") + git(subject, "commit", "-qm", "suite") + order = bench(subject, "bench_in.py", "i1") + assert "benchmark" not in order + document = result("i1") + assert document["benchmark"] == {"source": document["source"], "revision": document["revision"]} + assert document["provenance"]["benchmark_tree"] == document["provenance"]["subject_tree"] + + +def test_an_order_naming_the_wrong_benchmark_is_refused(subject, loose_suite, tmp): + order = bench(subject, loose_suite, "m1") + order["benchmark"] = {"source": {"uri": "https://example.org/elsewhere.git", "type": "git"}, "revision": "abc"} + path = tmp / "wrong.json" + path.write_bytes(core.pretty(order)) + with pytest.raises(runner.Refused, match="names benchmark https://example.org/elsewhere.git"): + runner.run(path, tmp / "out") diff --git a/schemas/work-order/0.1.0/schema.json b/schemas/work-order/0.1.0/schema.json index 2a9effd..18b0365 100644 --- a/schemas/work-order/0.1.0/schema.json +++ b/schemas/work-order/0.1.0/schema.json @@ -52,10 +52,10 @@ ] }, "benchmark": { - "description": "Where the benchmark suite's own code lives, when different from target (§3.3 addendum). Absent means the benchmarks live in the same checkout as target.", + "description": "Where the benchmark suite's own code lives, when different from target (§3.3 addendum). Absent means the benchmarks live in the same checkout as target. The revision is absent when the suite is not under version control; the runner then identifies the suite file by its content hash.", "type": "object", "additionalProperties": false, - "required": ["source", "revision"], + "required": ["source"], "properties": { "source": { "$ref": "#/$defs/sourceRef" }, "revision": { "$ref": "#/$defs/nonEmptyString" }