diff --git a/docs/design/harness-adapter.md b/docs/design/harness-adapter.md index 22b6708..e37e4f1 100644 --- a/docs/design/harness-adapter.md +++ b/docs/design/harness-adapter.md @@ -132,7 +132,7 @@ A collector is a named command registered for one of two points, `before` or `af {"blas": "openblas-0.3.27", "simd_baseline": ["SSE", "SSE2", "SSE3"]} ``` -The driving half captures that output as a provenance artifact and adds the facts to the context document's `environment.system` under the collector's name, here `environment.system.numpy-build.blas`, with the collector's name as their source. Translation therefore stays repeatable from artifacts, and a collector cannot collide with the snapshot or with another collector. Facts are copied to every result of the invocation they were collected around. A collector that fails, times out, or prints anything else contributes no facts, and those results carry the warning `collector-failed`; nothing is substituted and the run continues. +The driving half captures that output as a provenance artifact and adds the facts to the context document's `environment.system` under the collector's name, here `environment.system.numpy-build.blas`, with the collector's name as their source. Translation therefore stays repeatable from artifacts, and a collector cannot collide with the snapshot or with another collector. Facts are copied to every result of the invocation they were collected around. A collector may also be held to declared facts (`verify`, `expect`) and refuse the order when they differ; the prototype implements this beyond the design's "nothing checks one against the other" (`prototype-design.md` §3). A collector that fails, times out, or prints anything else contributes no facts, and those results carry the warning `collector-failed`; nothing is substituted and the run continues. **Setup file.** The setup file is a channel for declared facts (`benchmark-environments.md` §3.2), next to the work order and project configuration, for the case where the party that prepares an environment is not the party that runs benchmarks in it. That party writes what it knows into `.benchx/setup.json` at the root of the benchmark suite, as one JSON object of facts in the same shape: diff --git a/docs/design/prototype-design.md b/docs/design/prototype-design.md index 6ae9b60..02f1aa1 100644 --- a/docs/design/prototype-design.md +++ b/docs/design/prototype-design.md @@ -80,7 +80,7 @@ a TARGET ending in `.py` selects it. walking up from the working directory; its directory is the root, and with no file the root is the working directory (`schemas/config/0.1.0`). It holds `project`, `source.uri`, `provider.command`, an `adapter` block (`name`, `suite`, `quantities`, `precision`), -`environment_variables`, `timeouts`, `paths.store` and `paths.results` (default +`environment_variables`, `collectors`, `timeouts`, `paths.store` and `paths.results` (default `store.parquet` and `results/` in the root, and relative to the root), and `compare` (`profile`, `rounds`, `k`, `min_rounds`). A command-line argument wins; if it is missing the value is read from the file; if that is missing too, a built-in default applies, where one @@ -321,6 +321,24 @@ the same way report the same configuration. Nobody declared these, so the detect (`harness-adapter.md` §5.1), for what the cache cannot tell, such as how a dependency was installed. The prototype has no placement rules, so they land in `environment.system.setup`, which the default identity policy matches. +- **Context collectors** (`harness-adapter.md` §5.1), named under `collectors` in + `benchx.conf.json` and copied into the work order. The runner runs each command once, + before the cases, in the harness's working directory and environment (`{root}` in an + argument is the project root), under `timeout_s` (default 60). It prints one JSON object. + The runner saves it whole as an artifact, and puts its `summary` object and its + sha256 into every result's `environment.system.`. A collector that fails adds no + facts and gives the results `collector-failed`. Only `before`, once per order, is + implemented. The commands are executed, so `bx run` refuses an order that has + collectors unless given `--allow-collectors`; `bx bench` and `bx compare` build the + order from the user's own configuration and allow them. +- **Verification** of a collector's facts: `verify: "how_built"` requires every key of the + target's `how_built`, when that is an object (a provider may declare facts there; as a + string it is text and never checked), to be among the collected facts with exactly the + same value, at any depth; extra collected keys are fine. `expect` requires facts by + value. A failed check, or a collector that cannot run, refuses the order before any case + (`on_fail: refuse`, the default), or with `on_fail: warn` gives every result the + warning `collector-mismatch`. `prototype/examples/numpy-bench` uses this to check that + the numpy a provider described is the numpy the workers import. `benchmark-environments.md` §3.4 asks that every fact carry its source (`detected`, `declared`, or a plugin name). The prototype does not record it diff --git a/docs/design/prototype-scope.md b/docs/design/prototype-scope.md index 4e5e517..fab7a2a 100644 --- a/docs/design/prototype-scope.md +++ b/docs/design/prototype-scope.md @@ -40,7 +40,7 @@ one run: | Component | In prototype | Deferred | Out of scope | |---|---|---|---| -| Runner | execute a `build` or `working_tree` target per a work order; apply the order's `environment_variables` to the launched process; capture the zero-configuration snapshot, the build configuration, and source identity; account for planned cases; refuse an order it cannot apply exactly, including one that names a deferred field | `python_env` targets, context collectors beyond the built-in ones, environment policy (enforcing launch settings, verifying requested hardware, refusing an unsatisfiable run) | building revisions, tuning the machine (`benchmark-environments.md` §2.3) | +| Runner | execute a `build` or `working_tree` target per a work order; apply the order's `environment_variables` to the launched process; capture the zero-configuration snapshot, the build configuration, and source identity; account for planned cases; refuse an order it cannot apply exactly, including one that names a deferred field | `python_env` targets, context collectors that run after the harness or per case (the prototype runs them once per order, before the cases), environment policy (enforcing launch settings, verifying requested hardware, refusing an unsatisfiable run) | building revisions, tuning the machine (`benchmark-environments.md` §2.3) | | Adapter | Google Benchmark, both halves: drive filter and repetitions, translate output; pyperf, both halves, for Python functions in a working tree, through `bx bench` and `bx compare` (the default adapter of both) | all other harnesses; pyperf's `tags`, `date`, per-run load and frequency, and `pyperf check` | — | | Store | one Parquet file; ingest per schema §5.4 (validated and durable, auto-create, unit-conflict rejection, idempotency, attempt integrity); one default identity policy; queries: list series, series history, get result | continuity events, quarantine, artifact storage, identity-policy authoring | — | | Comparator | run mode with the `revisions` and `environments` profiles; eligibility guard; comparison document | history mode; `single-run`, `variants`, and `cross-machine` profiles; batch requests; provisional noise rules | — | diff --git a/prototype/benchx/cli.py b/prototype/benchx/cli.py index 466ff76..933c149 100644 --- a/prototype/benchx/cli.py +++ b/prototype/benchx/cli.py @@ -55,7 +55,7 @@ def cmd_run(args): out = Path(args.out) if args.out else cfg.results_dir snapshot.ignore(cfg.store_path, cfg.results_dir, out) try: - summary = runner.run(args.order, out) + summary = runner.run(args.order, out, allow_collectors=args.allow_collectors) except runner.Refused as e: return _refused(e) print(f"{summary['results']} result(s) for {summary['cases']} case(s) -> {summary['out']}" @@ -127,6 +127,7 @@ def _common(cfg, args, *, forced_adapter=None, forced_suite=None) -> dict: "source_uri": args.source_uri or cfg.section("source").get("uri"), "build": not args.no_build, "provider": cfg.provider, + "collectors": cfg.collectors, "results_dir": cfg.results_dir, # An asv case is every round of its benchmark, each bounded by the benchmark's own timeout. "case_s": cfg.pick(getattr(args, "case_timeout", None), timeouts.get("case_s"), @@ -465,6 +466,8 @@ def main(argv=None): p.add_argument("--out", help="directory for result files (default ./results)") p.add_argument("--no-ingest", action="store_true", help="only write the result files; do not deliver them to the local store") + p.add_argument("--allow-collectors", action="store_true", + help="run the order's context collectors: commands the order names (otherwise refused)") p.set_defaults(fn=cmd_run) p = sub.add_parser("bench", help="benchmark one target: the single-target form of bx compare", diff --git a/prototype/benchx/collectors.py b/prototype/benchx/collectors.py new file mode 100644 index 0000000..3b9f69d --- /dev/null +++ b/prototype/benchx/collectors.py @@ -0,0 +1,113 @@ +"""Context collectors (harness-adapter.md §5.1), and the check against declared facts. + +A collector is a command the order names. The runner runs it once, before the +cases, in the harness's working directory and environment, outside any timed +region and under a time limit. It prints one JSON object of facts. The facts go +to the artifacts in full, and to every result's `environment.system.` as their +`summary` object (if they have one) and their sha256. + +A collector that fails, times out, or prints anything else adds no facts and +gives the results the warning `collector-failed`; nothing is substituted. + +A collector may also be held to what was declared: `verify: how_built` requires +every key of the target's `how_built` object to be among the collected facts +with exactly the same value, and `expect` requires facts by value. A failed +check refuses the order, or with `on_fail: warn` gives the results the warning +`collector-mismatch`. A collector that cannot run cannot verify, so under +`refuse` that refuses too. +""" + +import json +import subprocess +from dataclasses import dataclass, field +from pathlib import Path + +from . import core +from .core import Refusal, loader + +DEFAULT_TIMEOUT_S = 60.0 +MAX_DIFFS = 5 + + +@dataclass +class Collected: + system: dict = field(default_factory=dict) # name -> what goes into environment.system + artifacts: list = field(default_factory=list) # (kind, media_type, path) + warnings: list = field(default_factory=list) + refusals: list = field(default_factory=list) + + +def _differences(wanted, found, path=""): + """The paths at which `found` does not contain `wanted` exactly. A mapping is + compared key by key (extra keys in `found` are fine); anything else by its + canonical JSON, so `true` is not `1` as Python's `==` would have it.""" + if isinstance(wanted, dict): + if not isinstance(found, dict): + yield f"{path or '.'}: expected an object, found {_short(found)}" + return + for key, value in wanted.items(): + if key not in found: + yield f"{path}/{key}: missing (declared {_short(value)})" + else: + yield from _differences(value, found[key], f"{path}/{key}") + elif core.canonical(wanted) != core.canonical(found): + yield f"{path or '.'}: declared {_short(wanted)}, observed {_short(found)}" + + +def _short(value, limit=80) -> str: + text = json.dumps(value, sort_keys=True) + return text if len(text) <= limit else text[:limit] + "..." + + +def _collect(spec, cwd, env) -> tuple[dict | None, str | None]: + """The facts a collector prints, or (None, why not).""" + try: + done = subprocess.run(spec["command"], cwd=cwd, env=env, capture_output=True, text=True, + timeout=spec.get("timeout_s", DEFAULT_TIMEOUT_S)) + except OSError as e: + return None, f"cannot run {spec['command'][0]!r}: {e}" + except subprocess.TimeoutExpired: + return None, f"timed out after {spec.get('timeout_s', DEFAULT_TIMEOUT_S):g} s" + if done.returncode != 0: + last = (done.stderr.strip().splitlines() or ["no message"])[-1] + return None, f"exit {done.returncode}: {last}" + facts, errors = loader.parse(done.stdout) + if facts is None: + return None, "; ".join(str(error) for error in errors) + return facts, None + + +def run_all(specs, *, declared, cwd, env, artifacts_dir, taken=()) -> Collected: + """Run each collector. `declared` is the target's how_built, `taken` the + environment.system names already in use.""" + out = Collected() + for spec in specs: + name = spec["name"] + if name in taken or name in out.system: + out.refusals.append(Refusal(f"collector {name!r}: that name is already in environment.system")) + continue + checks = bool(spec.get("verify") or spec.get("expect")) + refuse = spec.get("on_fail", "refuse") == "refuse" + facts, why = _collect(spec, cwd, env) + problems = [] + if facts is None: + out.warnings.append("collector-failed") + problems.append(f"collector {name!r} failed: {why}") + else: + digest = core.sha256_hex(core.canonical(facts)) + path = Path(artifacts_dir) / f"collector-{name}.json" + path.write_bytes(core.pretty(facts)) + out.artifacts.append(("collector-output", "application/json", path)) + summary = facts.get("summary") if isinstance(facts.get("summary"), dict) else {} + out.system[name] = {**summary, "sha256": digest} + if spec.get("verify") == "how_built": + if isinstance(declared, dict): + problems += [f"collector {name!r}: how_built {d}" for d in _differences(declared, facts)] + else: + problems.append(f"collector {name!r}: verify: how_built, but the target's how_built is not an object") + problems += [f"collector {name!r}: expect {d}" for d in _differences(spec.get("expect", {}), facts)] + if problems: + out.warnings.append("collector-mismatch") + if problems and checks and refuse: # an unchecked collector that fails only warns + out.refusals += [Refusal(problem) for problem in problems[:MAX_DIFFS]] + return out diff --git a/prototype/benchx/compare.py b/prototype/benchx/compare.py index 8e91f27..9de9e64 100644 --- a/prototype/benchx/compare.py +++ b/prototype/benchx/compare.py @@ -239,6 +239,8 @@ def _declared(document) -> dict: runner saw: the declarations are recorded as given and never judged.""" info = document["provenance"].get("info", {}) side = {k: info[k] for k in ("how_built", "activation", "shell", "target_provider") if k in info} + if isinstance(side.get("how_built"), dict): # facts, shown as the JSON text they are + side["how_built"] = json.dumps(side["how_built"], sort_keys=True, separators=(",", ":")) system = document["coordinates"]["environment"].get("system") or {} if env := (system.get("process") or {}).get("environment"): side["observed_env"] = env diff --git a/prototype/benchx/config.py b/prototype/benchx/config.py index 5b7a79c..4d12347 100644 --- a/prototype/benchx/config.py +++ b/prototype/benchx/config.py @@ -92,6 +92,17 @@ def store_path(self) -> Path: def results_dir(self) -> Path: return self._path(self.section("paths").get("results", DEFAULT_RESULTS)) + @property + def collectors(self) -> list[dict]: + """The context collectors, as a work order lists them. `{root}` in a command + argument is the project root, because a collector runs in the harness's + working directory, not here.""" + out = [] + for name, spec in self.section("collectors").items(): + command = [argument.replace("{root}", str(self.root)) for argument in spec["command"]] + out.append({"name": name, **spec, "command": command}) + return out + @property def provider(self) -> tuple[list[str], Path] | None: """The provider command, and the directory it runs in (the root).""" diff --git a/prototype/benchx/orders.py b/prototype/benchx/orders.py index a8f32a1..070c98c 100644 --- a/prototype/benchx/orders.py +++ b/prototype/benchx/orders.py @@ -27,7 +27,8 @@ def default_run_key(prefix: str) -> str: def make_order(*, suite: dict, target: dict, precision: dict, run_key: str, quantities=("wall-time",), case_s: float = CASE_TIMEOUT_S, order_s: float = ORDER_TIMEOUT_S, project=None, labels=None, - environment_variables=None, round_=None, slot=None, benchmark=None) -> dict: + environment_variables=None, round_=None, slot=None, benchmark=None, + collectors=None) -> dict: """One resolved work order. Optional fields that are None or empty are left out. Raises OrderError when the result is not a valid work order. @@ -59,6 +60,8 @@ def make_order(*, suite: dict, target: dict, precision: dict, run_key: str, quan order["project"] = project if benchmark: order["benchmark"] = benchmark + if collectors: + order["collectors"] = list(collectors) errors = core.check(order, "work-order") if errors: raise OrderError("; ".join(str(error) for error in errors)) diff --git a/prototype/benchx/runner.py b/prototype/benchx/runner.py index 244e0cc..dd3c86c 100644 --- a/prototype/benchx/runner.py +++ b/prototype/benchx/runner.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone from pathlib import Path -from . import __version__, adapters, core, snapshot +from . import __version__, adapters, collectors, core, snapshot from . import target as target_mod from .adapters.base import Unsupported from .core import Refusal, Rejected @@ -118,8 +118,10 @@ def _benchmark_mismatches(stated: dict | None, found: dict | None): f"but the suite is in {found['uri']}{'@' + found['revision'] if 'revision' in found else ''}") -def run(order_path, out_dir) -> dict: - """Execute one order; write results and the order into out_dir.""" +def run(order_path, out_dir, allow_collectors=False) -> dict: + """Execute one order; write results and the order into out_dir. The order's + context collectors are commands the runner would execute, so they run only + if `allow_collectors`.""" inspection = core.inspect(Path(order_path), "work-order") if inspection.data is None: raise Refused(inspection.errors) @@ -127,6 +129,9 @@ def run(order_path, out_dir) -> dict: refusals = list(check(order)) if refusals: raise Refused(refusals) + if order.get("collectors") and not allow_collectors: + raise Refused([Refusal("the order names context collectors, which are commands to run; " + "pass --allow-collectors to run it")]) adapter = adapters.get(order["suites"][0]["adapter"]) applied_protocol, invocation = adapter.protocol(order["precision"]) @@ -177,6 +182,11 @@ def run(order_path, out_dir) -> dict: slot, round_ = order.get("slot"), order.get("round") artifacts_dir = out_dir / "artifacts" / (f"slot-{slot}" if slot is not None else ref[7:19]) artifacts_dir.mkdir(parents=True, exist_ok=True) + collected = collectors.run_all(order.get("collectors", []), declared=order["target"].get("how_built"), + cwd=target_path, env=child_env, artifacts_dir=artifacts_dir, taken=system) + if collected.refusals: + raise Refused(collected.refusals) + system.update(collected.system) order_file = out_dir / f"workorder-{ref[7:]}.json" order_file.write_bytes(core.pretty(order)) @@ -209,6 +219,7 @@ def run(order_path, out_dir) -> dict: if native_path.exists(): artifacts.insert(0, _artifact("native-output", "application/json", native_path)) artifacts += [_artifact(kind, media_type, path) for kind, media_type, path in run_.get("artifacts", [])] + artifacts += [_artifact(kind, media_type, path) for kind, media_type, path in collected.artifacts] native_system, native_info = adapter.context_facts(run_["native"]) case_system, case_warnings = snapshot.merge(system, native_system), [] if hasattr(adapter, "observed_environment"): @@ -290,7 +301,7 @@ def run(order_path, out_dir) -> dict: "procedure": procedure, "provenance": provenance, } - warnings = [] if condition else ([setup_warning] if setup_warning else []) + case_warnings + warnings = [] if condition else ([setup_warning] if setup_warning else []) + collected.warnings + case_warnings if warnings: document["quality"] = {"warnings": warnings} errors = core.check(document, "measurement-result") diff --git a/prototype/benchx/session.py b/prototype/benchx/session.py index a123549..45cd90b 100644 --- a/prototype/benchx/session.py +++ b/prototype/benchx/session.py @@ -117,7 +117,7 @@ def _benchmark(harness, path: Path, suite: str, subject_source: str) -> dict | N def _order(side: dict, *, harness, suite, filter, quantities, precision, environment_variables, project, run_key, source_uri, round_=None, slot=None, labels=None, case_s=orders.CASE_TIMEOUT_S, - order_s=orders.ORDER_TIMEOUT_S) -> dict: + order_s=orders.ORDER_TIMEOUT_S, collectors=None) -> dict: """One side's order; for one round and slot of a comparison, or neither (`bx bench`).""" target = {"kind": side["kind"], "path": str(side["path"]), "source_dir": side["source"], "source": {"uri": source_uri, "type": "git"}, **side["text"]} @@ -126,7 +126,7 @@ def _order(side: dict, *, harness, suite, filter, quantities, precision, environ quantities=quantities, run_key=run_key, project=project, labels={**side["labels"], **(labels or {})}, case_s=case_s, order_s=order_s, environment_variables=environment_variables, round_=round_, slot=slot, - benchmark=side.get("benchmark")) + benchmark=side.get("benchmark"), collectors=collectors) except orders.OrderError as e: raise SessionError(f"work order{'' if round_ is None else f' for round {round_}'}: {e}") from None @@ -168,14 +168,16 @@ def _run_side(side: dict, path: Path, store=None, **order) -> tuple[dict, dict | store's outcome (None without a store).""" path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(_order(side, **order), indent=1) + "\n") - summary = runner.run(path, path.parent.parent) + # The order is this session's own, built from the user's configuration: its collectors may run. + summary = runner.run(path, path.parent.parent, allow_collectors=True) return summary, store.ingest_paths(summary["files"]) if store else None def bench_target(target: str, *, suite: str, adapter: str, store=None, flags=None, file_precision=None, environment_variables=None, filter=None, quantities=("wall-time",), project=None, run_key=None, out=None, source_uri=None, case_s=orders.CASE_TIMEOUT_S, order_s=orders.ORDER_TIMEOUT_S, - progress=None, build=True, provider=None, config_path=None, results_dir=None) -> dict: + progress=None, build=True, provider=None, config_path=None, results_dir=None, + collectors=None) -> dict: """Run one target once: a comparison session of one order, with no round, slot or comparison. Results are delivered to `store` unless it is None. @@ -196,7 +198,7 @@ def bench_target(target: str, *, suite: str, adapter: str, store=None, flags=Non summary, outcome = _run_side(side, out / "orders" / "order.json", store, harness=harness, suite=suite, filter=filter, quantities=quantities, precision=precision, environment_variables=environment_variables, project=project, run_key=run_key, - source_uri=source_uri, case_s=case_s, order_s=order_s) + source_uri=source_uri, case_s=case_s, order_s=order_s, collectors=collectors) return {**summary, "outcome": outcome, "precision": precision} @@ -204,7 +206,8 @@ def compare_targets(baseline: str, contender: str, *, profile: str, suite: str, adapter, flags=None, file_precision=None, environment_variables=None, filter=None, quantities=("wall-time",), project=None, label=None, run_key=None, out=None, source_uri=None, k=3.0, min_rounds=3, progress=None, build=True, provider=None, results_dir=None, - case_s=orders.CASE_TIMEOUT_S, order_s=orders.ORDER_TIMEOUT_S, config_path=None) -> dict: + case_s=orders.CASE_TIMEOUT_S, order_s=orders.ORDER_TIMEOUT_S, config_path=None, + collectors=None) -> dict: """Alternate baseline and contender over `rounds` rounds, deliver every result to `store`, and return the run-mode comparison document. @@ -254,7 +257,8 @@ def compare_targets(baseline: str, contender: str, *, profile: str, suite: str, harness=harness, suite=suite, filter=filter, quantities=quantities, precision=precision, environment_variables=environment_variables, project=project, run_key=run_key, source_uri=source_uri, - round_=round_, slot=slot, labels=side_labels, case_s=case_s, order_s=order_s) + round_=round_, slot=slot, labels=side_labels, case_s=case_s, order_s=order_s, + collectors=collectors) if outcome["rejected"]: codes = sorted({r["code"] for r in outcome["rejected"]}) raise SessionError(f"the store rejected results of round {round_} ({name}): {', '.join(codes)}") diff --git a/prototype/benchx/target.py b/prototype/benchx/target.py index 9cc8eeb..375368e 100644 --- a/prototype/benchx/target.py +++ b/prototype/benchx/target.py @@ -25,6 +25,7 @@ SIDECAR = ".benchx-target.json" TEXT_FIELDS = ("how_built", "activation", "shell", "provider") +MAX_FACTS_CHARS = 65536 # how_built as an object, serialized; as text it is limited to 8192 class TargetError(Exception): @@ -78,6 +79,9 @@ def _validated(raw: str, where: str) -> dict: if errors: problems = "; ".join(str(error) for error in errors) raise TargetError(f"{where} is not a target description: {problems}") + facts = description.get("how_built") + if isinstance(facts, dict) and len(json.dumps(facts)) > MAX_FACTS_CHARS: + raise TargetError(f"{where}: how_built facts are over {MAX_FACTS_CHARS} characters as JSON") return description diff --git a/prototype/tests/test_benchmark_source.py b/prototype/tests/test_benchmark_source.py index a29dac3..98358f3 100644 --- a/prototype/tests/test_benchmark_source.py +++ b/prototype/tests/test_benchmark_source.py @@ -61,7 +61,7 @@ def bench(subject, suite, run_key, *extra): def result(run_key): - path = next(p for p in sorted((RESULTS / run_key).glob("*.json")) if not p.name.startswith("workorder-")) + path = next(p for p in sorted((RESULTS / run_key).glob("*_wall-time.json"))) document = json.loads(path.read_text()) assert core.check(document, "measurement-result") == [] return document diff --git a/prototype/tests/test_collectors.py b/prototype/tests/test_collectors.py new file mode 100644 index 0000000..23c33fe --- /dev/null +++ b/prototype/tests/test_collectors.py @@ -0,0 +1,213 @@ +"""Context collectors and the check of a target's declared facts against them (benchx.collectors).""" + +import json +import sys +import textwrap + +import pytest + +from benchx import collectors, compare, config, core, runner +from conftest import git + + +def spec(name="facts", script='print(\'{"a": 1}\')', **extra): + return {"name": name, "command": [sys.executable, "-c", script], **extra} + + +def run_all(specs, tmp, declared=None, taken=()): + (tmp / "artifacts").mkdir(exist_ok=True) + return collectors.run_all(specs, declared=declared, cwd=tmp, env={}, artifacts_dir=tmp / "artifacts", taken=taken) + + +def messages(collected): + return [str(refusal) for refusal in collected.refusals] + + +def test_facts_are_recorded_in_full_and_in_summary(tmp): + script = 'import json; print(json.dumps({"big": [1, 2], "summary": {"version": "9"}}))' + collected = run_all([spec(script=script)], tmp) + assert collected.warnings == [] and collected.refusals == [] + assert collected.system["facts"]["version"] == "9" + assert collected.system["facts"]["sha256"] == core.sha256_hex(core.canonical({"big": [1, 2], "summary": {"version": "9"}})) + ((kind, media_type, path),) = collected.artifacts + assert (kind, media_type) == ("collector-output", "application/json") + assert json.loads(path.read_text())["big"] == [1, 2] + + +@pytest.mark.parametrize("script", ["import sys; sys.exit(3)", "print('not json')", "print('[1, 2]')", + "print('{\"a\": NaN}')", "print('{\"a\": 1e999}')", + "print('{\"a\": 1, \"a\": 2}')", "print('{\"a\": 9007199254740993}')", + "import time; time.sleep(5)"]) +def test_a_failed_collector_adds_no_facts_and_only_warns(tmp, script): + collected = run_all([spec(script=script, timeout_s=1)], tmp) + assert collected.warnings == ["collector-failed"] and collected.system == {} and collected.refusals == [] + + +def test_a_missing_command_is_a_failure(tmp): + collected = run_all([{"name": "x", "command": ["/no/such/collector"]}], tmp) + assert collected.warnings == ["collector-failed"] + + +def test_declared_facts_must_be_observed_exactly_extras_are_fine(tmp): + script = 'import json; print(json.dumps({"a": {"b": 1, "c": [1, 2]}, "extra": 0}))' + ok = run_all([spec(script=script, verify="how_built")], tmp, declared={"a": {"b": 1, "c": [1, 2]}}) + assert ok.refusals == [] and ok.warnings == [] + bad = run_all([spec(script=script, verify="how_built")], tmp, declared={"a": {"b": 2, "c": [1, 2]}, "gone": True}) + assert any("how_built /a/b: declared 2, observed 1" in m for m in messages(bad)) + assert any("how_built /gone: missing" in m for m in messages(bad)) + assert bad.warnings == ["collector-mismatch"] + + +def test_verify_needs_declared_facts(tmp): + collected = run_all([spec(verify="how_built")], tmp, declared="spin build") + assert "not an object" in messages(collected)[0] + + +def test_expect_compares_by_value(tmp): + assert run_all([spec(expect={"a": 1})], tmp).refusals == [] + collected = run_all([spec(expect={"a": 2})], tmp) + assert "expect /a: declared 2, observed 1" in messages(collected)[0] + + +@pytest.mark.parametrize("declared, printed", [(True, "1"), (1, "true"), (False, "0"), ([True], "[1]"), + ({"k": 1}, '{"k": true}')]) +def test_a_bool_is_not_an_int_in_a_comparison(tmp, declared, printed): + script = f"print('{{\"a\": {printed}}}')" + for check in ({"expect": {"a": declared}}, {"verify": "how_built"}): + collected = run_all([spec(script=script, **check)], tmp, declared={"a": declared}) + assert collected.warnings == ["collector-mismatch"] and messages(collected), check + + +def test_equal_values_still_match_whatever_their_spelling(tmp): + script = 'print(\'{"a": [1.0, {"b": true}], "c": "x"}\')' + wanted = {"a": [1, {"b": True}], "c": "x"} + collected = run_all([spec(script=script, expect=wanted, verify="how_built")], tmp, declared=wanted) + assert collected.warnings == [] and collected.refusals == [] + + +def test_warn_runs_on_and_marks_the_results(tmp): + collected = run_all([spec(expect={"a": 2}, on_fail="warn")], tmp) + assert collected.refusals == [] and collected.warnings == ["collector-mismatch"] + assert "facts" in collected.system # the facts are still recorded + + +def test_a_collector_that_cannot_run_cannot_verify(tmp): + broken = spec(script="import sys; sys.exit(1)", verify="how_built") + assert "failed" in messages(run_all([broken], tmp, declared={"a": 1}))[0] + assert run_all([{**broken, "on_fail": "warn"}], tmp, declared={"a": 1}).refusals == [] + + +def test_a_name_may_not_shadow_other_system_facts(tmp): + assert "already in environment.system" in messages(run_all([spec(name="os")], tmp, taken={"os": 1}))[0] + assert "already in environment.system" in messages(run_all([spec(), spec()], tmp))[0] + + +def test_comparison_shows_structured_how_built_as_json_text(): + document = {"coordinates": {"environment": {}}, "provenance": {"info": {"how_built": {"b": 1, "a": {"c": 2}}}}} + assert compare._declared(document)["how_built"] == '{"a":{"c":2},"b":1}' + + +# --- configuration and the runner --------------------------------------------------- + +def write_config(directory, **collectors): + path = directory / config.CONFIG_FILE + path.write_text(json.dumps({"schema_version": "benchx/config/0.1.0", "collectors": collectors})) + return path + + +def test_the_root_is_substituted_in_a_collector_command(tmp): + write_config(tmp, numpy={"command": ["python", "{root}/facts.py"], "verify": "how_built"}) + (entry,) = config.load(tmp).collectors + assert entry == {"name": "numpy", "command": ["python", f"{tmp.resolve()}/facts.py"], "verify": "how_built"} + + +@pytest.mark.parametrize("bad", [{"command": []}, {"command": ["x"], "on_fail": "ignore"}, + {"command": ["x"], "verify": "activation"}, {"command": ["x"], "speed": 1}]) +def test_a_bad_collector_is_refused(tmp, bad): + write_config(tmp, c=bad) + with pytest.raises(config.ConfigError): + config.load(tmp) + + +pytest.importorskip("pyperf") +from benchx import cli # noqa: E402 + +SUITE = "def bench_a():\n sum(range(100))\n" +FAST = ["--processes", "2", "--values", "2", "--loops", "20"] + + +@pytest.fixture +def project(tmp, monkeypatch): + """A subject checkout, a suite and a provider that declares how_built = {"k": 1}.""" + subject = tmp / "subject" + subject.mkdir() + git(subject, "init", "-q") + git(subject, "remote", "add", "origin", "https://example.org/subject.git") + (subject / "lib.py").write_text("X = 1\n") + git(subject, "add", "-A") + git(subject, "commit", "-qm", "one") + root = tmp / "proj" + root.mkdir() + (root / "bench_a.py").write_text(SUITE) + (root / "provider.py").write_text(textwrap.dedent(f''' + import json, sys + sys.stdin.read() + print(json.dumps({{"schema_version": "benchx/target-description/0.1.0", + "target": {{"kind": "build", "path": "{subject}", "source_dir": "{subject}", + "source": {{"uri": "https://example.org/subject.git", "type": "git"}}}}, + "how_built": {{"k": 1}}}})) + ''')) + (root / "facts.py").write_text("import json, os; print(json.dumps({'k': int(os.environ.get('K', 1)), 'summary': {'s': 1}}))\n") + monkeypatch.chdir(root) + return root + + +def configure(root, **collector): + (root / config.CONFIG_FILE).write_text(json.dumps({ + "schema_version": "benchx/config/0.1.0", + "provider": {"command": [sys.executable, "provider.py"]}, + "adapter": {"name": "pyperf", "suite": str(root / "bench_a.py")}, + "collectors": {"facts": {"command": [sys.executable, "{root}/facts.py"], "verify": "how_built", **collector}}})) + + +def result_of(run_key): + from pathlib import Path + path = next(p for p in sorted(Path("results", run_key).glob("*_wall-time.json"))) + return json.loads(path.read_text()) + + +def test_bench_runs_the_collector_and_records_its_facts(project): + configure(project) + assert cli.main(["bench", "@x", *FAST, "--run-key", "c1"]) == cli.EXIT_OK + document = result_of("c1") + assert core.check(document, "measurement-result") == [] + assert document["coordinates"]["environment"]["system"]["facts"]["s"] == 1 and "sha256" in document["coordinates"]["environment"]["system"]["facts"] + assert document["provenance"]["info"]["how_built"] == {"k": 1} + assert "collector-output" in {a["kind"] for a in document["provenance"]["artifacts"]} + assert "quality" not in document + + +def test_a_mismatch_refuses_the_order_before_any_case(project, capsys, monkeypatch): + configure(project) + monkeypatch.setenv("K", "2") + assert cli.main(["bench", "@x", *FAST, "--run-key", "c2"]) == cli.EXIT_REFUSED + assert "how_built /k: declared 1, observed 2" in capsys.readouterr().err + assert not list(project.glob("results/c2/*_wall-time.json")) + + +def test_a_warn_mismatch_runs_and_marks_every_result(project, monkeypatch): + configure(project, on_fail="warn") + monkeypatch.setenv("K", "2") + assert cli.main(["bench", "@x", *FAST, "--run-key", "c3"]) == cli.EXIT_OK + assert result_of("c3")["quality"]["warnings"] == ["collector-mismatch"] + + +def test_bx_run_refuses_an_order_with_collectors_unless_allowed(project, tmp, capsys): + configure(project) + assert cli.main(["bench", "@x", *FAST, "--run-key", "c4", "--no-ingest"]) == cli.EXIT_OK + order = project / "results" / "c4" / "orders" / "order.json" + assert cli.main(["run", str(order), "--out", str(tmp / "o1")]) == cli.EXIT_REFUSED + assert "--allow-collectors" in capsys.readouterr().err + with pytest.raises(runner.Refused): + runner.run(order, tmp / "o2") + assert cli.main(["run", str(order), "--out", str(tmp / "o3"), "--allow-collectors", "--no-ingest"]) == cli.EXIT_OK diff --git a/prototype/tests/test_session.py b/prototype/tests/test_session.py index 761dea3..1af71c4 100644 --- a/prototype/tests/test_session.py +++ b/prototype/tests/test_session.py @@ -71,7 +71,7 @@ def test_session_refusals(tmp, builds, kwargs, message): def test_a_refused_order_stops_the_session(tmp, builds, monkeypatch): - def refuse(*_): + def refuse(*_, **__): raise runner.Refused("deferred") monkeypatch.setattr(runner, "run", refuse) with pytest.raises(runner.Refused): diff --git a/schemas/config/0.1.0/schema.json b/schemas/config/0.1.0/schema.json index 5dc94a5..c39ba9f 100644 --- a/schemas/config/0.1.0/schema.json +++ b/schemas/config/0.1.0/schema.json @@ -52,6 +52,22 @@ "precision": { "description": "A work-order precision, possibly partial.", "type": "object" } } }, + "collectors": { + "description": "Context collectors, by name (harness-adapter.md §5.1), copied into every work order. The commands run in the harness's working directory, so a command argument may contain {root}, replaced by the project root. verify: how_built requires the target's how_built object to be observed exactly; expect requires facts by exact value; on_fail says what a failure does.", + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": ["command"], + "properties": { + "command": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "timeout_s": { "type": "number", "exclusiveMinimum": 0 }, + "verify": { "const": "how_built" }, + "expect": { "type": "object" }, + "on_fail": { "enum": ["refuse", "warn"] } + } + } + }, "environment_variables": { "description": "Set for the benchmark's processes; each becomes a workload parameter.", "type": "object", diff --git a/schemas/work-order/0.1.0/schema.json b/schemas/work-order/0.1.0/schema.json index 18b0365..46d1791 100644 --- a/schemas/work-order/0.1.0/schema.json +++ b/schemas/work-order/0.1.0/schema.json @@ -51,6 +51,32 @@ { "$ref": "#/$defs/targetArtifact" } ] }, + "collectors": { + "description": "Context collectors (harness-adapter.md §5.1): commands the runner runs once, before the cases, in the harness's working directory and environment. Each prints one JSON object of facts, recorded under coordinates.environment.system.. The runner executes these commands, so `bx run` refuses an order that has them unless told to allow it.", + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name", "command"], + "properties": { + "name": { "$ref": "#/$defs/token" }, + "command": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "timeout_s": { "type": "number", "exclusiveMinimum": 0 }, + "verify": { + "description": "how_built: every key of the target's how_built object must be in the collected facts with exactly the same value.", + "const": "how_built" + }, + "expect": { + "description": "Facts the collected output must contain, by exact value.", + "type": "object" + }, + "on_fail": { + "description": "What a failed verify or expect does: refuse the order (the default) or warn in every result.", + "enum": ["refuse", "warn"] + } + } + } + }, "benchmark": { "description": "Where the benchmark suite's own code lives, when different from target (§3.3 addendum). Absent means the benchmarks live in the same checkout as target. The revision is absent when the suite is not under version control; the runner then identifies the suite file by its content hash.", "type": "object", @@ -344,9 +370,11 @@ "type": "object", "properties": { "how_built": { - "description": "What the user did to build the target, such as the cmake commands or the name of the build script.", - "type": ["string", "null"], - "maxLength": 8192 + "description": "What the user did to build the target, such as the cmake commands or the name of the build script; or, as an object, structured facts about the build that a collector with verify: how_built must observe exactly (an object is never parsed from text). A string is never checked.", + "oneOf": [ + { "type": ["string", "null"], "maxLength": 8192 }, + { "type": "object" } + ] }, "activation": { "description": "What the user ran to enter the target's environment, such as sourcing an activate script, loading modules, or exporting library paths. The user activates before invoking benchx; the runner records the environment it inherited beside this declaration.",