From f445850981c9af1337d6491c0bfd56077f155510 Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Tue, 17 Nov 2026 20:21:48 +0100 Subject: [PATCH] feat(verify): read artifacts through rivet, adopt score status vocabulary Closes #5 for rivet artifact files. - tools/verify.py reads artifacts with `rivet list --format json --full` instead of PyYAML. variants/bindings.yaml stays on PyYAML until pulseengine/rivet#1094. - All 34 artifacts move from `status: approved` to `status: valid`. The vendored score schema allows draft/valid/invalid/in_progress/obsolete; rivet v0.13.0 did not check the value, v0.41.1 rejects all 34. - CI installs rivet v0.41.1 (v0.13.0 has no `list --full`). - A build or discovery error on a waived comp-req is now FAILED, not KNOWN-GAP: only a test that ran and failed is waivable. - tools/test_verify.py: 9 bucketing cases with rivet and bazel stubbed, run in CI. Reverting the discovery-error fix fails exactly one case. - DR-KVS-ARTIFACTS-THROUGH-RIVET records the decision. Local evidence: rivet 0.41.1 `make validate`: PASS (18 warnings, unchanged) old PyYAML reader vs rivet reader: same 6 comp-reqs, same verified-by lists, same waivers; only order differs (rivet sorts by id) missing or failing rivet binary: verify.py exits 1 `make verify` with bazel: not run to completion locally (bazel server under taskpolicy -b did not start within 120 s on a loaded host); CI runs it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01KEjmjapTTSvsFZti3qcD56 --- .github/workflows/validate.yml | 5 +- Makefile | 4 +- README.md | 22 +++--- artifacts/architecture.yaml | 28 ++++---- artifacts/requirements.yaml | 24 +++---- artifacts/safety-and-decisions.yaml | 50 ++++++++++--- tools/test_verify.py | 89 +++++++++++++++++++++++ tools/verify.py | 106 +++++++++++++++++----------- variants/bindings.yaml | 2 +- variants/feature-model.yaml | 2 +- verification/README.md | 2 +- 11 files changed, 239 insertions(+), 95 deletions(-) create mode 100644 tools/test_verify.py diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 501dc33..a3476f3 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -33,7 +33,7 @@ jobs: - name: Install rivet run: | cargo install --git https://github.com/pulseengine/rivet \ - --tag v0.13.0 \ + --tag v0.41.1 \ rivet-cli --bin rivet --locked rivet --version @@ -51,6 +51,9 @@ jobs: - name: bazel build (AADL → WIT → wit-bindgen → wasm component) run: bazel build //... + - name: verify.py bucketing self-test (rivet and bazel stubbed) + run: python3 -m unittest discover -s tools -p 'test_verify.py' -v + - name: artifact-driven verification gate (real bazel test per artifact) # verify.py shells out to `bazel test --test_arg=--exact …` per # entry in each comp-req's verified-by list. Honest signal: red diff --git a/Makefile b/Makefile index c449a95..f220fb2 100644 --- a/Makefile +++ b/Makefile @@ -55,7 +55,7 @@ wit: # ── Artifact-driven verification gate (always runs) ───────────────── # Same shape as the LS-N gate in pulseengine/meld. Walks every -# comp-req with status:approved, finds tests by name convention, +# comp-req with status:valid, reads its verified-by tests through rivet, # reports PASSED / FAILED / MISSING per artifact. Exits 1 if any # FAILED or MISSING. # @@ -64,7 +64,7 @@ wit: # scopes in or out. VARIANT ?= dev verify: - @$(PYTHON) tools/verify.py --artifacts artifacts --variant $(VARIANT) + @$(PYTHON) tools/verify.py --rivet $(RIVET) --schemas $(SCHEMAS) --variant $(VARIANT) # ── sigil-signed release manifest (optional) ──────────────────────── attest: diff --git a/README.md b/README.md index b8f4edf..3a21b56 100644 --- a/README.md +++ b/README.md @@ -62,29 +62,31 @@ the methodology is the lesson. **The gate reports this as KNOWN-GAP.** Until 2026-11 the gate went red on this finding, which hid every other regression behind the same -signal. The gap is now waived by `DR-KVS-KEY-VALIDATION-GAP`, an approved +signal. The gap is now waived by `DR-KVS-KEY-VALIDATION-GAP`, a valid decision record tagged `known-gap`. The four surface tests still run on every `make verify` and are listed by name. Current output (dev variant): ``` BUCKET ID EVIDENCE ───────────────────────────────────────────────────────────────────── +PASSED COMP-REQ-KVS-ATOMIC-STORE 5 test(s) all green PASSED COMP-REQ-KVS-KEY-ENCODING 1 test(s) all green PASSED COMP-REQ-KVS-VALUE-CHECKSUM 5 test(s) all green -PASSED COMP-REQ-KVS-ATOMIC-STORE 5 test(s) all green +KNOWN-GAP COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP: + - test_..._32_byte_cap_enforced KNOWN-GAP COMP-REQ-KVS-KEY-NAMING 3 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP: - test_..._space_rejected - test_..._dot_rejected - test_..._slash_rejected -KNOWN-GAP COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP: - - test_..._32_byte_cap_enforced ───────────────────────────────────────────────────────────────────── 3 PASSED, 0 FAILED, 0 MISSING, 2 KNOWN-GAP, 0 STALE-GAP ``` The waiver only covers FAILED. A waived comp-req whose tests all pass is reported as STALE-GAP and fails the gate until the decision record -is removed. MISSING evidence is never waivable. `make verify VARIANT=prod` +is removed. MISSING evidence, a malformed `verified-by:` entry and a +build or discovery error are never waivable: only a test that ran and +failed is. `tools/test_verify.py` checks these rules in CI. `make verify VARIANT=prod` additionally lists COMP-REQ-KVS-INLINE-STORAGE as MISSING and exits 1. INLINE-STORAGE is MISSING (not PASSED) on purpose: the spec asks for @@ -104,7 +106,7 @@ they belong to. Pulseengine's `verified-by:` mechanism forces test-to-requirement traceability at the level of named test functions, and the gate then asserts each named test PASSED. Requirements with no verifying test stay loudly red; a failing test is only accepted with a -named, approved decision record. +named, valid decision record. ## What this gate measures @@ -116,7 +118,7 @@ Each comp-req artifact carries a `verified-by:` list of 2. **Runs** — invokes `bazel test --test_arg=--exact --test_arg=` per entry. Any non-zero exit → bucket `FAILED`. All green → bucket `PASSED`. -3. **Waives** — a `FAILED` comp-req that an approved `known-gap` +3. **Waives** — a `FAILED` comp-req that a valid `known-gap` decision record links with `satisfies` → bucket `KNOWN-GAP` (exit 0). If such a comp-req passes → bucket `STALE-GAP` (exit 1). @@ -126,8 +128,8 @@ falsifications, the response options are explicit: - Add runtime validation to the impl → tests go green. - Edit the upstream RST to drop those requirements → artifact updates, tests can be deleted. -- Downgrade the comp-req's `status` from `approved` to `draft` (the - gate only enforces approved artifacts). +- Downgrade the comp-req's `status` from `valid` to `draft` (the + gate only enforces valid artifacts). All three are legitimate engineering responses to evidence — the gate makes the choice explicit, where a coverage-wedge visualisation makes @@ -142,7 +144,7 @@ it implicit. | `tests/surface/surface_tests.rs` | One `#[test]` per comp-req, asserting requirement-as-test against the vendored upstream code (verbatim spec text, no embellishment) | None — eclipse tests are organized by module, not requirement | | `arch/kvs.aadl` (spar AADL) | Typed feature group + subprogram signatures + ARP4761 safety properties | None | | `arch/kvs.wit` (binary contract) | WIT interface that wit-bindgen turns into a Rust trait. `src/lib.rs` implements that trait by **delegating to the vendored eclipse-score `rust_kvs`**; the .wasm component therefore links the *real* upstream code, not a toy stub. If the upstream API drifts or the WIT signature drifts, the Rust compile fails — binary-contract enforcement runs over real code. | None — interface stops at the rendered diagram | -| `tools/verify.py` (artifact-driven gate) | Walks every approved comp-req, reads `verified-by:`, runs `bazel test` per entry, exits red on any gap | None — eclipse renders a coverage pie chart | +| `tools/verify.py` (artifact-driven gate) | Walks every valid comp-req (read through `rivet list --format json --full`), reads `verified-by:`, runs `bazel test` per entry, exits red on any gap | None — eclipse renders a coverage pie chart | | `attestation/release-manifest.yaml` (sigil) | Signed in-toto-style attestation tying artifact hashes + WIT hash + evidence hashes | Green CI badge | ## What's *real* infrastructure vs *example skeleton* diff --git a/artifacts/architecture.yaml b/artifacts/architecture.yaml index dab17b8..b50f557 100644 --- a/artifacts/architecture.yaml +++ b/artifacts/architecture.yaml @@ -10,7 +10,7 @@ artifacts: - id: FEAT-PERSISTENCY type: feat title: Persistency - status: approved + status: valid description: > User-visible persistency feature exposed through a managed API. Eclipse: feat__persistency. @@ -33,7 +33,7 @@ artifacts: - id: COMP-KVS type: comp title: persistency::kvs - status: approved + status: valid description: > Key-value store implementation of the persistency feature. Eclipse: comp__persistency_kvs. @@ -56,7 +56,7 @@ artifacts: - id: LOGIC-ARC-INT-KVS type: logic-arc-int title: KVS Interface - status: approved + status: valid description: > Logical interface of the key-value store: store / load / delete / snapshot operations over (key, value) pairs. @@ -69,7 +69,7 @@ artifacts: - id: LOGIC-ARC-INT-OP-KVS-STORE type: logic-arc-int title: Store - status: approved + status: valid description: Persist a (key, value) pair durably. fields: safety-level: ASIL_B @@ -82,7 +82,7 @@ artifacts: - id: LOGIC-ARC-INT-OP-KVS-LOAD type: logic-arc-int title: Load - status: approved + status: valid description: Retrieve a previously-stored value by key, with integrity check. fields: safety-level: ASIL_B @@ -95,7 +95,7 @@ artifacts: - id: LOGIC-ARC-INT-OP-KVS-DELETE type: logic-arc-int title: Delete - status: approved + status: valid description: Remove a key-value pair durably. fields: safety-level: ASIL_B @@ -108,7 +108,7 @@ artifacts: - id: LOGIC-ARC-INT-OP-KVS-SNAPSHOT-CREATE type: logic-arc-int title: Create Snapshot - status: approved + status: valid description: Atomically capture the current state of the KVS. fields: safety-level: ASIL_B @@ -121,7 +121,7 @@ artifacts: - id: LOGIC-ARC-INT-OP-KVS-SNAPSHOT-RESTORE type: logic-arc-int title: Restore Snapshot - status: approved + status: valid description: Atomically restore the KVS state from a prior snapshot. fields: safety-level: ASIL_B @@ -136,7 +136,7 @@ artifacts: - id: DD-STA-KVS-KEY-VALIDATOR type: dd-sta title: Key validator - status: approved + status: valid description: > Static design of the syntactic key validator. Validates the alphabet, length boundaries, and leading-dot rejection. @@ -151,7 +151,7 @@ artifacts: - id: DD-STA-KVS-BACKEND-STRATEGY type: dd-sta title: Storage backend strategy - status: approved + status: valid description: > Strategy pattern: `KvsBackend` trait with two implementations (RocksDb for production, InMemory for tests). @@ -166,7 +166,7 @@ artifacts: - id: DD-DYN-KVS-ATOMIC-STORE type: dd-dyn title: Atomic store sequence - status: approved + status: valid description: > Sequence design for the atomic store path: WAL append → fsync → live-table update → log rotation. Power-loss between any @@ -184,7 +184,7 @@ artifacts: - id: SW-UNIT-KVS-VALIDATOR type: sw-unit title: KVS key validator implementation - status: approved + status: valid description: Rust source implementing the syntactic key validator. fields: safety-level: ASIL_B @@ -200,7 +200,7 @@ artifacts: - id: SW-UNIT-KVS-BACKEND type: sw-unit title: KVS RocksDB backend implementation - status: approved + status: valid description: Rust source implementing the RocksDB-backed `KvsBackend`. fields: safety-level: ASIL_B @@ -216,7 +216,7 @@ artifacts: - id: SW-UNIT-KVS-ATOMIC-STORE type: sw-unit title: KVS atomic-store implementation - status: approved + status: valid description: Rust source implementing the WAL-based atomic store. fields: safety-level: ASIL_B diff --git a/artifacts/requirements.yaml b/artifacts/requirements.yaml index 8cca667..b5e4685 100644 --- a/artifacts/requirements.yaml +++ b/artifacts/requirements.yaml @@ -14,7 +14,7 @@ artifacts: - id: STKH-REQ-PERSIST-STORAGE type: stkh-req title: Persistent storage capability - status: approved + status: valid description: > The SW-platform shall provide persistent storage for application data that survives power-cycles and software updates. @@ -31,7 +31,7 @@ artifacts: - id: STKH-REQ-PERSIST-INTEGRITY type: stkh-req title: Stored-data integrity - status: approved + status: valid description: > The SW-platform shall detect corruption of persisted data caused by power-loss-during-write or media degradation. @@ -47,7 +47,7 @@ artifacts: - id: FEAT-REQ-PERSIST-STORE type: feat-req title: Store data - status: approved + status: valid description: > The persistency feature shall accept (key, value) pairs and persist them durably across power cycles. @@ -67,7 +67,7 @@ artifacts: - id: FEAT-REQ-PERSIST-LOAD type: feat-req title: Load data - status: approved + status: valid description: > The persistency feature shall retrieve a previously-stored value by key, or report not-found. @@ -84,7 +84,7 @@ artifacts: - id: FEAT-REQ-PERSIST-INTEGRITY-CHECK type: feat-req title: Integrity check on load - status: approved + status: valid description: > The persistency feature shall compute and verify a checksum on every load, returning an error rather than corrupted data @@ -102,7 +102,7 @@ artifacts: - id: FEAT-REQ-PERSIST-NO-RUNTIME-ALLOC type: feat-req title: No dynamic allocation post-init - status: approved + status: valid description: > The persistency feature shall not perform heap allocation after initialization (no malloc/free on the hot path). @@ -125,7 +125,7 @@ artifacts: - id: COMP-REQ-KVS-KEY-NAMING type: comp-req title: Key Naming - status: approved + status: valid description: > The component shall accept keys that consist solely of alphanumeric characters, underscores, or dashes. @@ -156,7 +156,7 @@ artifacts: - id: COMP-REQ-KVS-KEY-LENGTH type: comp-req title: Key Length - status: approved + status: valid description: > The component shall limit the maximum length of a key to 32 bytes. Eclipse: comp_req__kvs__key_length (verbatim). @@ -179,7 +179,7 @@ artifacts: - id: COMP-REQ-KVS-KEY-ENCODING type: comp-req title: Key Encoding - status: approved + status: valid description: > The component shall encode each key as valid UTF-8. Eclipse: comp_req__kvs__key_encoding (verbatim). @@ -204,7 +204,7 @@ artifacts: - id: COMP-REQ-KVS-VALUE-CHECKSUM type: comp-req title: Per-value checksum - status: approved + status: valid description: > Each stored value shall carry a CRC32C checksum computed at store time and verified at load time. @@ -232,7 +232,7 @@ artifacts: - id: COMP-REQ-KVS-ATOMIC-STORE type: comp-req title: Atomic store under power loss - status: approved + status: valid description: > A store operation shall either complete fully and durably or leave the previous value intact. Partial writes must @@ -258,7 +258,7 @@ artifacts: - id: COMP-REQ-KVS-INLINE-STORAGE type: comp-req title: Inline storage, no runtime allocation - status: approved + status: valid description: > KVS shall use pre-allocated inline buffers sized at component initialization; no heap allocation on the hot path. diff --git a/artifacts/safety-and-decisions.yaml b/artifacts/safety-and-decisions.yaml index 4e2c35f..6b368a6 100644 --- a/artifacts/safety-and-decisions.yaml +++ b/artifacts/safety-and-decisions.yaml @@ -9,7 +9,7 @@ artifacts: - id: FMEA-KVS-SILENT-CORRUPTION type: fmea-entry title: Silent corruption of stored value - status: approved + status: valid description: > Power loss mid-write leaves a stored value in a torn state, not detected by integrity check. @@ -37,7 +37,7 @@ artifacts: - id: FMEA-KVS-CHECKSUM-BYPASS type: fmea-entry title: Checksum verification bypassed on hot path - status: approved + status: valid description: > An optimisation path skips checksum verification under certain conditions, allowing corrupted data through. @@ -66,7 +66,7 @@ artifacts: - id: DR-KVS-BACKEND-CHOICE type: decision-record title: JSON file backend with snapshot rotation (no WAL backend) - status: approved + status: valid description: Selection of the durable storage backend that ships today. fields: rationale: > @@ -99,7 +99,7 @@ artifacts: - id: DR-KVS-INLINE-STORAGE-GAP type: decision-record title: Inline-storage requirement is unmet by the current backend - status: approved + status: valid description: Tracking the gap between the spec and the impl. fields: rationale: > @@ -132,11 +132,11 @@ artifacts: - id: DR-KVS-KEY-VALIDATION-GAP type: decision-record title: Upstream rust_kvs does not enforce key naming or key length - status: approved + status: valid description: > Known-gap waiver for COMP-REQ-KVS-KEY-NAMING and COMP-REQ-KVS-KEY-LENGTH. The verify gate reports both as - KNOWN-GAP instead of FAILED while this record is approved. + KNOWN-GAP instead of FAILED while this record is valid. fields: rationale: > Vendored rust_kvs set_value accepts keys containing space, '.' @@ -164,17 +164,47 @@ artifacts: - type: belongs-to target: COMP-KVS + - id: DR-KVS-ARTIFACTS-THROUGH-RIVET + type: decision-record + title: Artifacts use the score status vocabulary and are read through rivet + status: valid + description: > + The verify gate reads artifacts with `rivet list --format json --full` + instead of parsing artifacts/*.yaml with PyYAML (issue #5), and every + artifact uses `status: valid` instead of `approved`. + fields: + rationale: > + The vendored score schema allows status draft, valid, invalid, + in_progress or obsolete. All 34 artifacts said `approved`; rivet + v0.13.0 did not check the value, rivet v0.41.1 rejects all 34. + `valid` is also eclipse-score's own term (`:status: valid`). Org + policy (#5) is that a repository reads its own artifact files + through rivet, which needs `list --full` (not in v0.13.0). + alternatives: > + (a) keep rivet v0.13.0 and the PyYAML reader, leaving the status + values unchecked; (b) extend the schema with `approved`, diverging + from the vendored score schema. + consequences: > + CI installs rivet v0.41.1. The gate scope is comp-reqs with status + valid; known-gap waivers must be valid decision-records. + variants/bindings.yaml is still read with PyYAML until + pulseengine/rivet#1094 lands. + tags: [tooling, decision, verification] + links: + - type: belongs-to + target: COMP-KVS + # ── Test specifications ──────────────────────────────────────────── # The artifact-driven verification gate (tools/verify.py) looks for # tests matching `test__*` - # for every comp-req with status: approved. Each test-spec below + # for every comp-req with status: valid. Each test-spec below # names the test convention explicitly so the artifact ↔ test # binding is auditable. - id: TEST-SPEC-KVS-KEY-VALIDATOR type: test-spec title: Key validator unit tests - status: approved + status: valid description: > Unit-level coverage of the syntactic key validator. Boundary cases: empty string, single-char, 254/255/256 length, @@ -194,7 +224,7 @@ artifacts: - id: TEST-SPEC-KVS-CHECKSUM-FAULT-INJECT type: test-spec title: Checksum verification fault-injection tests - status: approved + status: valid description: > Fault-injection tests that flip bytes in the backing store between store and load. Each load must return IntegrityError. @@ -212,7 +242,7 @@ artifacts: - id: TEST-SPEC-KVS-ATOMIC-STORE-POWERLOSS type: test-spec title: Atomic-store power-loss simulation - status: approved + status: valid description: > Simulated power-loss between WAL append and table update; after recovery, only the previous-value or the new-value diff --git a/tools/test_verify.py b/tools/test_verify.py new file mode 100644 index 0000000..7653c09 --- /dev/null +++ b/tools/test_verify.py @@ -0,0 +1,89 @@ +"""Bucketing self-test for tools/verify.py, with rivet and bazel stubbed. + +Run: python3 -m unittest discover -s tools -p 'test_verify.py' +""" +from __future__ import annotations + +import json +import pathlib +import sys +import tempfile +import unittest +from unittest import mock + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import verify # noqa: E402 + +T = "//tests/surface:surface_tests" + + +def comp_req(aid, tests, status="valid"): + return {"id": aid, "type": "comp-req", "status": status, "tags": [], + "links": [], "fields": {"verified-by": [f"{T}:{t}" for t in tests]}} + + +def waiver(target, status="valid", tag="known-gap"): + return {"id": "DR-GAP", "type": "decision-record", "status": status, + "tags": [tag], "fields": {}, + "links": [{"type": "satisfies", "target": target}]} + + +class Bucketing(unittest.TestCase): + def run_gate(self, artifacts, present, passing, discovery_error=False): + def list_tests(target): + if discovery_error: + raise RuntimeError("bazel build failed") + return set(present) + + with tempfile.TemporaryDirectory() as d: + report = pathlib.Path(d) / "out.json" + with mock.patch.object(verify, "load_artifacts", return_value=artifacts), \ + mock.patch.object(verify, "list_tests", side_effect=list_tests), \ + mock.patch.object(verify, "run_one_test", + side_effect=lambda tgt, name: name in passing), \ + mock.patch("sys.stdout"): + code = verify.main(["--variants", d, "--report", str(report)]) + return code, json.loads(report.read_text())["counts"] + + def test_all_pass(self): + code, c = self.run_gate([comp_req("A", ["t1"])], ["t1"], {"t1"}) + self.assertEqual((code, c["passed"]), (0, 1)) + + def test_unwaived_failure_fails(self): + code, c = self.run_gate([comp_req("A", ["t1"])], ["t1"], set()) + self.assertEqual((code, c["failed"]), (1, 1)) + + def test_waived_failure_is_known_gap(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A")], ["t1"], set()) + self.assertEqual((code, c["known_gap"], c["failed"]), (0, 1, 0)) + + def test_waived_pass_is_stale(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A")], ["t1"], {"t1"}) + self.assertEqual((code, c["stale_gap"]), (1, 1)) + + def test_waiver_does_not_cover_discovery_error(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A")], ["t1"], set(), + discovery_error=True) + self.assertEqual((code, c["failed"], c["known_gap"]), (1, 1, 0)) + + def test_waiver_does_not_cover_missing(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A")], [], set()) + self.assertEqual((code, c["missing"]), (1, 1)) + + def test_draft_waiver_ignored(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A", status="draft")], + ["t1"], set()) + self.assertEqual((code, c["failed"]), (1, 1)) + + def test_untagged_record_ignored(self): + code, c = self.run_gate([comp_req("A", ["t1"]), waiver("A", tag="other")], + ["t1"], set()) + self.assertEqual((code, c["failed"]), (1, 1)) + + def test_non_valid_comp_req_out_of_scope(self): + code, c = self.run_gate([comp_req("A", ["t1"], status="draft")], ["t1"], set()) + self.assertEqual((code, sum(c.values())), (0, 0)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/verify.py b/tools/verify.py index ad63994..542918e 100644 --- a/tools/verify.py +++ b/tools/verify.py @@ -2,7 +2,7 @@ """verify.py — artifact-driven verification gate for example-kvs. Same shape as the LS-N gate in pulseengine/meld. Walks every comp-req -artifact whose status is `approved`, reads its `verified-by:` field +artifact whose status is `valid`, reads its `verified-by:` field (a list of strings of the form `:`), runs each test via `bazel test --test_arg=--exact --test_arg=`, and buckets the artifact: @@ -12,10 +12,11 @@ MISSING — verified-by is absent / empty, OR one of the listed tests was not present in the bazel target's test binary. - KNOWN-GAP — would be FAILED, but an approved decision-record tagged + KNOWN-GAP — would be FAILED, but a valid decision-record tagged `known-gap` links `satisfies` to the comp-req. Reported, - does not fail the gate. Only FAILED is waivable; MISSING - is not. + does not fail the gate. Only a test that ran and failed is + waivable; MISSING, a malformed entry or a build/discovery + error stays FAILED. STALE-GAP — a known-gap waiver exists but every listed test PASSED. Fails the gate until the waiver is removed, so a waiver cannot outlive the gap it records. @@ -26,8 +27,14 @@ a requirement, the surface test fires red and the gate goes red. That is the demonstration vs. eclipse-score's coverage-wedge approach. +Artifacts are read through rivet (`rivet list --format json --full`), +not parsed as YAML here: the repository reads its own artifact files +through rivet's owned YAML stack (issue #5). variants/bindings.yaml is +not a rivet artifact file and is still read with PyYAML until rivet +can print an arbitrary YAML file as JSON (pulseengine/rivet#1094). + Usage: - python3 tools/verify.py [--artifacts artifacts] + python3 tools/verify.py [--rivet rivet] [--schemas vendor/rivet-schemas] [--report .verify-output.json] [--no-run] # discovery only, do not invoke bazel @@ -72,54 +79,59 @@ def parse_verified_by(entry: str) -> tuple[str, str]: # ── artifact discovery ─────────────────────────────────────────────── -def expected_artifacts(artifact_dir: pathlib.Path, +def load_artifacts(rivet: str, schemas: str | None) -> list[dict]: + """Every artifact in the project, as `rivet list --format json --full` + reports it (id, type, status, tags, links, fields).""" + cmd = [rivet] + if schemas: + cmd += ["--schemas", schemas] + cmd += ["list", "--format", "json", "--full"] + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0: + raise RuntimeError( + f"{' '.join(cmd)} failed (exit {proc.returncode}):\n{proc.stderr}" + ) + return json.loads(proc.stdout)["artifacts"] + + +def expected_artifacts(artifacts: list[dict], skip_ids: set[str] | None = None) -> list[dict]: - """Approved comp-reqs with their (possibly empty) verified-by list. + """Valid comp-reqs with their (possibly empty) verified-by list. If `skip_ids` is provided, artifacts whose id is in the set are omitted (used to honor a variant's `out-of-scope-for:` list). """ skip = skip_ids or set() out = [] - for path in sorted(artifact_dir.glob("*.yaml")): - try: - data = yaml.safe_load(path.read_text()) or {} - except yaml.YAMLError: + for art in artifacts: + if art.get("type") != "comp-req": continue - for art in data.get("artifacts", []): - if art.get("type") != "comp-req": - continue - if art.get("status") != "approved": - continue - if art["id"] in skip: - continue - verified = art.get("fields", {}).get("verified-by", []) or [] - out.append({"id": art["id"], "verified-by": list(verified)}) + if art.get("status") != "valid": + continue + if art["id"] in skip: + continue + verified = (art.get("fields") or {}).get("verified-by", []) or [] + out.append({"id": art["id"], "verified-by": list(verified)}) return out -def known_gaps(artifact_dir: pathlib.Path) -> dict[str, list[str]]: - """Map comp-req id -> ids of approved `known-gap` decision-records. +def known_gaps(artifacts: list[dict]) -> dict[str, list[str]]: + """Map comp-req id -> ids of valid `known-gap` decision-records. - A decision-record waives a comp-req when it has status approved, + A decision-record waives a comp-req when it has status valid, carries the tag `known-gap`, and links `satisfies` to the comp-req. """ out: dict[str, list[str]] = {} - for path in sorted(artifact_dir.glob("*.yaml")): - try: - data = yaml.safe_load(path.read_text()) or {} - except yaml.YAMLError: + for art in artifacts: + if art.get("type") != "decision-record": continue - for art in data.get("artifacts", []): - if art.get("type") != "decision-record": - continue - if art.get("status") != "approved": - continue - if "known-gap" not in (art.get("tags") or []): - continue - for link in art.get("links", []) or []: - if link.get("type") == "satisfies" and link.get("target"): - out.setdefault(link["target"], []).append(art["id"]) + if art.get("status") != "valid": + continue + if "known-gap" not in (art.get("tags") or []): + continue + for link in art.get("links", []) or []: + if link.get("type") == "satisfies" and link.get("target"): + out.setdefault(link["target"], []).append(art["id"]) return out @@ -217,8 +229,10 @@ def run_one_test(target: str, test_name: str) -> bool: def main(argv: list[str]) -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--artifacts", type=pathlib.Path, - default=pathlib.Path("artifacts")) + ap.add_argument("--rivet", type=str, default="rivet", + help="rivet binary used to read the artifacts") + ap.add_argument("--schemas", type=str, default=None, + help="Schema directory passed to rivet --schemas") ap.add_argument("--variants", type=pathlib.Path, default=pathlib.Path("variants"), help="Directory containing feature-model.yaml + bindings.yaml") @@ -235,8 +249,9 @@ def main(argv: list[str]) -> int: if skip: print(f"# variant={args.variant} — skipping {len(skip)} out-of-scope comp-req(s): " f"{', '.join(sorted(skip))}") - artifacts = expected_artifacts(args.artifacts, skip_ids=skip) - waivers = known_gaps(args.artifacts) + all_artifacts = load_artifacts(args.rivet, args.schemas) + artifacts = expected_artifacts(all_artifacts, skip_ids=skip) + waivers = known_gaps(all_artifacts) passed: list[tuple[str, list[str]]] = [] failed: list[tuple[str, list[str]]] = [] missing: list[tuple[str, str]] = [] @@ -253,13 +268,16 @@ def main(argv: list[str]) -> int: # Parse + check presence + run. gone: list[str] = [] failed_here: list[str] = [] + # Malformed entries and build/discovery errors: the test never ran, + # so a known-gap waiver must not cover them. + broken_here: list[str] = [] passed_here: list[str] = [] for entry in entries: try: target, test_name = parse_verified_by(entry) except ValueError as e: - failed_here.append(f"{entry}: malformed ({e})") + broken_here.append(f"{entry}: malformed ({e})") continue # Presence check. @@ -268,7 +286,7 @@ def main(argv: list[str]) -> int: gone.append(f"{target}::{test_name}") continue except Exception as e: - failed_here.append(f"{entry}: discovery failed: {e}") + broken_here.append(f"{entry}: discovery failed: {e}") continue if args.no_run: @@ -283,6 +301,8 @@ def main(argv: list[str]) -> int: if gone: missing.append((aid, f"{len(gone)} listed test(s) not present in target: " + ", ".join(gone))) + elif broken_here: + failed.append((aid, broken_here + failed_here)) elif failed_here: if aid in waivers and not args.no_run: known.append((aid, failed_here, waivers[aid])) diff --git a/variants/bindings.yaml b/variants/bindings.yaml index 8ed34fa..a9ce616 100644 --- a/variants/bindings.yaml +++ b/variants/bindings.yaml @@ -35,7 +35,7 @@ bindings: Production builds — KvsDefaults::Required and KvsLoad::Required mean startup fails closed if persistent state is missing; snapshot ring sized for the target ECU's flash budget; every - approved comp-req must verify; release-mode + LTO + single + valid comp-req must verify; release-mode + LTO + single codegen-unit + panic=abort applied at the bazel layer. kvs-config: defaults: Required # KvsDefaults::Required diff --git a/variants/feature-model.yaml b/variants/feature-model.yaml index b9c09fb..f7c238b 100644 --- a/variants/feature-model.yaml +++ b/variants/feature-model.yaml @@ -40,7 +40,7 @@ feature-model: `prod` — production builds; strict constraints, full snapshot ring, mandatory persistency on startup. Every - approved comp-req must have a verifying test that runs + valid comp-req must have a verifying test that runs green. # Cross-variant constraint set. Empty for now — both variants are diff --git a/verification/README.md b/verification/README.md index d92b201..4f120d3 100644 --- a/verification/README.md +++ b/verification/README.md @@ -11,7 +11,7 @@ Two parts: 2. **`../tools/verify.py`** — the artifact-driven verification gate (see top-level README). Walks the rivet artifacts looking for - `comp-req`s with `status: approved`, finds tests by name + `comp-req`s with `status: valid`, finds tests by name convention (`test__*`), runs them, reports pass/fail/missing per artifact. Same shape as the LS-N gate in `pulseengine/meld`.