From 185a56661101f29549e7b4371668a3af49efee2e Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Fri, 13 Nov 2026 06:23:17 +0100 Subject: [PATCH] feat(verify): add KNOWN-GAP bucket waived by approved decision records Main has been red since #2 (2026-11-02) on four surface tests for COMP-REQ-KVS-KEY-NAMING and COMP-REQ-KVS-KEY-LENGTH: vendored rust_kvs accepts keys with space, '.', '/' and keys over 32 bytes. That red hid every other regression, including dependabot PRs #8 and #9. verify.py now reports a FAILED comp-req as KNOWN-GAP (exit 0) when an approved decision-record tagged known-gap links satisfies to it. A waived comp-req whose tests all pass is STALE-GAP (exit 1). MISSING is never waivable. DR-KVS-KEY-VALIDATION-GAP waives the two key comp-reqs. Local runs: make verify exit 0: 3 PASSED, 2 KNOWN-GAP verify.py --variant prod exit 1: 1 MISSING (INLINE-STORAGE) verify.py with a waiver on KEY-ENCODING (passing) exit 1: 1 STALE-GAP Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01KEjmjapTTSvsFZti3qcD56 --- README.md | 26 ++++++++---- artifacts/safety-and-decisions.yaml | 35 ++++++++++++++++ tools/verify.py | 62 +++++++++++++++++++++++++++-- 3 files changed, 112 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index d644fa9..b8f4edf 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,11 @@ spec/impl delta a CI signal, where a coverage dashboard hides it as a wedge." The eclipse-score project itself is healthy and well-run; the methodology is the lesson. -**The gate calls this out by going RED.** Current `make verify` output: +**The gate reports this as KNOWN-GAP.** Until 2026-11 the gate went +red on this finding, which hid every other regression behind the same +signal. The gap is now waived by `DR-KVS-KEY-VALIDATION-GAP`, an approved +decision record tagged `known-gap`. The four surface tests still run on +every `make verify` and are listed by name. Current output (dev variant): ``` BUCKET ID EVIDENCE @@ -68,17 +72,21 @@ BUCKET ID EVIDENCE PASSED COMP-REQ-KVS-KEY-ENCODING 1 test(s) all green PASSED COMP-REQ-KVS-VALUE-CHECKSUM 5 test(s) all green PASSED COMP-REQ-KVS-ATOMIC-STORE 5 test(s) all green -FAILED COMP-REQ-KVS-KEY-NAMING 3 test(s) failed: +KNOWN-GAP COMP-REQ-KVS-KEY-NAMING 3 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP: - test_..._space_rejected - test_..._dot_rejected - test_..._slash_rejected -FAILED COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed: +KNOWN-GAP COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP: - test_..._32_byte_cap_enforced -MISSING COMP-REQ-KVS-INLINE-STORAGE verified-by is absent or empty ───────────────────────────────────────────────────────────────────── -3 PASSED, 2 FAILED, 1 MISSING +3 PASSED, 0 FAILED, 0 MISSING, 2 KNOWN-GAP, 0 STALE-GAP ``` +The waiver only covers FAILED. A waived comp-req whose tests all pass +is reported as STALE-GAP and fails the gate until the decision record +is removed. MISSING evidence is never waivable. `make verify VARIANT=prod` +additionally lists COMP-REQ-KVS-INLINE-STORAGE as MISSING and exits 1. + INLINE-STORAGE is MISSING (not PASSED) on purpose: the spec asks for "no runtime heap allocation," upstream uses HashMap + Arc which do allocate, and a genuine verifying test requires witness @@ -88,14 +96,15 @@ DR-KVS-INLINE-STORAGE-GAP records the three response options The surface tests at [`tests/surface/surface_tests.rs`](tests/surface/surface_tests.rs) assert exactly what the upstream RST says — no pulseengine-side -embellishment. CI is intentionally red on this finding. +embellishment. This is the **demonstration**: a coverage-percentage dashboard would mark these comp-reqs "covered" because some test mentions the module they belong to. Pulseengine's `verified-by:` mechanism forces test-to-requirement traceability at the level of named test functions, and the gate then asserts each named test PASSED. Requirements with no -verifying test stay loudly red. +verifying test stay loudly red; a failing test is only accepted with a +named, approved decision record. ## What this gate measures @@ -107,6 +116,9 @@ Each comp-req artifact carries a `verified-by:` list of 2. **Runs** — invokes `bazel test --test_arg=--exact --test_arg=` per entry. Any non-zero exit → bucket `FAILED`. All green → bucket `PASSED`. +3. **Waives** — a `FAILED` comp-req that an approved `known-gap` + decision record links with `satisfies` → bucket `KNOWN-GAP` + (exit 0). If such a comp-req passes → bucket `STALE-GAP` (exit 1). Each comp-req artifact lists its evidence. For the KEY-NAMING / KEY-LENGTH falsifications, the response options are explicit: diff --git a/artifacts/safety-and-decisions.yaml b/artifacts/safety-and-decisions.yaml index 847fb98..4e2c35f 100644 --- a/artifacts/safety-and-decisions.yaml +++ b/artifacts/safety-and-decisions.yaml @@ -129,6 +129,41 @@ artifacts: - type: belongs-to target: COMP-KVS + - id: DR-KVS-KEY-VALIDATION-GAP + type: decision-record + title: Upstream rust_kvs does not enforce key naming or key length + status: approved + description: > + Known-gap waiver for COMP-REQ-KVS-KEY-NAMING and + COMP-REQ-KVS-KEY-LENGTH. The verify gate reports both as + KNOWN-GAP instead of FAILED while this record is approved. + fields: + rationale: > + Vendored rust_kvs set_value accepts keys containing space, '.' + and '/' and keys longer than 32 bytes (kvs.rs:238; no length + constant exists). The C++ binding carries an open TODO for key + handling. Four surface tests fail against the verbatim upstream + spec text. The finding is recorded in the README; keeping main + red on it hid every other regression behind the same signal. + alternatives: > + (a) add key validation to the vendored impl, diverging from + upstream; (b) propose validation upstream in + eclipse-score/persistency and re-vendor; (c) downgrade both + comp-reqs to draft. + consequences: > + The four surface tests still run and are reported as KNOWN-GAP + on every verify. When they pass, verify reports STALE-GAP and + fails until this record is removed. A new failure on any other + comp-req still fails the gate. + tags: [persistency, kvs, decision, keys, known-gap] + links: + - type: satisfies + target: COMP-REQ-KVS-KEY-NAMING + - type: satisfies + target: COMP-REQ-KVS-KEY-LENGTH + - type: belongs-to + target: COMP-KVS + # ── Test specifications ──────────────────────────────────────────── # The artifact-driven verification gate (tools/verify.py) looks for # tests matching `test__*` diff --git a/tools/verify.py b/tools/verify.py index b3adf61..ad63994 100644 --- a/tools/verify.py +++ b/tools/verify.py @@ -12,7 +12,15 @@ MISSING — verified-by is absent / empty, OR one of the listed tests was not present in the bazel target's test binary. -Exit code: 0 iff no FAILED and no MISSING. + KNOWN-GAP — would be FAILED, but an approved decision-record tagged + `known-gap` links `satisfies` to the comp-req. Reported, + does not fail the gate. Only FAILED is waivable; MISSING + is not. + STALE-GAP — a known-gap waiver exists but every listed test PASSED. + Fails the gate until the waiver is removed, so a waiver + cannot outlive the gap it records. + +Exit code: 0 iff no FAILED, no MISSING and no STALE-GAP. This is intentionally an honest gate. If upstream code does not honor a requirement, the surface test fires red and the gate goes red. That @@ -90,6 +98,31 @@ def expected_artifacts(artifact_dir: pathlib.Path, return out +def known_gaps(artifact_dir: pathlib.Path) -> dict[str, list[str]]: + """Map comp-req id -> ids of approved `known-gap` decision-records. + + A decision-record waives a comp-req when it has status approved, + carries the tag `known-gap`, and links `satisfies` to the comp-req. + """ + out: dict[str, list[str]] = {} + for path in sorted(artifact_dir.glob("*.yaml")): + try: + data = yaml.safe_load(path.read_text()) or {} + except yaml.YAMLError: + continue + for art in data.get("artifacts", []): + if art.get("type") != "decision-record": + continue + if art.get("status") != "approved": + continue + if "known-gap" not in (art.get("tags") or []): + continue + for link in art.get("links", []) or []: + if link.get("type") == "satisfies" and link.get("target"): + out.setdefault(link["target"], []).append(art["id"]) + return out + + def variant_scope(variants_dir: pathlib.Path, variant: str) -> set[str]: """Return the set of comp-req IDs out-of-scope for `variant`. @@ -203,9 +236,12 @@ def main(argv: list[str]) -> int: print(f"# variant={args.variant} — skipping {len(skip)} out-of-scope comp-req(s): " f"{', '.join(sorted(skip))}") artifacts = expected_artifacts(args.artifacts, skip_ids=skip) + waivers = known_gaps(args.artifacts) passed: list[tuple[str, list[str]]] = [] failed: list[tuple[str, list[str]]] = [] missing: list[tuple[str, str]] = [] + known: list[tuple[str, list[str], list[str]]] = [] + stale: list[tuple[str, list[str]]] = [] for art in artifacts: aid = art["id"] @@ -248,7 +284,12 @@ def main(argv: list[str]) -> int: missing.append((aid, f"{len(gone)} listed test(s) not present in target: " + ", ".join(gone))) elif failed_here: - failed.append((aid, failed_here)) + if aid in waivers and not args.no_run: + known.append((aid, failed_here, waivers[aid])) + else: + failed.append((aid, failed_here)) + elif aid in waivers and not args.no_run: + stale.append((aid, waivers[aid])) else: passed.append((aid, passed_here)) @@ -263,22 +304,35 @@ def main(argv: list[str]) -> int: print(f"{'':9s} {'':40s} - {e}") for aid, reason in missing: print(f"{'MISSING':9s} {aid:40s} {reason}") + for aid, entries, drs in known: + print(f"{'KNOWN-GAP':9s} {aid:40s} {len(entries)} test(s) failed, " + f"waived by {', '.join(drs)}:") + for e in entries: + print(f"{'':9s} {'':40s} - {e}") + for aid, drs in stale: + print(f"{'STALE-GAP':9s} {aid:40s} all tests PASSED; remove the " + f"known-gap waiver in {', '.join(drs)}") print("─" * 100) - print(f"{len(passed)} PASSED, {len(failed)} FAILED, {len(missing)} MISSING") + print(f"{len(passed)} PASSED, {len(failed)} FAILED, {len(missing)} MISSING, " + f"{len(known)} KNOWN-GAP, {len(stale)} STALE-GAP") # ── machine-readable report ───────────────────────────────────── args.report.write_text(json.dumps({ "passed": [{"id": a, "tests": e} for a, e in passed], "failed": [{"id": a, "tests": e} for a, e in failed], "missing": [{"id": a, "reason": r} for a, r in missing], + "known_gap": [{"id": a, "tests": e, "waived_by": d} for a, e, d in known], + "stale_gap": [{"id": a, "waived_by": d} for a, d in stale], "counts": { "passed": len(passed), "failed": len(failed), "missing": len(missing), + "known_gap": len(known), + "stale_gap": len(stale), }, }, indent=2)) - return 1 if (failed or missing) else 0 + return 1 if (failed or missing or stale) else 0 if __name__ == "__main__":