Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,25 +60,33 @@ spec/impl delta a CI signal, where a coverage dashboard hides it as
a wedge." The eclipse-score project itself is healthy and well-run;
the methodology is the lesson.

**The gate calls this out by going RED.** Current `make verify` output:
**The gate reports this as KNOWN-GAP.** Until 2026-11 the gate went
red on this finding, which hid every other regression behind the same
signal. The gap is now waived by `DR-KVS-KEY-VALIDATION-GAP`, an approved
decision record tagged `known-gap`. The four surface tests still run on
every `make verify` and are listed by name. Current output (dev variant):

```
BUCKET ID EVIDENCE
─────────────────────────────────────────────────────────────────────
PASSED COMP-REQ-KVS-KEY-ENCODING 1 test(s) all green
PASSED COMP-REQ-KVS-VALUE-CHECKSUM 5 test(s) all green
PASSED COMP-REQ-KVS-ATOMIC-STORE 5 test(s) all green
FAILED COMP-REQ-KVS-KEY-NAMING 3 test(s) failed:
KNOWN-GAP COMP-REQ-KVS-KEY-NAMING 3 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP:
- test_..._space_rejected
- test_..._dot_rejected
- test_..._slash_rejected
FAILED COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed:
KNOWN-GAP COMP-REQ-KVS-KEY-LENGTH 1 test(s) failed, waived by DR-KVS-KEY-VALIDATION-GAP:
- test_..._32_byte_cap_enforced
MISSING COMP-REQ-KVS-INLINE-STORAGE verified-by is absent or empty
─────────────────────────────────────────────────────────────────────
3 PASSED, 2 FAILED, 1 MISSING
3 PASSED, 0 FAILED, 0 MISSING, 2 KNOWN-GAP, 0 STALE-GAP
```

The waiver only covers FAILED. A waived comp-req whose tests all pass
is reported as STALE-GAP and fails the gate until the decision record
is removed. MISSING evidence is never waivable. `make verify VARIANT=prod`
additionally lists COMP-REQ-KVS-INLINE-STORAGE as MISSING and exits 1.

INLINE-STORAGE is MISSING (not PASSED) on purpose: the spec asks for
"no runtime heap allocation," upstream uses HashMap + Arc<Mutex> which
do allocate, and a genuine verifying test requires witness
Expand All @@ -88,14 +96,15 @@ DR-KVS-INLINE-STORAGE-GAP records the three response options

The surface tests at [`tests/surface/surface_tests.rs`](tests/surface/surface_tests.rs)
assert exactly what the upstream RST says — no pulseengine-side
embellishment. CI is intentionally red on this finding.
embellishment.

This is the **demonstration**: a coverage-percentage dashboard would
mark these comp-reqs "covered" because some test mentions the module
they belong to. Pulseengine's `verified-by:` mechanism forces
test-to-requirement traceability at the level of named test functions,
and the gate then asserts each named test PASSED. Requirements with no
verifying test stay loudly red.
verifying test stay loudly red; a failing test is only accepted with a
named, approved decision record.

## What this gate measures

Expand All @@ -107,6 +116,9 @@ Each comp-req artifact carries a `verified-by:` list of
2. **Runs** — invokes `bazel test --test_arg=--exact --test_arg=<name>`
per entry. Any non-zero exit → bucket `FAILED`. All green → bucket
`PASSED`.
3. **Waives** — a `FAILED` comp-req that an approved `known-gap`
decision record links with `satisfies` → bucket `KNOWN-GAP`
(exit 0). If such a comp-req passes → bucket `STALE-GAP` (exit 1).

Each comp-req artifact lists its evidence. For the KEY-NAMING / KEY-LENGTH
falsifications, the response options are explicit:
Expand Down
35 changes: 35 additions & 0 deletions artifacts/safety-and-decisions.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,41 @@ artifacts:
- type: belongs-to
target: COMP-KVS

- id: DR-KVS-KEY-VALIDATION-GAP
type: decision-record
title: Upstream rust_kvs does not enforce key naming or key length
status: approved
description: >
Known-gap waiver for COMP-REQ-KVS-KEY-NAMING and
COMP-REQ-KVS-KEY-LENGTH. The verify gate reports both as
KNOWN-GAP instead of FAILED while this record is approved.
fields:
rationale: >
Vendored rust_kvs set_value accepts keys containing space, '.'
and '/' and keys longer than 32 bytes (kvs.rs:238; no length
constant exists). The C++ binding carries an open TODO for key
handling. Four surface tests fail against the verbatim upstream
spec text. The finding is recorded in the README; keeping main
red on it hid every other regression behind the same signal.
alternatives: >
(a) add key validation to the vendored impl, diverging from
upstream; (b) propose validation upstream in
eclipse-score/persistency and re-vendor; (c) downgrade both
comp-reqs to draft.
consequences: >
The four surface tests still run and are reported as KNOWN-GAP
on every verify. When they pass, verify reports STALE-GAP and
fails until this record is removed. A new failure on any other
comp-req still fails the gate.
tags: [persistency, kvs, decision, keys, known-gap]
links:
- type: satisfies
target: COMP-REQ-KVS-KEY-NAMING
- type: satisfies
target: COMP-REQ-KVS-KEY-LENGTH
- type: belongs-to
target: COMP-KVS

# ── Test specifications ────────────────────────────────────────────
# The artifact-driven verification gate (tools/verify.py) looks for
# tests matching `test_<lowercase-id-with-dashes-as-underscores>_*`
Expand Down
62 changes: 58 additions & 4 deletions tools/verify.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,15 @@
MISSING — verified-by is absent / empty, OR one of the listed tests
was not present in the bazel target's test binary.

Exit code: 0 iff no FAILED and no MISSING.
KNOWN-GAP — would be FAILED, but an approved decision-record tagged
`known-gap` links `satisfies` to the comp-req. Reported,
does not fail the gate. Only FAILED is waivable; MISSING
is not.
STALE-GAP — a known-gap waiver exists but every listed test PASSED.
Fails the gate until the waiver is removed, so a waiver
cannot outlive the gap it records.

Exit code: 0 iff no FAILED, no MISSING and no STALE-GAP.

This is intentionally an honest gate. If upstream code does not honor
a requirement, the surface test fires red and the gate goes red. That
Expand Down Expand Up @@ -90,6 +98,31 @@ def expected_artifacts(artifact_dir: pathlib.Path,
return out


def known_gaps(artifact_dir: pathlib.Path) -> dict[str, list[str]]:
"""Map comp-req id -> ids of approved `known-gap` decision-records.

A decision-record waives a comp-req when it has status approved,
carries the tag `known-gap`, and links `satisfies` to the comp-req.
"""
out: dict[str, list[str]] = {}
for path in sorted(artifact_dir.glob("*.yaml")):
try:
data = yaml.safe_load(path.read_text()) or {}
except yaml.YAMLError:
continue
for art in data.get("artifacts", []):
if art.get("type") != "decision-record":
continue
if art.get("status") != "approved":
continue
if "known-gap" not in (art.get("tags") or []):
continue
for link in art.get("links", []) or []:
if link.get("type") == "satisfies" and link.get("target"):
out.setdefault(link["target"], []).append(art["id"])
return out


def variant_scope(variants_dir: pathlib.Path, variant: str) -> set[str]:
"""Return the set of comp-req IDs out-of-scope for `variant`.

Expand Down Expand Up @@ -203,9 +236,12 @@ def main(argv: list[str]) -> int:
print(f"# variant={args.variant} — skipping {len(skip)} out-of-scope comp-req(s): "
f"{', '.join(sorted(skip))}")
artifacts = expected_artifacts(args.artifacts, skip_ids=skip)
waivers = known_gaps(args.artifacts)
passed: list[tuple[str, list[str]]] = []
failed: list[tuple[str, list[str]]] = []
missing: list[tuple[str, str]] = []
known: list[tuple[str, list[str], list[str]]] = []
stale: list[tuple[str, list[str]]] = []

for art in artifacts:
aid = art["id"]
Expand Down Expand Up @@ -248,7 +284,12 @@ def main(argv: list[str]) -> int:
missing.append((aid,
f"{len(gone)} listed test(s) not present in target: " + ", ".join(gone)))
elif failed_here:
failed.append((aid, failed_here))
if aid in waivers and not args.no_run:
known.append((aid, failed_here, waivers[aid]))
else:
failed.append((aid, failed_here))
elif aid in waivers and not args.no_run:
stale.append((aid, waivers[aid]))
else:
passed.append((aid, passed_here))

Expand All @@ -263,22 +304,35 @@ def main(argv: list[str]) -> int:
print(f"{'':9s} {'':40s} - {e}")
for aid, reason in missing:
print(f"{'MISSING':9s} {aid:40s} {reason}")
for aid, entries, drs in known:
print(f"{'KNOWN-GAP':9s} {aid:40s} {len(entries)} test(s) failed, "
f"waived by {', '.join(drs)}:")
for e in entries:
print(f"{'':9s} {'':40s} - {e}")
for aid, drs in stale:
print(f"{'STALE-GAP':9s} {aid:40s} all tests PASSED; remove the "
f"known-gap waiver in {', '.join(drs)}")
print("─" * 100)
print(f"{len(passed)} PASSED, {len(failed)} FAILED, {len(missing)} MISSING")
print(f"{len(passed)} PASSED, {len(failed)} FAILED, {len(missing)} MISSING, "
f"{len(known)} KNOWN-GAP, {len(stale)} STALE-GAP")

# ── machine-readable report ─────────────────────────────────────
args.report.write_text(json.dumps({
"passed": [{"id": a, "tests": e} for a, e in passed],
"failed": [{"id": a, "tests": e} for a, e in failed],
"missing": [{"id": a, "reason": r} for a, r in missing],
"known_gap": [{"id": a, "tests": e, "waived_by": d} for a, e, d in known],
"stale_gap": [{"id": a, "waived_by": d} for a, d in stale],
"counts": {
"passed": len(passed),
"failed": len(failed),
"missing": len(missing),
"known_gap": len(known),
"stale_gap": len(stale),
},
}, indent=2))

return 1 if (failed or missing) else 0
return 1 if (failed or missing or stale) else 0


if __name__ == "__main__":
Expand Down
Loading