fix: require explicit parity evidence for SDK facts export

docs/pipeline/VERIFIED_FACTS_EXPORT.md

@@ -0,0 +1,38 @@
+# Recorded parity facts export
+
+Run `python tools/decomp_pipeline/export_verified_facts.py` from a checkout to
+write its three legacy SDK JSON files under `rebuild/sdk/`. Use `--root PATH`
+to select another checkout explicitly. The exporter reads that checkout's
+manifest, recorded parity audit, compiled sources and optional binding queue.
+It does not compile candidates, read retail inputs or run behavior tests.
+
+A function needs one manifest identity, one source identity and an explicit
+`EXACT` or `RELOC` audit row identifying the same source filename. Missing rows,
+ambiguous addresses, other verdicts and sources rejected by `crawl.purity` are
+excluded. Merely placing a file in `src/compiled` never supplies a `MATCH*` proof.
+Binding exports use this same eligible address set and manifest signatures.
+
+Function names, class labels and signatures remain manifest metadata. Each
+function includes its `label_trust` reason; a recorded matching address does not
+make a propagated class name authoritative.
+
+Struct inference uses only those eligible source files, grouped by the existing
+`label_trust` class policy. Unaudited declarations cannot contribute fields to
+an otherwise eligible class. Conflicting field declarations and sizes smaller
+than the complete field extent are excluded. These declarations remain inferred
+layout information; byte parity alone does not prove every field declaration.
+
+The legacy filenames and `verified_methods` key are retained for compatibility.
+Every payload includes `assurance` explaining that the evidence is **recorded
+parity**, not fresh source/dependency verification, signature certification or
+behavioral proof. Exported source hashes identify the current snapshot; they
+do not retroactively bind older audit rows to those bytes. Consumers requiring
+fresh or behavior-certified facts need additional evidence. This export does
+not repair the audit producer's missing dependency/proof freshness contract.
+
+Mirroring is **off by default**. The old `--no-mirror` flag remains valid.
+To copy all three files elsewhere, pass an explicit existing destination with
+`--mirror-dir PATH`; repeat it for multiple destinations. No sibling repository
+paths are discovered or written automatically. Required input failures leave
+existing outputs untouched because all payloads are built before writing.
+Observed source changes during struct inference also abort before writing.

rebuild/sdk/README.md

@@ -0,0 +1,21 @@
+# Historical SDK snapshots
+
+`verified_functions.json`, `struct_schema.json` and
+`forgefse_verified_bindings.json` are historical generated snapshots. They
+predate the exporter eligibility and assurance checks described in the
+[export contract](../../docs/pipeline/VERIFIED_FACTS_EXPORT.md). Their filenames
+and older verification labels do not establish current source freshness,
+behavioral certification, signature validity or independently proven layouts.
+
+The exporter repair does **not** regenerate or revalidate these checked-in
+payloads. In particular, an older `MATCH*` label may reflect the former
+missing-audit fallback. Preserve the original snapshots as historical evidence;
+do not promote their labels to new verification claims.
+
+New exports require explicit recorded `EXACT`/`RELOC` rows and carry an
+`assurance` section documenting their remaining limits. Snapshot source hashes
+identify the exported bytes; legacy audit rows still lack a complete current
+source/dependency proof binding. Mirroring is opt-in through explicit paths.
+
+`fse_native_overlay.json` is produced separately and is outside this exporter
+repair's validation scope.

tools/decomp_pipeline/export_verified_facts.py

@@ -1,93 +1,208 @@
 #!/usr/bin/env python3
-"""Export the byte-verified decomp facts in forms the downstream tools consume:
-  - rebuild/sdk/verified_functions.json  : every genuine byte-match (addr, name, cc, ret, parity)
-  - rebuild/sdk/struct_schema.json        : {class: {size, fields:[{offset,type,name}]}}  (FableForge/FQT def readers)
-  - rebuild/sdk/forgefse_verified_bindings.json : binding-queue targets now byte-verified (ForgeFSE)
-Uses only functions that ACTUALLY byte-match (from the parity audit), never behavior-only lands.
+"""Export recorded parity facts, with explicit assurance limits.
+
+Legacy output filenames are retained. EXACT/RELOC rows are recorded byte-parity
+evidence, not source-freshness, signature, behavioral or class-layout certification.
+Missing/ambiguous evidence is excluded. Default output stays in this checkout;
+copying to another project requires an explicit --mirror-dir destination.
 """
-import csv, json, glob, os, sys
+import argparse
+from collections import Counter, defaultdict
+import csv
+import hashlib
+import json
 from pathlib import Path
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import re
+import shutil
+import sys
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
 import class_struct
+import label_trust
+from crawl import purity
+
+ROOT = Path(__file__).resolve().parents[2]
+ASSURANCE = {
+    "basis": "explicit recorded EXACT/RELOC audit rows, matching source filenames, and crawl.purity filter",
+    "freshness": "not established: legacy audit rows are not bound to current source/dependency hashes",
+    "behavior": "not established by byte parity",
+    "signatures": "manifest metadata, not independently verified by this export",
+    "labels": "manifest metadata; trusted class labels are required for struct inference",
+    "layout": "inferred local declarations with trusted class labels, not independent layout proof",
+}
+
+
+def address(value):
+    value = (value or "").strip().lower().removeprefix("0x")
+    return value.zfill(8) if re.fullmatch(r"[0-9a-f]{1,8}", value) else None
+
+
+def basename(value):
+    return (value or "").replace("\\", "/").rsplit("/", 1)[-1]
+
+
+def tsv(path, *, optional=False):
+    if optional and not path.exists():
+        return []
+    with path.open(encoding="utf-8-sig", newline="") as stream:
+        return list(csv.DictReader(stream, delimiter="\t"))
+
+
+def by_address(rows):
+    grouped = defaultdict(list)
+    for row in rows:
+        if key := address(row.get("address")):
+            grouped[key].append(row)
+    return grouped
+
+
+def build_exports(root):
+    """Read only the selected root; build every payload before writing anything."""
+    root = Path(root).resolve()
+    manifest = label_trust.Manifest(root / "rebuild/manifest/functions.tsv")
+    man = by_address(manifest.rows)
+    audit = by_address(tsv(root / "rebuild/compile-gate/parity_audit.tsv"))
+    queue = tsv(root / "rebuild/backlog/forgefse-binding-queue.tsv", optional=True)
+    sources = defaultdict(list)
+    for path in sorted((root / "rebuild/src/compiled").rglob("*.cpp")):
+        if key := address(path.stem.rsplit("_", 1)[-1]):
+            sources[key].append(path)
+
+    excluded = Counter()
+    eligible = {}
+    snapshots = {}
+    by_class = defaultdict(list)
+    for key, paths in sorted(sources.items()):
+        if len(paths) != 1 or len(man.get(key, [])) != 1 or len(audit.get(key, [])) > 1:
+            excluded["ambiguous_or_missing_identity"] += 1
+            continue
+        if not audit.get(key):
+            excluded["missing_audit"] += 1
+            continue
+        row = man[key][0]
+        proof = audit[key][0]
+        path = paths[0]
+        if proof.get("verdict") not in ("EXACT", "RELOC"):
+            excluded["nonmatching_audit"] += 1
+            continue
+        if basename(proof.get("file")) != path.name or basename(row.get("compiled_source")) != path.name:
+            excluded["source_identity_mismatch"] += 1
+            continue
+        if not path.resolve().is_relative_to(root):
+            excluded["source_outside_root"] += 1
+            continue
+        try:
+            data = path.read_bytes()
+        except OSError:
+            excluded["unreadable_source"] += 1
+            continue
+        if not purity.is_genuine(data.decode("utf-8", errors="replace")):
+            excluded["assembly_source"] += 1
+            continue
+        label_reason = manifest.trust_reason(key)
+        eligible[key] = {
+            "address": f"0x{key}", "name": row.get("name", ""),
+            "module": row.get("module", "_global"),
+            "calling_convention": row.get("calling_convention", ""),
+            "return_type": row.get("return_type", ""),
+            "parameter_types": row.get("parameter_types", ""),
+            "parity": proof["verdict"],
+            "label_trust": label_reason or "TRUSTED_CLASS",
+            "source": path.relative_to(root).as_posix(),
+            # This identifies the exported snapshot; it does not bind the old audit to it.
+            "source_sha256": hashlib.sha256(data).hexdigest(),
+        }
+        if cls := manifest.trusted_module(key):
+            by_class[cls].append(path)
+        snapshots[path] = eligible[key]["source_sha256"]
 
-ROOT = Path(r"D:\Documents\FableTLC")
-def tsv(p):
-    try: return list(csv.DictReader(open(p, encoding="utf-8-sig"), delimiter="\t"))
-    except FileNotFoundError: return []
-
-def main():
-    man = {r["address"].lower(): r for r in tsv(ROOT / "rebuild/manifest/functions.tsv")}
-    audit = {r["address"].lower(): r["verdict"] for r in tsv(ROOT / "rebuild/compile-gate/parity_audit.tsv")}
-    landed = {os.path.basename(p).split("_")[-1].replace(".cpp", "").lower()
-              for p in glob.glob(str(ROOT / "rebuild/src/compiled/**/*.cpp"), recursive=True)}
-    # genuine byte-matches only: audited EXACT/RELOC, plus freshly-landed not-yet-audited (verify_and_land gated)
-    def parity_of(a):
-        v = audit.get(a)
-        if v in ("EXACT", "RELOC"): return v
-        if v is None and a in landed: return "MATCH*"  # landed after last audit; gated by verify_and_land
-        return None
-    verified = []
-    for a in sorted(landed):
-        pt = parity_of(a)
-        if not pt: continue
-        r = man.get(a, {})
-        verified.append({"address": f"0x{a}", "name": r.get("name", ""), "module": r.get("module", "_global"),
-                         "calling_convention": r.get("calling_convention", ""), "return_type": r.get("return_type", ""),
-                         "parameter_types": r.get("parameter_types", ""), "parity": pt})
-    sdk = ROOT / "rebuild/sdk"; sdk.mkdir(exist_ok=True)
-    json.dump({"count": len(verified), "functions": verified}, open(sdk / "verified_functions.json", "w"), indent=1)
-
-    # struct schema from byte-verified accessors/sizes
-    classes = set()
-    for p in glob.glob(str(ROOT / "rebuild/src/compiled/**/*.cpp"), recursive=True):
-        b = os.path.basename(p)
-        if "_" in b: classes.add(b.split("_")[0])
     schema = {}
-    for c in sorted(classes):
-        off_map, size, n = class_struct.facts_for(c)
-        if n < 1 or (not off_map and not size): continue
-        fields = [{"offset": f"0x{o:x}", "type": off_map[o][0], "name": off_map[o][1]} for o in sorted(off_map)]
-        entry = {"verified_methods": n}
-        if size and (not off_map or size >= max(off_map) ): entry["size"] = f"0x{size:x}"
-        if fields: entry["fields"] = fields
-        schema[c] = entry
-    json.dump({"count": len(schema), "classes": schema}, open(sdk / "struct_schema.json", "w"), indent=1)
-
-    # ForgeFSE: binding-queue targets now byte-verified
-    bq = tsv(ROOT / "rebuild/backlog/forgefse-binding-queue.tsv")
-    vset = {v["address"].lower().replace("0x", "") for v in verified}
-    fverified = []
-    for r in bq:
-        a = r.get("address", "").lower().replace("0x", "")
-        if a in vset:
-            fverified.append({"address": f"0x{a}", "api_names": r.get("api_names", ""),
-                              "qualified_name": r.get("qualified_name", ""), "calling_convention": r.get("calling_convention", ""),
-                              "return_type": r.get("return_type", ""), "bridge_kind": r.get("bridge_kind", ""),
-                              "parity": parity_of(a)})
-    json.dump({"count": len(fverified), "note": "Retail address+CC+signature+behaviour byte-confirmed; ready for ForgeFSE owner/vtable + runtime-probe binding step.",
-               "bindings": fverified}, open(sdk / "forgefse_verified_bindings.json", "w"), indent=1)
-
-    print(f"verified_functions: {len(verified)}")
-    print(f"struct_schema classes: {len(schema)}  (with size: {sum(1 for v in schema.values() if 'size' in v)}, with fields: {sum(1 for v in schema.values() if 'fields' in v)})")
-    print(f"forgefse_verified_bindings: {len(fverified)} / {len(bq)} queue targets")
+    excluded_classes = Counter()
+    for cls, paths in sorted(by_class.items()):
+        provenance = {}
+        off_map, size, used = class_struct.facts_for(cls, files=paths, provenance=provenance)
+        if used < 1 or (not off_map and not size):
+            continue
+        if any(len({(typ, name, span) for typ, name, span, _ in rows}) != 1
+               for rows in provenance.values()):
+            excluded_classes["conflicting_fields"] += 1
+            continue
+        end = max((offset + rows[0][2] for offset, rows in provenance.items()), default=0)
+        if size is not None and size < end:
+            excluded_classes["size_conflict"] += 1
+            continue
+        entry = {"verified_methods": used, "inference": "source_declarations"}
+        if size is not None:
+            entry["size"] = f"0x{size:x}"
+        if off_map:
+            entry["fields"] = [
+                {"offset": f"0x{offset:x}", "type": off_map[offset][0],
+                 "name": off_map[offset][1], "size": f"0x{provenance[offset][0][2]:x}"}
+                for offset in sorted(off_map)
+            ]
+        schema[cls] = entry
+
+    bindings = []
+    for row in queue:
+        key = address(row.get("address"))
+        if key not in eligible:
+            continue
+        fact = eligible[key]
+        bindings.append({
+            "address": fact["address"], "api_names": row.get("api_names", ""),
+            "qualified_name": row.get("qualified_name", ""),
+            "calling_convention": fact["calling_convention"], "return_type": fact["return_type"],
+            "bridge_kind": row.get("bridge_kind", ""), "parity": fact["parity"],
+        })
+    for path, digest in snapshots.items():
+        if not path.resolve().is_relative_to(root) or hashlib.sha256(path.read_bytes()).hexdigest() != digest:
+            raise ValueError(f"source changed while exporting: {path.relative_to(root)}")
+    return {
+        "verified_functions.json": {
+            "count": len(eligible), "assurance": ASSURANCE,
+            "excluded": dict(sorted(excluded.items())), "functions": list(eligible.values()),
+        },
+        "struct_schema.json": {
+            "count": len(schema), "assurance": ASSURANCE,
+            "excluded": dict(sorted(excluded_classes.items())), "classes": schema,
+        },
+        "forgefse_verified_bindings.json": {
+            "count": len(bindings), "assurance": ASSURANCE,
+            "note": "Recorded parity targets only. Signature, owner/vtable and runtime binding checks remain required.",
+            "bindings": bindings,
+        },
+    }
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--root", type=Path, default=ROOT, help="checkout to read and write (default: this checkout)")
+    mirrors = parser.add_mutually_exclusive_group()
+    mirrors.add_argument("--no-mirror", action="store_true", help="compatibility flag; local-only is already the default")
+    mirrors.add_argument("--mirror-dir", action="append", type=Path, default=[],
+                         help="explicit existing destination for all three exports; repeat for multiple destinations")
+    args = parser.parse_args(argv)
+    root = args.root.resolve()
+    sdk = root / "rebuild/sdk"
+    for destination in args.mirror_dir:
+        if not destination.is_dir() or destination.resolve() == sdk.resolve():
+            parser.error(f"mirror destination must be an existing directory distinct from {sdk}: {destination}")
+    try:
+        exports = build_exports(root)
+    except (OSError, ValueError, KeyError) as error:
+        print(f"export failed: {error}", file=sys.stderr)
+        return 1
+    sdk.mkdir(parents=True, exist_ok=True)
+    for name, payload in exports.items():
+        with (sdk / name).open("w", encoding="utf-8", newline="\n") as stream:
+            json.dump(payload, stream, indent=1)
+            stream.write("\n")
+        for destination in args.mirror_dir:
+            shutil.copy2(sdk / name, destination / name)
+        print(f"{name}: {payload['count']}")
     print(f"-> {sdk}")
+    return 0
 
-    # Mirror consumable facts into the downstream project repos (like export_fse_native_overlay.py).
-    if "--no-mirror" not in sys.argv:
-        import shutil
-        mirrors = {
-            "struct_schema.json": [Path(r"D:\Code\FableForge\docs\re_reference"), Path(r"D:\Code\FQT\FQT\FSE_Source\docs")],
-            "verified_functions.json": [Path(r"D:\Code\FableForge\docs\re_reference"), Path(r"D:\Code\ForgeFSE\docs"), Path(r"D:\Code\FQT\FQT\FSE_Source\docs")],
-            "forgefse_verified_bindings.json": [Path(r"D:\Code\ForgeFSE\docs")],
-        }
-        n = 0
-        for fname, dests in mirrors.items():
-            for d in dests:
-                if d.is_dir():
-                    shutil.copy2(sdk / fname, d / fname); n += 1
-                else:
-                    print(f"  (skip missing dir {d})")
-        print(f"mirrored {n} files into FableForge / ForgeFSE / FQT doc dirs")
 
 if __name__ == "__main__":
-    main()
+    raise SystemExit(main())