Empirical evidence about the evaluator proposed by the Judgment Pack Specification's RFC 0006 (Draft), in two tracks that answer two different questions and are judged by different standards:
- Agreement — do independent implementations produce the same result? Clean-room evaluators derived from the specification text alone. Their agreement tests whether the prose actually pins the semantics; their divergences locate the places it does not. Deterministic, offline, CI-verified.
- Efficacy — does representing a policy this way actually help? Preregistered experiments on third-party benchmarks. API-dependent, costly, non-deterministic, and run manually — never in CI.
The two are kept visibly separate on purpose. Conformance evidence is something a specification project is the right author of; efficacy evidence is something it is the wrong author of, and the only remedy is method: preregistration, public harness and data, and reported negative results.
Nothing in this repository claims any JPS conformance. JPS 0.1.0-draft defines no evaluator
conformance class and forbids evaluator-conformance claims outright (Core §3.4). Every
implementation here is experimental, may change or be removed without compatibility promise, and
evaluates nothing normatively: a disposition is data, not an authorization, a decision, or an
executed action.
| Path | What it is |
|---|---|
CLEAN-ROOM-PROTOCOL.md |
The reusable recipe every implementation must follow to count as evidence |
reference/ |
Pinned snapshots of the specification texts the rooms are built from |
harness/ |
The post-hoc referee: runs identical inputs through implementations and diffs dispositions |
python/ |
Agreement track. Implementation #2 (the first lives in judgment-pack-runtime): stdlib-only Python, written clean-room by a coding agent from the reference texts, with its interpretation log in python/DECISIONS.md |
studies/ |
Efficacy, expressiveness, and interoperability track. Preregistered experiments on third-party benchmarks and independently developed systems, from policy expressiveness (001–003) through trustworthy-input lineage (005–008) to receipt-protocol interoperability (013, 014). See the studies index for the full matrix and the open-source projects each one builds on. |
docs/adr/ |
Decision records for this repository — why a given study is being run, and what comes next |
Future implementations (TypeScript, Rust, …) get sibling directories — but the clean room itself never happens inside this repository: an implementer working here could read the existing implementations. See the protocol.
Navigation and status only — every finding, and every qualification that finding carries, lives in the study's own directory. Statuses below are the ones the studies index records; it remains the canonical matrix, with the external source each study builds on.
| Study | Track | Status | Evidence |
|---|---|---|---|
| 001 | Expressiveness / efficacy | Run — H1 not supported: the pack arm scored lowest on the registered endpoint (one of the two registered model families ran); expressiveness result | RESULTS-FIRST-PROMPT-ARMS.md, DEVIATIONS.md, Results |
| 002 | Expressiveness | Completed expressiveness result | Results, study |
| 003 | Expressiveness | Completed escape census | Results, study |
| 004 | Composition | Completed composition-closure study; no efficacy claim | RESULTS.md, run-log.md |
| 005 | Trustworthy input | Completed semantic-source-discovery study | README.md, RESULTS.md, ANALYSIS.md |
| 006 | Trustworthy input | Deterministic phase completed; model-authoring phase terminated before inference because of infrastructure, so model usability is not estimable | README.md, RESULTS.md |
| 007 | Trustworthy input | Completed evidence-lineage model replication | README.md, RESULTS.md, ANALYSIS.md |
| 008 | Trustworthy input | Completed offline replay and probe with no model runs; read with the study's stated qualifications | README.md, RESULTS.md, ANALYSIS.md |
| 009 | Blinded authorship | Completed corrected retained attempt as a constructed existence witness — not discovery evidence, not general pipeline validation | README.md, RESULTS.json, ANALYSIS.md |
| 010 | Blinded authorship | Run (frozen) | ANALYSIS.md, RESULTS.json |
| 011 | Blinded authorship | Run (frozen) | ANALYSIS.md, RESULTS.json |
| 012 | Blinded authorship | Frozen + run — R1 unsupported; retracts a published claim | ANALYSIS.md, CORRECTION.md |
| 013 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 014 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 015 | Interoperability | Frozen + run — R1 holds (27 cells); reviewer holdout diverged on 7 of 8 cells | README.md, ANALYSIS.md, results/ |
| 016 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 017 | Currency governance | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 018 | Currency governance | Frozen + run — R1 holds; reviewer holdout diverged on three preregistered cells | ANALYSIS.md, results/ |
| 019 | Blinded authorship | Frozen + run — R1 inconclusive (control gate failed: E1 floor) | ANALYSIS.md, results/ |
| 021 | History replay | Run — R1 holds (131/131, holdout 8/8) | PREREGISTRATION.md, PREREG-REVIEW.md |
| 022 | Outside-format binding | Run — R1 holds (19/19, holdout 6/6) | PREREGISTRATION.md, PREREG-REVIEW.md |
This repository claims no JPS conformance for anything in it, and the table above adds no aggregate headline: each study answers a different preregistered question and must be read with its own qualifications.
- 13/13 semantic agreement between the Go reference runtime's experimental evaluator
(v0.2.0) and
python/on RFC 0006's nine appendix instances plus three probes — identical kind, outcome, reason set, and handoff state (harness/README.md). - Draft RFC 0016 (outcome values): 66 rows, 62 where both implementations give the RFC's
answer, a disposition byte for byte or an error class. Three more agree with each other and
not with the RFC, for a reason the RFC's own Compatibility section gives. One differs: what a
carrier does with an unpaired surrogate, which Core does not say
(
harness/RFC0016-AGREEMENT.md). CI runs the comparison against the pinned runtime. - Two specification gaps found and recorded as RFC 0006 unresolved questions: number
representability, and the disposition's concrete JSON serialization (the two implementations
agreed on all semantics while serializing
handoffincompatibly — seepython/DECISIONS.mdentry 3).
Study 001's comparison has run, and on the preregistered endpoint the pack arm loses. All
three arms ran over the full corpus at k = 5, with one of the two registered model families. On
the registered answerable population (216 instances) pass^5 was A 0.727, A′ 0.778 and B 0.579:
B − A = −0.148, 95% CI [−0.213, −0.088], the opposite sign to H1. H5 (B against A′) and H4
(accuracy) are not supported either, and H3 (citation) fails. H2 (escalation) meets its registered
rule over all 432 twins, with false escalations on 27.8% of answerable trials against arm A's 1.1%.
The study traces the endpoint loss to 60 answerable instances the pack cannot decide, because the
benchmark omits a schedule the policy needs. An earlier draft reported the opposite by scoring the
wrong population.
RESULTS-FIRST-PROMPT-ARMS.md
has the tables and DEVIATIONS.md the
corrections.
Later studies have retained results; each answers a different preregistered question and must be read with its study-level qualifications. Study 002 ran no comparison arm. The expressiveness results below are on deliberately opposite policy types and by different model families:
| Study 001 — CBA, arithmetic-dense | Study 002 — airline, qualitative | Study 003 — census, 12 decisions | |
|---|---|---|---|
| Prepared facts | 124 | 5 | 58 |
| Prepared determinations | 13 | 1 | 40 |
| Decisions with ≥1 determination | 1/1 | 1/1 | 12/12 |
The escape is universal in the census frame because real request inputs are collections — and one device dominates: quantification over collections forced 25 of 40 determinations (arithmetic: 6). Encoders were hypothesis-blind; classification was two-way independent with zero fact-class disagreements. See Study 003, Study 002 and RFC 0007.
Apache-2.0. This repository is maintained alongside, but is not part of, the normative specification; the specification repository owns the standard and its conformance corpus.