Derive eval decisions from atomic evidence

[vriesd]

Phase 2 of the eval engineering plan adds pure analysis over the strict v2 report boundary. The legacy qualifier, runner, CLI, and workflows remain unchanged.

What changed

• Derives VERIFIED, NOT VERIFIED, and INCONCLUSIVE from frozen required cells and atomic attempts.
• Enforces provider, sample, and pass-rate policy per represented scheduled provider.
• Compares exact caller-supplied artifact, evaluator, host, actor, instruction, requested-model, and actual-model provenance.
• Keeps report-only failures outside release decisions.
• Adds descriptive reviewer detection, false-positive, incomplete-assignment, and Wilson interval metrics.
• Adds opaque paired complete, incomplete, tie, reserve, and arm-win accounting without allocation, direction, or bootstrap claims.
• Records the How, Architect, Arena, independent judge, Interrogate, Deslop, and recovery trail.

Verification

• Focused atomic, advisory, report-boundary, and legacy qualification suites pass 59 tests.
• bun run check passes 448 tests with one intentional live-host skip and zero failures.
• Flow contribution commit and push preflights passed. Gitleaks was unavailable, so the optional local secret scan was skipped.

Canary gating stays in Phase 9. Provenance computation stays in Phase 3. Allocation, directional effects, and bootstrap intervals stay in Phase 7.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ff4963ea24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Validate the model named by an unobserved exception

When the host reports actualModel.kind === "unobserved", any allow-unobserved expectation is accepted without comparing its value. Thus an independently computed expectation can name the wrong provider, model, or revision while compareExpectedProvenance still returns a match and deriveReleaseDecision can return VERIFIED; compare the exception's model value with the expected/requested identity before accepting it.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Enforce a stable artifact for each opaque arm token

When the same opaque arm tokens are reused across paired blocks, this check only requires each block to contain both artifacts, so a report may assign artifact A to opaque-blue in one block and artifact B to it in another and still pass provenance. analyzePairs then pools wins by that token, mixing candidate and baseline outcomes and corrupting the paired result; derive an unordered token-to-artifact association from one block and require it to remain consistent across all blocks.

Useful? React with 👍 / 👎.