Cut release qualification over to v2 evidence - #36
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc62d89274
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return { | ||
| kind: "release", | ||
| artifact, | ||
| evaluator: first.evaluator, |
There was a problem hiding this comment.
When the qualifier is given a valid but different catalog with the same case IDs, this copies the evaluator identity directly from the report, so provenance comparison is tautological and never checks that evaluator.policyCatalogSha256 matches the explicitly supplied catalog. Because that catalog controls release, minProviders, minScoredAttempts, and minPassRate, accidentally swapping in—or deliberately supplying—a weaker catalog can produce VERIFIED for evidence evaluated under another policy; recompute and compare the catalog provenance digest before deriving the decision.
Useful? React with 👍 / 👎.
| if ( | ||
| expectedArtifact && | ||
| canonicalJson(entry.artifact) !== canonicalJson(expectedArtifact) | ||
| ) { |
There was a problem hiding this comment.
When a VERIFIED decision is generated at commit C and then committed for release at commit D, this full-object comparison can never succeed: inspectArtifact records the current Git HEAD as sourceCommit, so the committed record names C while the release tarball rebuilt from D names D (and committing the decision also changes the source-tree digest). Since the release workflow only scans committed decisions, every major release will reject every legitimately generated decision before reaching the canary; the comparison needs an artifact identity that can remain stable across the evidence-record commit, or another non-circular handoff.
Useful? React with 👍 / 👎.
| - name: Apply release-qualification thresholds | ||
| if: steps.matrix.outputs.models != '' | ||
| run: bun run qualify | ||
| id: qualify | ||
| if: always() && steps.matrix.outputs.models != '' && steps.run.outputs.report != '' | ||
| continue-on-error: true |
There was a problem hiding this comment.
In the scheduled/dispatch matrix-evals job, a NOT VERIFIED or INCONCLUSIVE result exits this step nonzero, but continue-on-error: true converts that into a successful conclusion and the following upload step normally succeeds; the eval-run step is also allowed to fail, and there is no final step that re-raises either outcome. Consequently a failed release qualification leaves the workflow green, masking regressions from the schedule; retain the continuation for artifact publication but add a final gate that fails from the recorded qualifier outcome.
Useful? React with 👍 / 👎.
2 participants
Phase 5 removes summary-only evidence from release authority and switches scheduled eval and publication safety to explicit v2 inputs.
Verification: actionlint and bun run check pass 473 tests, one intentional skip, zero failures. Four-model Interrogate found no unresolved blocker. Contribution preflights passed; optional gitleaks scan unavailable.