Repository navigation
feat(dojo-verify): default to warn, per the pre-registered eval - #11
Merged
Merged
Conversation
…ruz's ruling The with/without eval (evals/PREREG-2026-10-17-evidence.md) detected no benefit from blocking and crossed the control cost bar, so its rule says warn. Cruz chose warn on 2026-10-18. mode now defaults to warn and an unknown value falls back to warn; block stays one setting away. The ruling is recorded below the marker, so the frozen digest is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pUc6p2gu3ur7FYwbX6sBc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Cruz · project thread
Before: dojo-verify's Stop hook blocked by default. If a reply claimed success with no passing check behind it, the hook stopped it. The changelog said "the default stays
blockuntil that is decided."After: the hook warns by default. It shows a system message and does not block. An unknown
modevalue also falls back towarn.blockis still available as a setting.The pre-registered with/without eval (
evals/PREREG-2026-10-17-evidence.md) found no benefit from blocking. Claims made without a check fell from 5 in 30 runs to 4 in 30, a difference with a 95% interval of −15 to +22 points. The plugin also cost 1.29 times as much on the control tasks, above the 1.25 limit. Under the pre-registered rule, that meanswarn. Cruz chosewarnon 2026-10-18.How: the default changed in
plugin.jsonand the fallback inget_mode(). The test helper now setsmode=blockexplicitly, since most tests cover the block path, and new tests cover the warn default and the unknown-value fallback. The README modes table,ACCURACY.mdandCHANGELOG.mdare updated. The ruling is recorded below the pre-registration's marker line, and the frozen digest still matches (d5cac861…).Checks run locally: the dojo-verify tests (153 pass),
scripts/hook_accuracy.py(unchanged, since it pinsblock) andscripts/plugin-lint.py(passes, with the same 7 warnings as on main).This replaces TresPies-source/CoworkPluginsByDojoGenesis#2, because DojoGenesis is the canonical copy.
Generated by Claude Code