Repository navigation
eval(routing): pre-registered routed vs all-opus eval of dojo-router and dojo-flow - #14
Merged
Merged
Conversation
…jo-flow Pre-registration digest (sha256 of everything above the marker): d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a Funded by Cruz's 2026-10-23 ruling with a hard cap of $100 for all eval spend, pilots and judge included. Two studies, each with a frozen rule: dojo-router with and without on an Opus session that delegates (five hr-* cases, one careful), and the dojo-flow build workflow with its default role map against every role on opus (two hf-* tasks, one careful). Cost is agent cost (the eval's cost minus the judge's); quality is hidden acceptance tests that never enter the sandbox, with an Opus file judge as the second reference. The tasks, hidden tests and references were written by an independent model author from evals/routing-2026-10-23/held-out-brief.md without access to this repository. grade_hidden.py selfcheck passes on all seven (scaffold FAILs, reference PASSes; selfcheck-before-freeze.txt). No run has happened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-registration digest unchanged (verify --expect passes): d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a The pilots ($8.48 of the $100 cap, excluded from analysis) found four harness fixes, each of a kind section 6 allows. --trust-plugin; no shell grant (the eval refuses one on this Windows machine because the sandbox gate is off), so agents in both arms run no commands; Workflow granted for dojo-flow; and the files judge moved outside the eval, because its focus: files view arrives empty here. No prompt, criterion, metric, rule or n changed. The dojo-flow pilot's contract gate refused all four builds as coupled before any routed role ran. The main run goes ahead as frozen. analyze.py, stats.py, grade_hidden.py, judge_files.py and judge_runs.py are committed here with their sha256 in the amendment, before any main run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-registration digest unchanged (verify --expect passes): d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a dojo-router (60 runs): routed agent cost 0.617 of all-opus (95% interval 0.570 to 0.665); hidden tests 24/24 in both arms, careful case 6/6 in both. The rule outputs "saves, no measured loss". The saving came from the session following the router skill (30/30 runs) and pinning its own dispatches; the hooks never fired at the top level, and the mod acted once, on a helper's own dispatch. dojo-flow (20 runs): the contract gate refused every build as coupled before any routed role ran, so routing never acted. The rule outputs "inconclusive: routing did not take effect". The gate is given track names and files only, not the track prompts. Spend $52.15 of the $100 cap, pilots included ($8.48). No shell for agents (Windows sandbox gate), as amendment 1 records. READMEs not changed: the wording is Cruz's call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cruz chose (2026-10-24) to add the README sentence the eval's rule allowed: "saves, no measured loss" gives the README leave to cite the measured cost ratio with its conditions. The sentence keeps them: 0.62 (95 percent interval 0.57 to 0.67), 60 runs, five tasks, no shell, Claude Code 2.1.305, dojo-router 0.1.1, the saving coming from the skill rather than the hooks, and what was not measured. The limits bullet that said nothing measures routing now points at it. The plugin's own public-copy tests ban the percent sign, so it says "95 percent". CHANGELOG gains the eval under Unreleased. dojo-router shape tests: 19 run, OK. The full router suite has one failure, test_the_size_cap_is_512_kb_and_the_edge_is_exact, which fails the same way on unmodified origin/main on this machine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pre-registered eval of model routing against an all-opus baseline, built with the
dojo-settlemethod used for the dojo-verify eval (PR #7): freeze the rule by digest in a commit, then run. Funded by Cruz's 2026-10-23 ruling with a hard cap of $100, pilots and judge included. Do not merge without Cruz's go. At Cruz's choice (2026-10-24), dojo-router's README now cites the measured ratio with the eval's conditions (c45e233); dojo-flow's README is unchanged, as its verdict requires.d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a, recorded in7404987and pushed at 21:15:15 UTC, before the first eval call at 21:15:42 UTC.prereg.py verify --expectstill passes.76f2779, pushed before any main run.620b56d.c45e233.spend-ledger.md). All runs drew on the logged-in claude.ai plan; no API key was used.Results
Findings that matter more than the verdicts
dojo-router:routerbefore dispatching and named a model on all 62 of its Agent calls. The classic hook never fired at the top level. The mod acted once, on a helper's own unpinned dispatch, which ran on Sonnet..dojo/dag-contract.mdbehind, so a re-run in the same repository needs a newcontractPath.Limits
Pre-existing test failure: in the full dojo-router suite,
test_the_size_cap_is_512_kb_and_the_edge_is_exactfails on this Windows machine. It fails the same way on unmodifiedmain. The 19 shape and public-copy tests pass.No shell: on this Windows machine the eval refuses shell grants (the sandbox feature gate is off), so no agent could run a command.
One platform and version: Claude Code 2.1.305, dojo-router 0.1.1, dojo-flow 0.1.0.
Model-written tasks: the tasks were written by an independent Sonnet author.
Judge moved outside the eval: the eval's
focus: filesview is empty on this platform, so the files judge ran outside it (amendment 1).Full write-up:
evals/routing-2026-10-23/PREREG-2026-10-23-routing.md.For Cruz to decide
c45e233). The plugin's public-copy tests ban the percent sign, so it says "95 percent interval".🤖 Generated with Claude Code