diff --git a/CHANGELOG.md b/CHANGELOG.md index df92246..c88666f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,10 @@ method, the rubric and the tables. It started from five misfires reported from o `plugins/dojo-verify/evals/PREREG-2026-10-17-evidence.md`, frozen before any run, with eight held-out cases. Its result, below the marker: no benefit detected, and its rule says the default should be `warn`. Following that rule (Cruz's ruling, 2026-10-18), the default is now `warn`; `block` stays one setting away. +- **A pre-registered eval of routing against all-opus** — `evals/routing-2026-10-23/`, frozen before any run, with + seven held-out tasks. dojo-router: an Opus session that delegates spent 0.62 as much with the plugin as without, + with no measured quality loss, and its README now says so with the eval's conditions. dojo-flow: inconclusive, + because its contract gate refused every build before any routed role ran. - **A runtime health report** — `scripts/health_report.py` replays local Claude Code transcripts through the suite's own detector code and reports, per rule, what it counted and out of how many, with each finding's evidence and each detector's measured accuracy beside it. The same input gives byte-identical output. diff --git a/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md b/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md new file mode 100644 index 0000000..d809c70 --- /dev/null +++ b/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md @@ -0,0 +1,513 @@ +# Pre-registration: does routing change cost or quality against all-opus? + +Everything above the marker line near the end of this file is the pre-registration. Its digest was recorded with +`prereg.py freeze` and committed before any run. Results go below the marker, so adding them does not change the +digest. + +## 1. What is decided + +Whether the READMEs of dojo-router (0.1.1) and dojo-flow (0.1.0) may state a measured effect of routing on cost or +quality. Both say today that nothing measures what routing changes and claim no savings. That stance stays unless +the rule below says otherwise. Per plugin, the rule's verdict maps to what its README may say: + +- **Saves, no measured loss:** the README may cite the measured cost ratio with its conditions (tasks, runs, date, + Claude Code version, plugin version) and a link to this file. +- **Saves, quality not resolved:** the README may cite the cost ratio and must say the quality difference was not + resolved at this sample size. +- **Saves at a quality cost:** the README cites both, the saving and the loss. +- **No detectable difference in cost:** the README keeps "no savings claimed" and may add that a pre-registered eval + found no detectable difference on these tasks. +- **Costs more:** the README says routing did not save on these tasks. +- **Inconclusive** (routing did not take effect, the study could not run in this harness, or too few runs + finished): the README stays as it is, and this file says why. + +The decision is Cruz's. This experiment produces the recommendation. Funding: Cruz's ruling of 2026-10-23 (item 4 +of `handoffs/2026-10-23_windows-decisions-for-cruz.md` in TresPies-AI-Orchestration), a hard cap of $100 for all +eval spend, pilots and judge included. + +## 2. Incumbent, measured here + +The incumbent is all-opus, measured in this harness, never quoted. + +- **dojo-router:** the session runs on Opus with no routing plugin, so a subagent dispatch that names no model runs + on the session's model. That is the without arm of a `--ablation with-without` run. Whether Claude Code itself + pins some subagents to a smaller model when no plugin intervenes is measured, not assumed: the share of subagent + runs that used an Opus model in this arm is reported. +- **dojo-flow:** the same build workflow, same goal, tracks and gates, with every role's model set to `opus` + through `args.models`. Effort per role stays at the workflow's defaults in both arms, so only the model changes. + +## 3. References and confound + +- **Cost reference:** the eval's own per-run cost minus the judge's cost (`costUsd` minus `judge_cost_usd`), called + agent cost below: the executor session plus every subagent and workflow agent it started. It is the list-price + equivalent that Claude Code computes. The runs bill to the logged-in claude.ai subscription, so it measures usage, + not an invoice. +- **Cost attribution (second view):** the model id and token use of each subagent and workflow agent, read from + the run transcripts kept with `--keep-temp`. It shows whether routing took effect and where the money went. If + the transcripts turn out not to be kept or readable, this view is "not measured" and the cost reference stands + alone; the arms differ only in routing, so a cost difference is still attributable to it. +- **Quality, primary:** hidden acceptance tests. The independent author wrote them with each task; they are kept + outside the plugin directories and are never in the agent's sandbox. `grade_hidden.py` runs them on each kept + work directory after the run. Deterministic. +- **Quality, independent second:** `quality-judge`, an `llm` grader with `focus: files`, judged by Opus + (`--judge-model opus`) against the case's expected outcome. It reads files only, not the trace, so the router's + warning text in the trace can't prime it. +- **Confound the judge brings:** it is Opus grading code that, in the all-opus arm, Opus wrote. A model tends to + favour its own output, so the judge could lean toward the all-opus arm. The hidden tests can't lean either way, + so a quality loss counts only when the hidden tests show it, and the judge must agree in sign. +- **Confound in the task set:** the dojo-router prompts ask the agent to use subagents. Without that request an + Opus session might not dispatch at all, and routing would have nothing to act on. The result applies to sessions + that delegate, not to every session. +- **Not references:** `dispatched` (an Agent call happened), `hook-warned` (the router's warning appears in the + trace, with-only), `workflow-ran` and `scorecard-relayed` (dojo-flow ran and its verdict came back). They show + what happened and enter no metric. + +## 4. The rule and the bars + +### Decision rule + +Definitions, for each study (R: dojo-router; F: dojo-flow) separately: + +- **A run counts** when the eval reports an agent cost for it and `grade_hidden.py` returns PASS or FAIL for its + work directory. A run that hits `max_turns` or times out counts and is graded on what it left. A harness error + with no agent output (for example a rate limit) and a hidden-test COULD NOT RUN are missing: reported by cell, + never re-run, never imputed. +- **Agent cost** c = `costUsd` − `judge_cost_usd` for the run. +- **Pooled mean cost per arm** = the mean over the study's cases of each case's mean c, so every case weighs the + same. Cost pools every case of the study, the careful case included. +- **Cost ratio** ρ = routed pooled mean / all-opus pooled mean, with a 95% percentile bootstrap interval: + resample runs with replacement within each case and arm, 10,000 resamples, seed 20261023. +- **Quality** = the hidden-test pass rate per arm, pooled over the study's standard cases (the careful case is + judged separately, below). F has only one standard task, so F's quality pools both of its tasks, and the + careful-case check still applies to the careful task alone. Δq = routed rate − all-opus rate, with a 95% + Newcombe hybrid score interval (method 10). Δj is the same difference for `quality-judge` (a run passes when + the judge's verdict is PASS). +- **Routing took effect** when, in the routed arm, at least half of the subagent and workflow-agent runs used a + non-Opus model, and that share is at least 25 points above the all-opus arm's. If the transcripts can't show it, + this is "not measured". + +The rule, applied per study: + +1. **Cost.** Savings detected when ρ's interval lies entirely below 1. Increase detected when it lies entirely + above 1. Otherwise no detectable difference. +2. **Quality.** Loss detected when Δq's interval lies entirely below 0 and Δj ≤ 0. Non-inferior when Δq's lower + bound is −0.15 or above and Δj ≥ −0.15. Otherwise not resolved. (A gain, Δq's interval entirely above 0 with + Δj ≥ 0, is reported as unexpected.) +3. **Careful case.** Harm flagged when the routed arm has at least 2 fewer hidden-test passes than the all-opus + arm on the careful case, out of that cell's runs. +4. **Verdict.** Savings and non-inferior: "saves, no measured loss". Savings and not resolved: "saves, quality not + resolved". Savings and loss: "saves at a quality cost". No detectable difference: "no detectable difference in + cost". Increase: "costs more". A flagged careful-case harm is added to any verdict and must appear in the + README text the verdict allows. +5. **Inconclusive** overrides 4 when: fewer than half of the planned runs (after any reduction in section 6) + counted in either arm; or no savings were detected and routing did not take effect or was not measured (a null + where routing may never have acted says nothing about routing); or the study could not run in this harness + (section 6, pilot). In F the workflow script sets each agent's model from the map itself, so routing counts as + having acted there unless the transcripts show otherwise. + +### Numeric bars + +- n: R runs 5 cases × 6 runs × 2 arms = 60 runs (24 per arm on standard cases, 6 per arm on the careful case). + F runs 2 tasks × 5 runs × 2 arms = 20 runs (10 per arm). The reduction ladder in section 6 may lower these + before a main run starts, never after. +- Cost bar: ρ's 95% interval entirely below 1.00 (savings) or entirely above 1.00 (increase). +- Quality bars: loss when Δq's 95% interval is entirely below 0; non-inferiority margin −0.15. +- Careful-case harm bar: 2 or more fewer hidden-test passes in the routed arm. +- Manipulation bar: at least 50% of routed-arm subagent runs on a non-Opus model, and 25 points above all-opus. +- Null: ρ's interval includes 1. Keep "no savings claimed". +- Power, stated in advance. Quality: with 24 runs per arm and both arms passing nearly every run, the Newcombe + interval for Δq is about ±0.14, so non-inferiority at −0.15 can only be shown when both arms nearly always pass; + only a large drop is detected as a loss (from a perfect all-opus arm, 24 of 24 against 18 of 24 gives an interval + of −0.45 to −0.06, so about 25 points; from a lower start, more). F's 10 runs per arm can detect only a drop of + about 40 to 50 points and cannot show non-inferiority at all (10 of 10 in both arms gives ±0.28), so F's quality + outcome can only be "loss" or "not resolved". Cost: the spread between runs is unknown until the pilot. If + routing moves most subagent spend off Opus, a ratio near 0.7 should resolve at these n; a ratio near 0.95 + probably won't, and a null then does not show there is no saving. + +## 5. What would refute the option you want + +The option the builder wants is that routing saves money without hurting quality; that is why both plugins +exist. It is refuted, in plain terms, by any of these: + +- ρ's interval includes 1 or lies above it: no saving shown on these tasks. +- The hidden tests show a quality loss (Δq's interval entirely below 0). +- The careful case flags harm. +- In R, the all-opus arm already runs half or more of its subagents on a non-Opus model: Claude Code's default + already routes, and the plugin has little left to do. +- The routed arm saves only by doing less (fewer subagents, fewer turns) while its hidden-test pass rate falls. + +## 6. Frozen grid and held-out tasks + +### Frozen grid + +Two studies, each run once. R runs first, then F. + +**Study R (dojo-router)** + +``` +claude plugin eval plugins/dojo-router --case 'hr-*' --runs 6 --ablation with-without --scaffold --keep-temp \ + --no-publish --judge-model opus --allow-tools Bash Edit Write --max-cost-usd -j 2 \ + --json /r-main.json --output-dir /r-main +``` + +- Cases: `hr-note-cache`, `hr-page-size`, `hr-reminder-every`, `hr-retry-limit` (standard); `hr-limiter-starve` + (careful). +- Executor `model: opus` in every case, `max_turns` 16, `timeout_seconds` 900. Allowed tools: Read, Edit, Write, + Bash, Glob, Grep, Agent, Skill. +- dojo-router 0.1.1 at the commit that adds this file, every option at its default (`mode` warn, `default_tier` + sonnet, `explore_tier` haiku, `auto_route` true, `opus_cap` 0). The operator's shell sets + `CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1`, so the mod tier routes if the eval passes that variable to its child; + whether it does is pilot check P2. Whichever tier acts is the with arm, and the write-up names it. + +**Study F (dojo-flow)** + +``` +claude plugin eval plugins/dojo-flow --case 'hf-*' --runs 5 --ablation none --scaffold --keep-temp \ + --no-publish --judge-model opus --allow-tools Bash Edit Write --max-cost-usd -j 2 \ + --json /f-main.json --output-dir /f-main +``` + +- Cases: `hf-gradebook-routed`, `hf-gradebook-allopus` (standard task); `hf-semver-ranges-routed`, + `hf-semver-ranges-allopus` (careful task). Each task has two case files, `-routed` and `-allopus`, identical + except the `models` value in the prompt. Routed passes the workflow's own defaults explicitly (scout haiku, builder sonnet, integrator sonnet, + reviewer sonnet, checker haiku, judge opus); all-opus passes `opus` for all six roles. +- Executor `model: opus`, `max_turns` 12, `timeout_seconds` 1800. Allowed tools: Read, Edit, Write, Glob, Grep, + Bash, Skill, Workflow (Edit and Write are there for the workflow's builders; the prompt tells the session itself + not to edit). dojo-flow 0.1.0 at the commit that adds this file. +- The evaluator's prompt, the same for every F case apart from the args block: + > Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow tool + > with name "dojo-flow:build"; if that name is not known, call it with scriptPath set to the plugin's + > workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not change + > the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to finish, then + > reply with its scorecard exactly as returned. + +**Hidden tests.** After each main run, `grade_hidden.py grade ` for every run. Before the +freeze, `grade_hidden.py selfcheck` was run on every case: the scaffold as written must FAIL the hidden tests and +the scaffold with the author's reference laid over it must PASS (the known-bad and known-good inputs). + +**Pilot first, excluded from analysis.** The same two commands with `--runs 1`: R is 10 runs with a cap of $5, F is +4 runs with a cap of $12. Its only purposes: + +- P1: the plugin loads (`suite.plugins` lists it with an empty problems list). +- P2: in R's with arm, which tier acted: the mod (dispatches that named no model ran on a non-Opus model) or the + classic hook only (its warning in the trace), or neither. +- P3: `--keep-temp` keeps each run's work directory and transcripts, and `grade_hidden.py` runs on them. +- P4: every grader runs without error. +- P5: in F, the child can call the Workflow tool, the workflow reaches a scorecard within the run, and its agents' + cost is inside the run's cost. +- P6: agent cost and judge cost per run, per study. + +If a check fails and the fix touches no prompt, grader, metric, rule or n (a flag, a path, a granted tool, a field +the eval reads differently than assumed), make it, record a dated amendment below the marker, and pilot that study +again; the re-pilot counts against the cap. If F's P5 can't be fixed that way, F is not run, and its result is +"could not run in this harness" with the pilot evidence. If P2 shows neither tier acting, R still runs and the +write-up says so; the rule's manipulation bar then decides how a null reads. The analysis script that implements +section 4 is written after the pilot and committed, with its sha256 below the marker, before any main run. + +**Stop rule and cap arithmetic.** The cap is $100 for every `claude plugin eval` invocation: pilots, re-pilots, main +runs, judge included. Each invocation's top-level `costUsd` goes into `spend-ledger.md` as soon as it ends. + +- Every invocation gets `--max-cost-usd` before it starts. The eval checks its cap before it launches each run, so + it can overrun by the runs in flight (2 at `-j 2`). Reserve for that: 2 × the largest pilot cost per run in that + study × 1.5. +- Pilot caps: R $5, F $12. +- Main cap R = min(1.3 × R's pilot cost per run × 60, $35, $100 − all spend so far − R's reserve). +- Main cap F = min(1.3 × F's pilot cost per run × 20, $100 − all spend so far − F's reserve). +- If a study's projected main cost (pilot cost per run × planned runs) doesn't fit its cap, lower runs per case + before it starts, in this order, until it fits: F to 4, R to 5, F to 3, R to 4. If F doesn't fit at 3, F is not + run, and the write-up says it wasn't for budget. +- If a cap trips mid-run: stop, analyse the runs that finished, report them as partial. No top-up runs, no re-runs + of single cases. + +**Cost estimate, before any run.** Guesses, at list prices of $4 / $20 per million input / output tokens for Opus +5.5 and $2 / $10 for Sonnet 5.5 (Haiku's price was not looked up): an R run of up to 16 turns with two or three +subagents at about $0.25–0.50 including the judge, so $15–30 for R's main run; an F run (about ten workflow agents +on three tiers, or all on Opus) at about $1.50–4, so $30–80 for F's main run, which is why the reduction ladder +exists. Pilots about $4 and $10. Not counted in the cap: the task author's and this session's own usage, which +build the eval but are not eval runs. + +### Baseline arm (without) + +- **R:** produced by `--ablation with-without`: the same cases, model and flags, with dojo-router absent. Every + case pins the executor to `opus`, and each eval child starts from a fresh config directory and home with no user + plugins or settings, so an unpinned dispatch in this arm runs on Opus unless Claude Code itself picks another + model, which the transcripts show. +- **F:** `--ablation none`. The all-opus arm is the `-allopus` case of each task, run beside the `-routed` case + with the same flags. Both arms load dojo-flow, because the workflow under test lives in it. + +The headline is the routed-minus-all-opus difference in each study, never one arm's number alone. + +### Held-out task author + +An independent model instance (Claude Sonnet 5.5, a subagent of the session building this eval) given only +`held-out-brief.md`, pasted into its instructions, and told not to read any repository, plugin or file outside its +output folder. It wrote the prompts, scaffolds, hidden tests and reference solutions, and for F the goal, tracks +and gates, and it checked that each hidden test fails on the scaffold and passes on its reference. It is not the +builder of either plugin, but it is a model, not a person. The evaluator (the same building session) wrote the +graders, F's instruction wrapper and this rule, and changed none of the author's text except where an amendment +below says so. Cruz accepted a model author for the dojo-verify eval on 2026-10-17; this eval uses the same kind +of author under the 2026-10-23 funding ruling, and the write-up names it as a handicap. + +### Task where the protocol should lose + +One careful task per study, written to make routing lose. + +- **R:** `hr-limiter-starve`. A token bucket starves clients that retry or poll more often than one token + interval, and the prompt asks the agent to have a subagent review the bucket and find the real cause; the + visible tests pass before the fix, and a plausible fix that banks idle time still fails the hidden tests. The + work the prompt asks the agent to hand to a helper needs judgment, not lookup. Routed, that helper runs on + Sonnet (a general-purpose dispatch goes to `default_tier`) or Haiku (an Explore dispatch), where all-opus gives + it Opus. Routing should do no better here and may do worse. +- **F:** `hf-semver-ranges`. The `semver` track must implement full Semantic Versioning 2.0 precedence; a quick + version passes the 10 visible tests and fails 11 hidden ones. That track's builder must get a subtle rule right. + Routed, the builder is Sonnet; all-opus, it is Opus. + +If the routed arm does better than all-opus on a careful case by 2 or more hidden-test passes, suspect the harness +or the hidden tests before reading anything into it. + +MEASUREMENTS BEGIN BELOW THIS LINE + +## Status note before any spend (2026-10-23 21:15 UTC) + +The digest above, `d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a`, was recorded at +21:14:21 UTC, committed as `7404987`, and pushed to `refs/heads/eval/routing-2026-10-23` at 21:15:15 UTC, before +the first eval invocation at 21:15:42 UTC. The digest and the cap arithmetic went to Cruz in the session's first +status message, and into the shared notes of the session group, before anything was spent. + +## Amendment 1 (2026-10-23, after the pilots, before any main run) + +Four fixes, each of the kinds section 6 allows (a flag, a granted tool, a field the eval reads differently than +assumed). No prompt, criterion, metric, rule or n changed. The pilots (ledger rows 1 to 6, $8.48 in all) are +excluded from analysis. + +1. **`--trust-plugin` added** to both commands. Without it the eval stops at an interactive trust prompt that a + non-interactive shell can't answer. The plugins are this repository's own code. +2. **No shell for any agent.** The first R pilot started no agent and spent $0.12 (the in-eval judge only): on + this Windows machine the eval refuses any run that grants a shell tool, because the Windows sandbox is behind a + feature gate that is off for this session and the eval sets `failIfUnavailable`. The grants are now + `--allow-tools Edit Write` for R and `--allow-tools Edit Write Workflow` for F. Agents and workflow agents in + both arms can read, search, edit, write and dispatch, but can run no command: no tests, no gates, no git, no + `pwd`. This holds for both arms alike. `hr-reminder-every`'s "run them when you're done" can't be followed, and + F's integrator and audit can't run the gates, so no F scorecard can say VERIFIED. Read every result here as + what routing does in a session without a shell. The hidden tests run outside the eval and are unaffected. +3. **Workflow granted in F.** It is a gated tool. The first F pilot withheld it, and no workflow ran ($0.54). +4. **The files judge runs outside the eval.** On this machine the eval hands the `focus: files` judge an empty + view ("(no file changes)") even when the agent changed files: in the R pilot it voted FAIL three times on all + 10 runs, all of which pass the hidden tests. `judge_files.py` runs the same grader after the eval: the same + criteria string, read from each case.yaml; Opus; three votes, the majority deciding; and files only, never + the trace. The files are the run's changes against a fresh scaffold. Planted inputs: scaffold plus reference + PASS 3 of 3; the scaffold untouched FAIL 3 of 3. On the 10 R pilot runs it voted PASS 3 of 3 on all 10, + agreeing with the hidden tests. Each vote is `claude -p --model opus` on the same claude.ai login, with no + tools, setting sources, MCP servers or slash commands; a probe showed no user hook reaches it. The in-eval + `quality-judge` still runs (about $0.006 a run) and its verdicts are void; Δj uses `judge_files.py`. Its + spend is outside `claude plugin eval`, so it is logged in the ledger and counted against the $100. + +**Pilot checks.** + +- P1: both plugins load; `suite.plugins` lists dojo-router 0.1.1 and dojo-flow 0.1.0, with no problems listed. +- P2: in R's with arm, the session named a model on every Agent dispatch by itself, so there was no unpinned + dispatch for either tier to act on (`hook-warned` 0 of 5). 8 of 9 with-arm subagents ran on + `claude-haiku-5-5` or `claude-sonnet-5-5`; on the careful task the session chose Opus for its helper. In the + without arm, all 11 subagents ran on `claude-opus-5-5`, Explore included, so the baseline is all-Opus as + assumed. What made the with arm name models is not isolated here; the plugin's `router` skill is in its + context, and its hooks never fired. +- P3: each kept run holds its work directory (`home/cwd`) and transcripts with exact model ids + (`config/projects/.../subagents/*.jsonl`; F's workflow agents under `subagents/workflows/wf_*/`). + `grade_hidden.py` runs on them: 10 of 10 R pilot runs PASS. +- P4: every grader runs; `quality-judge` is void, as item 4 says. +- P5: in F the Workflow tool works once granted, and the workflow reached a scorecard in 4 of 4 runs. Run costs of + $0.75–0.88 are well above what the session's own transcript accounts for, consistent with the workflow agent's + cost being inside the run. **But in all 4 the contract gate (role `judge`, Opus in both arms) returned REFUSED: + it found the two tracks coupled, because one calls the other's functions and the visible tests import both + modules. Only the contract agent was dispatched, so no routed role ran.** Two of the refusals also say the gate + was never given the track prompts. That is true of `build.js`: the contract prompt lists track names and owned + files only, while asking the gate to check that "each track prompt matches the code". Under section 6, P5 passes + (the workflow runs and reports in this harness), so F's main run goes ahead as frozen. If the gate keeps + refusing, the rule's manipulation bar decides how that reads. dojo-flow is the artifact under test and is not + changed here. +- P6, cost per run: R $0.371 in the eval (the largest single run $0.549), plus $0.081 for the files judge. F + $0.797 (the largest $0.881). + +**Caps, by the stop rule.** Spent so far: $8.4803. + +- R reserve = 2 × 0.549 × 1.5 = $1.65. R main cap = min(1.3 × 0.3709 × 60 = $28.93, $35, 100 − 8.48 − 1.65 = + $89.87) = **$28.93**. Projected: $22.26 for the eval and $4.86 for the files judge. Runs stay at 6 per case. +- F reserve = 2 × 0.881 × 1.5 = $2.64. F main cap = min(1.3 × 0.7968 × 20 = $20.72, 100 − spend at that point − + 2.64), worked out from the ledger before F starts. Projected: $15.94 for the eval and about $1.60 for the files + judge. Runs stay at 5 per case. +- Projected total, pilots included: about $53. + +**Analysis code, frozen before any main run** (sha256 of each file as committed with this amendment): +- `d6498c583dd8fe408747ccd622da19113e0903c5eda5ae1950f9d587922bd33c` `analyze.py` +- `811595b01977df1277df9b8609c13f4a0b94f5c8202cd1fd8e3e122676ea8961` `stats.py` +- `86a1fdf9ba6d73020d6e5ab5788d3640e647f2f5c23d5d325cf242634a2cfd66` `grade_hidden.py` +- `aef332021ec17eef39e5c0cb1b300e67a2932437173f4a97c0b65072391a61a5` `judge_files.py` +- `24f82686ef6bd94f6af10eaadaa38770455c6d0c9709a0d422f68a5f06f498bb` `judge_runs.py` +- `7d27f380fe1c66048d032d8b6af6d48f5faecec93a558625346bcc3b87d5f085` `build_cases.py` + +## Results + +Both main runs used the tree of commit `76f2779` (the pre-registration plus amendment 1), Claude Code 2.1.305 on +Windows 11, the frozen grids with the amendment 1 flags, and the logged-in claude.ai account. Every session +transcript in both studies ran on `claude-opus-5-5`. Per-run rows: `results-2026-10-23-main.json`. Spend, run by +run: `spend-ledger.md`. + +### Study R (dojo-router) + +The run started 2026-10-23 at 21:38:50 UTC and took 2,374 s. All 60 runs finished with no errors: $21.86 against a +cap of $28.93, not partial. The files judge then cost $4.74 (60 runs × 3 Opus votes). Every run counted. + +| case | arm | runs | mean agent cost (sd) | hidden pass | judge pass | mean turns | helpers off Opus | mean wall s | +|---|---|---|---|---|---|---|---|---| +| hr-note-cache | routed | 6 | $0.311 (0.049) | 6 | 6 | 10.8 | 12 of 12 | 68 | +| hr-note-cache | all-opus | 6 | $0.494 (0.072) | 6 | 6 | 9.8 | 0 of 13 | 84 | +| hr-page-size | routed | 6 | $0.188 (0.026) | 6 | 6 | 7.8 | 11 of 11 | 55 | +| hr-page-size | all-opus | 6 | $0.355 (0.005) | 6 | 6 | 6.0 | 0 of 12 | 56 | +| hr-reminder-every | routed | 6 | $0.233 (0.019) | 6 | 6 | 8.3 | 19 of 19 | 62 | +| hr-reminder-every | all-opus | 6 | $0.477 (0.017) | 6 | 6 | 6.2 | 0 of 24 | 72 | +| hr-retry-limit | routed | 6 | $0.243 (0.042) | 6 | 6 | 7.2 | 14 of 14 | 68 | +| hr-retry-limit | all-opus | 6 | $0.440 (0.093) | 6 | 6 | 5.5 | 0 of 13 | 80 | +| hr-limiter-starve (careful) | routed | 6 | $0.388 (0.096) | 6 | 6 | 7.5 | 7 of 11 | 111 | +| hr-limiter-starve (careful) | all-opus | 6 | $0.441 (0.041) | 6 | 5 | 8.7 | 0 of 7 | 128 | + +- Pooled mean agent cost: routed $0.2726, all-opus $0.4415. ρ = 0.617, 95% bootstrap interval 0.570 to 0.665. + Per case: 0.63, 0.53, 0.49, 0.55, and 0.88 on the careful case. On the four standard cases alone, 0.552 (0.511 + to 0.593). +- Hidden tests, standard cases: routed 24 of 24, all-opus 24 of 24. Δq = 0, Newcombe interval −0.138 to +0.138. +- Files judge, standard cases: 24 of 24 in both arms. Δj = 0. +- Careful case: hidden tests 6 of 6 in both arms. The judge disagreed with the hidden tests once: one all-opus run + passed every hidden test, and the judge failed it 2 votes to 1. +- Helpers (subagent transcripts): 63 of 67 off Opus when routed, 0 of 69 in all-opus. +- What the sessions did: in all 30 routed runs, the session invoked the `dojo-router:router` skill before its first + dispatch, then named a model on every one of its own 62 Agent calls: haiku 38, sonnet 20, opus 4, all four Opus + calls on the careful case. In all-opus, none of the session's 68 Agent calls named a model (55 Explore, 13 + general-purpose), and all of them ran on Opus. `hook-warned`, which reads the session's trace, fired in 0 of 30 + routed runs. +- Helpers dispatching helpers, all on the careful case: routed, 5 such calls, 2 naming haiku and 3 naming none; + all-opus, 1, naming none, which ran on Opus. One routed case is clear-cut: a helper running on Opus dispatched a + general-purpose helper without a model, and that helper ran on Sonnet, dojo-router's `default_tier`. So the mod + tier loaded in the eval child (the operator's shell sets `CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1`) and acted at + least once. The other two unpinned calls came from a helper already on Sonnet and ran on Sonnet, so they don't + show which tier acted. +- Split halves: runs 1–3 give ρ = 0.643 (0.604 to 0.686), runs 4–6 give 0.591 (0.531 to 0.658). +- The routed arm took about one to two more turns on four of five cases (fewer on the careful case), and less wall + time on all five. +- The in-eval `quality-judge` (void, amendment 1) cost $0.44 of the $21.86. Agent cost excludes it. + +### Study F (dojo-flow) + +The run started 2026-10-23 at 22:18:45 UTC and took 1,998 s. All 20 runs finished with no errors: $15.75 against a +cap of $20.72, not partial. The files judge then cost $1.31. Every run counted. + +| task | arm | runs | mean agent cost | scorecard | agents dispatched | hidden pass | judge pass | mean wall s | +|---|---|---|---|---|---|---|---|---| +| hf-gradebook | routed | 5 | $0.728 | REFUSED 5 of 5 | 1 per run (contract, Opus) | 0 | 0 | 197 | +| hf-gradebook | all-opus | 5 | $0.718 | REFUSED 5 of 5 | 1 per run (contract, Opus) | 0 | 0 | 181 | +| hf-semver-ranges (careful) | routed | 5 | $0.829 | REFUSED 5 of 5 | 1 per run (contract, Opus) | 0 | 0 | 201 | +| hf-semver-ranges (careful) | all-opus | 5 | $0.843 | REFUSED 5 of 5 | 1 per run (contract, Opus) | 0 | 0 | 208 | + +- In all 20 runs, the contract gate (role `judge`, Opus in both arms) found the two tracks coupled and stopped the + build. No scout, builder, integrator, reviewer or checker ran. The only file any run added is + `.dojo/dag-contract.md`, which the gate wrote while refusing. No session edited code itself. +- Pooled mean agent cost: routed $0.7787, all-opus $0.7807. ρ = 0.997 (0.921 to 1.080). +- Hidden tests 0 of 10 in both arms, files judge 0 of 10 in both. Helpers off Opus: 0 of 10 in both arms. + +## Ruling applied mechanically + +**Study R.** + +1. Cost: ρ's interval, 0.570 to 0.665, lies entirely below 1. **Savings detected.** +2. Quality: Δq's interval, −0.138 to +0.138, is not entirely below 0. Its lower bound is −0.15 or above, and Δj = 0 + is −0.15 or above. **Non-inferior.** +3. Careful case: 6 hidden passes in each arm, so 0 fewer. **No harm flagged.** +4. Routing took effect: 94% of routed helpers ran off Opus, against 0%, which clears 50% and a 25-point gap. +5. Inconclusive conditions: none apply (60 of 60 runs counted, savings detected). + +**The rule outputs, for dojo-router: "saves, no measured loss".** Section 1 lets its README cite the measured cost +ratio with its conditions. Refutation checks (section 5): ρ's interval excludes 1; the hidden tests show no loss; +the careful case shows no harm; the all-opus arm ran 0% of helpers off Opus, so Claude Code's default does not +already route; and the routed arm did not save by doing less (more turns, same pass rate). None of them fires. + +Proposed README text for dojo-router, not applied here (the wording is Cruz's decision): *"In a pre-registered eval +(evals/routing-2026-10-23), an Opus session asked to delegate lookups spent 0.62 as much with dojo-router installed +as without (95% interval 0.57 to 0.67; 60 runs, five tasks, no shell, Claude Code 2.1.305, dojo-router 0.1.1), +with every hidden test passing in both arms. The saving came from the session following the router skill, not from +the hooks. Harder tasks, sessions with a shell and sessions that don't delegate were not measured."* + +**Study F.** + +1. Cost: ρ's interval, 0.921 to 1.080, includes 1. No detectable difference. +2. Quality: Δq's interval, −0.278 to +0.278, has a lower bound below −0.15. Not resolved. +3. Careful case: 0 and 0 passes. No harm flagged. +4. Routing took effect: 0% of routed helpers ran off Opus, against 0%. **Did not take effect.** +5. No savings were detected and routing did not take effect. + +**The rule outputs, for dojo-flow: "inconclusive: routing did not take effect".** Its README stays as it is. Every +build stopped at a gate that runs on Opus in both arms, before any routed role started. Section 5's first bullet +(no saving shown) is literally true here, but it refutes nothing about routing. + +## What outranks the ruling + +- **dojo-router's saving came from its skill, not its hooks.** The README describes a plugin that notices unpinned + dispatches and warns, blocks or fills in the model. Here none of that happened: the session read the `router` + skill in every routed run and pinned every one of its own dispatches, so the hooks had nothing to see at the top + level. The mod acted only once, on a helper's own dispatch (Results, above). The measured saving belongs to the + plugin as installed (skill plus hooks), and nearly all of it came through the skill. An arm with the skill + removed and the hooks kept, or the reverse, would split the two; that was not run. +- **The skill spent Opus where judgment was asked for.** On the careful task, the session named Opus for 4 of its + 6 helper dispatches, as the skill's role table says for a judge. So the saving there was small (0.88), and + quality held. That is the table doing its job on the task built to make routing lose. +- **The quality result is a ceiling, not a margin.** Every run in both arms passed every hidden test. Non-inferiority + clears the −0.15 margin only because both arms were perfect (lower bound −0.138), as section 4's power note + said would be the only way. These tasks were too easy for helpers on Haiku or Sonnet to fail. Harder lookups + might separate the arms. +- **dojo-flow's first live run found a gate problem, not a cost result.** This is the first time the build workflow + ran against real models (the README says none had been made). Its contract gate refused 24 of 24 builds, pilots + included, on two tasks an outside author wrote to the brief. Two causes: + - The independence test counts any track that calls another track's functions as coupled. Two tracks that share + a stated interface, which is the usual shape of a parallel build, can't pass it. That leaves either one track, + or tracks with nothing at all between them. + - The contract prompt gives the gate track names and owned files only, not the track prompts, yet tells it to + "check that each track prompt matches the code". 12 of the 20 main-run traces say so. + + The brief asked for exactly this shape of task: a shared interface stated in both prompts. So the task set and + the gate's rule collided. A task set built to the gate's rule might pass it; this one tells you the rule is + stricter than the README's "Contract before fan-out" suggests. +- **A refused build leaves its contract file behind.** All 20 refusals wrote `.dojo/dag-contract.md`, and the + workflow refuses to overwrite that file. So a re-run in the same repository fails until the file is removed or a + new `contractPath` is passed. + +## Gaps and handicaps + +- **No shell.** On this Windows machine the eval refuses to grant a shell tool (amendment 1). No agent, helper or + workflow agent could run a test, a gate, git or any command. With a shell, sessions do more work per task, and + the helpers' share of the cost, and so the saving, may differ. +- **Sessions that delegate.** The R prompts asked for subagents. A session that never dispatches gets nothing from + routing, and this eval says nothing about it. +- **One platform, one version.** Windows 11; Claude Code 2.1.305; dojo-router 0.1.1; dojo-flow 0.1.0; + `claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`. Not replicated on a second sample; the split halves + above are the only check on noise. +- **Model-written tasks, evaluator-written brief.** The task author is a model (Sonnet 5.5). The brief was the + evaluator's, and it shaped the F tasks into a form the dojo-flow gate refuses. +- **The judge.** It ran outside the eval because the eval's files view is empty here (amendment 1). It is Opus + judging Opus-written code in one arm. It disagreed with the hidden tests once in 80 runs. +- **Cost figures** are Claude Code's list-price equivalents for runs on a claude.ai subscription: plan usage, not + invoices. Agent cost leaves out the in-eval judge ($0.44 in R, $0.16 in F). +- **Most of what dojo-router's README describes was not exercised.** The warn and block paths never fired at the + top level, and the mod acted once, on a nested dispatch. Their claims are neither supported nor refuted here, + beyond that one routing. +- **Kept run folders** (traces, transcripts, work directories) stay in the running machine's temp directory and are + not in this repository. The per-run rows are. +- **Spend:** $52.15 of the $100 cap, of which $8.48 was pilots, excluded from analysis (`spend-ledger.md`). The + remaining $47.86 was not spent; no follow-up was run. + +## Digest check after write-up + +``` +dojo-settle: digest matches the digest you passed with --expect (sha256 d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a). This shows the text above the marker is unchanged since that digest was recorded; it does not show the text is true. +``` + +The digest was first recorded in commit `7404987` (pushed 2026-10-23 21:15:15 UTC), before any eval run, and +again in the session's first status message to Cruz. Amendment 1 and the analysis code are commit `76f2779`, +pushed before the first main run. These ids are on branch `eval/routing-2026-10-23`; if the branch is merged by +rebase they will change, and the originals stay fetchable from the pull request. diff --git a/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md.sha256 b/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md.sha256 new file mode 100644 index 0000000..7301044 --- /dev/null +++ b/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md.sha256 @@ -0,0 +1,8 @@ +dojo-settle prereg digest +sha256: d66e5b96bc78d9987d5fc13f60b3b85918b60b1bd0bd04676631bf505645187a +bytes: 19079 +lines: 267 +file: PREREG-2026-10-23-routing.md +recorded_at_utc_local_clock: 2026-10-23T21:14:21Z +region: bytes before the first marker line; BOM dropped, CRLF as LF +note: this sidecar can be rewritten along with the document; record the sha256 elsewhere diff --git a/evals/routing-2026-10-23/analyze.py b/evals/routing-2026-10-23/analyze.py new file mode 100644 index 0000000..1eb9b86 --- /dev/null +++ b/evals/routing-2026-10-23/analyze.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +"""Apply section 4 of PREREG-2026-10-23-routing.md to a finished main run. Standard library only. + + analyze.py R [--careful hr-limiter-starve] + analyze.py F [--careful hf-semver-ranges] + +Reads the eval's own JSON (per-run cost, judge cost, error, trace path), runs the hidden tests on each kept work +directory (grade_hidden.py), reads the files judge's verdicts (judge_files.py, one JSON line per run), and counts +the model of every subagent and workflow agent from the kept transcripts. Prints the per-cell table, the pooled +figures and what the rule outputs, and writes the same as JSON next to the eval JSON (-analysis.json). + +Arms: in R the eval's `with` arm is routed and `without` is all-opus. In F every case runs in one arm and the arm +is the case name's suffix (`-routed` / `-allopus`); the task is the name without it. +""" +import glob +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import grade_hidden # noqa: E402 +import stats # noqa: E402 + +NON_OPUS = ("haiku", "sonnet") + + +def agent_models(root): + """Model of each subagent / workflow agent transcript under the kept config dir: list of model ids.""" + out = [] + for path in sorted(glob.glob(os.path.join(root, "config", "projects", "**", "*.jsonl"), recursive=True)): + if os.sep + "subagents" + os.sep not in path and "/subagents/" not in path: + continue + seen = None + with open(path, encoding="utf-8") as f: + for line in f: + try: + e = json.loads(line) + except ValueError: + continue + m = (e.get("message") or {}).get("model") + if e.get("type") == "assistant" and m and m != "": + seen = m + break + if seen: + out.append(seen) + return out + + +def load_runs(study, eval_json, judge_jsonl): + d = json.load(open(eval_json, encoding="utf-8")) + verdicts = {} + if os.path.exists(judge_jsonl): + for line in open(judge_jsonl, encoding="utf-8"): + j = json.loads(line) + verdicts[(j["case"], j["arm"], j["run"])] = j + rows = [] + for c in d["cases"]: + for eval_arm, runs in c["arms"].items(): + for i, r in enumerate(runs): + name = c["name"] + if study == "R": + task, arm = name, ("routed" if eval_arm == "with" else "allopus") + else: + task, arm = name.rsplit("-", 1) + root = os.path.dirname(os.path.dirname(r["tracePath"])) if r.get("tracePath") else "" + work = os.path.join(root, "home", "cwd") + hidden, detail = grade_hidden.run_hidden(os.path.join(HERE, "cases", task), work) + j = verdicts.get((name, eval_arm, i + 1), {}) + no_output = bool(r.get("error")) and not r.get("turns") + rows.append({ + "case": name, "task": task, "arm": arm, "run": i + 1, + "cost": r["costUsd"], "judge_cost_in_eval": r.get("judgeCostUsd") or 0.0, + "agent_cost": r["costUsd"] - (r.get("judgeCostUsd") or 0.0), + "turns": r.get("turns"), "duration_s": r.get("durationSeconds"), "error": r.get("error"), + "hidden": hidden, "hidden_detail": detail, + "judge": j.get("verdict", "MISSING"), "judge_votes": j.get("votes"), "judge_cost": j.get("cost_usd", 0.0), + "agents": agent_models(root) if root else [], + "counts": (not no_output) and hidden in ("PASS", "FAIL"), + }) + return d, rows + + +def analyze(study, eval_json, judge_jsonl, careful): + d, rows = load_runs(study, eval_json, judge_jsonl) + counted = [r for r in rows if r["counts"]] + tasks = sorted({r["task"] for r in rows}) + cells = {} + for t in tasks: + for a in ("routed", "allopus"): + rs = [r for r in counted if r["task"] == t and r["arm"] == a] + allrs = [r for r in rows if r["task"] == t and r["arm"] == a] + cells[(t, a)] = { + "planned_or_seen": len(allrs), "counted": len(rs), + "mean_agent_cost": (sum(r["agent_cost"] for r in rs) / len(rs)) if rs else None, + "hidden_pass": sum(r["hidden"] == "PASS" for r in rs), + "judge_pass": sum(r["judge"] == "PASS" for r in rs), + "mean_turns": (sum(r["turns"] or 0 for r in rs) / len(rs)) if rs else None, + "agents": sum(len(r["agents"]) for r in rs), + "agents_non_opus": sum(sum(1 for m in r["agents"] if any(k in m for k in NON_OPUS)) for r in rs), + "models": sorted({m for r in rs for m in r["agents"]}), + } + # Cost: every task, careful included. + cost = {a: {t: [r["agent_cost"] for r in counted if r["task"] == t and r["arm"] == a] for t in tasks} for a in ("routed", "allopus")} + rho = stats.bootstrap_ratio(cost["routed"], cost["allopus"]) if all(any(v) for v in cost["routed"].values()) and all(any(v) for v in cost["allopus"].values()) else None + # Quality: standard tasks in R; both tasks in F. + qtasks = [t for t in tasks if (t != careful or study == "F")] + def rate(arm, key): + rs = [r for r in counted if r["task"] in qtasks and r["arm"] == arm] + return sum(r[key] == "PASS" for r in rs), len(rs) + xq1, nq1 = rate("routed", "hidden"); xq0, nq0 = rate("allopus", "hidden") + xj1, nj1 = rate("routed", "judge"); xj0, nj0 = rate("allopus", "judge") + dq = stats.newcombe_diff(xq1, nq1, xq0, nq0) if nq1 and nq0 else None + dj = (xj1 / nj1 - xj0 / nj0) if nj1 and nj0 else None + # Careful case. + cr, ca = cells.get((careful, "routed")), cells.get((careful, "allopus")) + harm = (ca["hidden_pass"] - cr["hidden_pass"] >= 2) if cr and ca else None + careful_win = (cr["hidden_pass"] - ca["hidden_pass"] >= 2) if cr and ca else None + # Manipulation. + ag_r = sum(c["agents"] for (t, a), c in cells.items() if a == "routed") + no_r = sum(c["agents_non_opus"] for (t, a), c in cells.items() if a == "routed") + ag_o = sum(c["agents"] for (t, a), c in cells.items() if a == "allopus") + no_o = sum(c["agents_non_opus"] for (t, a), c in cells.items() if a == "allopus") + if ag_r and ag_o: + share_r, share_o = no_r / ag_r, no_o / ag_o + took_effect = share_r >= 0.5 and share_r - share_o >= 0.25 + manip = "took effect" if took_effect else "did not take effect" + else: + share_r = share_o = None + manip = "not measured" + took_effect = (study == "F") # the F script sets each agent's model from the map itself + # Rule. + if rho is None: + cost_out = "not computable" + elif rho[2] < 1: + cost_out = "savings detected" + elif rho[1] > 1: + cost_out = "increase detected" + else: + cost_out = "no detectable difference" + if dq is None or dj is None: + q_out = "not computable" + elif dq[2] < 0 and dj <= 0: + q_out = "loss detected" + elif dq[1] >= -0.15 and dj >= -0.15: + q_out = "non-inferior" + else: + q_out = "not resolved" + q_gain = bool(dq and dj is not None and dq[1] > 0 and dj >= 0) + if cost_out == "savings detected": + verdict = {"non-inferior": "saves, no measured loss", "not resolved": "saves, quality not resolved", + "loss detected": "saves at a quality cost"}.get(q_out, "saves, quality not computable") + elif cost_out == "increase detected": + verdict = "costs more" + else: + verdict = "no detectable difference in cost" + planned = {a: sum(1 for r in rows if r["arm"] == a) for a in ("routed", "allopus")} + n_counted = {a: sum(1 for r in counted if r["arm"] == a) for a in ("routed", "allopus")} + reasons = [] + if any(n_counted[a] * 2 < planned[a] for a in planned): + reasons.append("fewer than half of the planned runs counted in an arm") + if cost_out != "savings detected" and (manip == "did not take effect" or (manip == "not measured" and study == "R")): + reasons.append("no savings detected and routing did not take effect or was not measured") + if reasons: + verdict = "inconclusive: " + "; ".join(reasons) + if harm: + verdict += " (careful-case harm flagged)" + return { + "study": study, "eval_json": os.path.basename(eval_json), "claude_version": d.get("claudeVersion"), + "started_at": d.get("startedAt"), "eval_cost_usd": d.get("costUsd"), "partial": d.get("partial"), + "cells": {"%s|%s" % k: v for k, v in cells.items()}, + "runs_planned_or_seen": planned, "runs_counted": n_counted, + "missing": [{k: r[k] for k in ("case", "arm", "run", "error", "hidden", "hidden_detail")} for r in rows if not r["counts"]], + "pooled_mean_agent_cost": {a: (stats.pooled_mean({t: v for t, v in cost[a].items() if v}) if any(cost[a].values()) else None) for a in cost}, + "cost_ratio": rho, "cost_outcome": cost_out, + "quality_tasks": qtasks, "hidden": {"routed": [xq1, nq1], "allopus": [xq0, nq0], "dq_newcombe": dq}, + "judge": {"routed": [xj1, nj1], "allopus": [xj0, nj0], "dj": dj}, + "quality_outcome": q_out, "quality_gain_unexpected": q_gain, + "careful": careful, "careful_harm": harm, "careful_routed_wins_by_2": careful_win, + "agents_non_opus": {"routed": [no_r, ag_r], "allopus": [no_o, ag_o], "share_routed": share_r, "share_allopus": share_o}, + "manipulation": manip, "verdict": verdict, + "runs": rows, + } + + +def main(argv): + if len(argv) < 4 or argv[1] not in ("R", "F"): + print(__doc__.strip(), file=sys.stderr) + return 2 + study, eval_json, judge_jsonl = argv[1], argv[2], argv[3] + careful = argv[argv.index("--careful") + 1] if "--careful" in argv else ("hr-limiter-starve" if study == "R" else "hf-semver-ranges") + res = analyze(study, eval_json, judge_jsonl, careful) + out = os.path.splitext(eval_json)[0] + "-analysis.json" + with open(out, "w", encoding="utf-8", newline="\n") as f: + json.dump(res, f, indent=1) + print("study %s claude %s eval cost $%.4f partial %s" % (study, res["claude_version"], res["eval_cost_usd"], res["partial"])) + print("task | arm | counted/seen | mean agent cost | hidden pass | judge pass | mean turns | non-opus agents | models") + for k, c in sorted(res["cells"].items()): + t, a = k.split("|") + mc = "—" if c["mean_agent_cost"] is None else "$%.3f" % c["mean_agent_cost"] + mt = "—" if c["mean_turns"] is None else "%.1f" % c["mean_turns"] + print("%s | %s | %d/%d | %s | %d | %d | %s | %d/%d | %s" % (t, a, c["counted"], c["planned_or_seen"], mc, c["hidden_pass"], c["judge_pass"], mt, c["agents_non_opus"], c["agents"], ",".join(c["models"]))) + print("pooled mean agent cost:", res["pooled_mean_agent_cost"]) + print("cost ratio routed/all-opus (point, 95% lo, hi):", res["cost_ratio"], "->", res["cost_outcome"]) + print("hidden:", res["hidden"], "| judge:", res["judge"], "->", res["quality_outcome"], "| unexpected gain:", res["quality_gain_unexpected"]) + print("careful %s: harm %s, routed wins by 2+: %s" % (res["careful"], res["careful_harm"], res["careful_routed_wins_by_2"])) + print("non-opus agents:", res["agents_non_opus"], "->", res["manipulation"]) + print("missing runs:", len(res["missing"])) + print("VERDICT:", res["verdict"]) + print("wrote", out) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/routing-2026-10-23/build_cases.py b/evals/routing-2026-10-23/build_cases.py new file mode 100644 index 0000000..f809daa --- /dev/null +++ b/evals/routing-2026-10-23/build_cases.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""Turn the held-out author's tasks into `claude plugin eval` cases (PREREG-2026-10-23-routing.md). + + build_cases.py + +For each task folder in (case.json, scaffold.sh, hidden/, reference/): +- the author's files are copied unchanged to evals/routing-2026-10-23/cases// (hidden tests and the + reference stay there, outside every plugin directory, so no agent sandbox ever holds them); +- `hr-*` tasks become plugins/dojo-router/evals//{case.yaml, scaffold.sh}; +- `hf-*` tasks become plugins/dojo-flow/evals/-routed/ and -allopus/, identical except the models map. + +The prompts are the author's (hr) or the evaluator's fixed wrapper around the author's args (hf). The graders are +the evaluator's, fixed by the pre-registration. Never overwrites an existing case folder. +""" +import json +import os +import shutil +import sys + +HERE = "evals/routing-2026-10-23" + +HEADER = ( + "# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md).\n" + "# The task was written by an independent model author from held-out-brief.md, without access to this\n" + "# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders\n" + "# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run\n" + "# without a cap and a yes.\n" +) + +ROUTED = {"scout": "haiku", "builder": "sonnet", "integrator": "sonnet", "reviewer": "sonnet", + "checker": "haiku", "judge": "opus"} +ALL_OPUS = {k: "opus" for k in ROUTED} + +HF_WRAPPER = ( + "Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow " + "tool with name \"dojo-flow:build\"; if that name is not known, call it with scriptPath set to the plugin's " + "workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not " + "change the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to " + "finish, then reply with its scorecard exactly as returned.\n\nargs (add repo yourself):\n" +) + + +def judge(task): + return { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": ( + "You are grading the final state of a small Python project after a coding task. Read the project's " + "files. A correct finish: " + task["expected_outcome"].strip() + " Decide whether the code now does " + "what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it " + "does not, or only partly does." + ), + } + + +def hr_case(task): + return { + "schema_version": "1.0", + "name": task["name"], + "description": task["description"], + "tags": ["held-out", "routing", task["kind"]], + "runs": 6, + "context": {"scaffold_script": "scaffold.sh", "add_dirs": []}, + "execution": { + "prompt": task["prompt"], + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": ["Read", "Edit", "Write", "Bash", "Glob", "Grep", "Agent", "Skill"], + }, + "graders": [ + judge(task), + {"type": "tool_used", "name": "dispatched", "tool": "Agent", "min": 1}, + {"type": "regex", "name": "hook-warned", "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", "arm": "with-only"}, + ], + "expected_outcome": task["expected_outcome"], + } + + +def hf_case(task, arm): + args = {"goal": task["goal"], "tracks": task["tracks"], "gates": task["gates"], + "models": ROUTED if arm == "routed" else ALL_OPUS} + return { + "schema_version": "1.0", + "name": task["name"] + "-" + arm, + "description": task["description"] + " (" + ("routed: workflow defaults" if arm == "routed" else "every role on opus") + ")", + "tags": ["held-out", "routing", task["kind"], arm], + "runs": 5, + "context": {"scaffold_script": "scaffold.sh", "add_dirs": []}, + "execution": { + "prompt": HF_WRAPPER + json.dumps(args, indent=2), + "max_turns": 12, + "timeout_seconds": 1800, + "model": "opus", + "allowed_tools": ["Read", "Edit", "Write", "Glob", "Grep", "Bash", "Skill", "Workflow"], + }, + "graders": [ + judge(task), + {"type": "tool_used", "name": "workflow-ran", "tool": "Workflow", "min": 1}, + {"type": "regex", "name": "scorecard-relayed", "target": "last_message", + "pattern": "verdict: (VERIFIED|UNVERIFIED|REFUSED|PLAN ONLY)"}, + ], + "expected_outcome": task["expected_outcome"], + } + + +def write_case(folder, case, scaffold): + os.makedirs(folder) # refuses an existing folder + with open(os.path.join(folder, "case.yaml"), "w", encoding="utf-8", newline="\n") as f: + f.write(HEADER + json.dumps(case, indent=2, ensure_ascii=False) + "\n") + shutil.copyfile(scaffold, os.path.join(folder, "scaffold.sh")) + + +def main(argv): + if len(argv) != 3: + print(__doc__.strip(), file=sys.stderr) + return 2 + src, repo = argv[1], argv[2] + for name in sorted(os.listdir(src)): + tdir = os.path.join(src, name) + if not os.path.isfile(os.path.join(tdir, "case.json")): + continue + with open(os.path.join(tdir, "case.json"), encoding="utf-8") as f: + task = json.load(f) + assert task["name"] == name, "case.json name %r != folder %r" % (task["name"], name) + shutil.copytree(tdir, os.path.join(repo, HERE, "cases", name)) + scaffold = os.path.join(tdir, "scaffold.sh") + if name.startswith("hr-"): + write_case(os.path.join(repo, "plugins", "dojo-router", "evals", name), hr_case(task), scaffold) + elif name.startswith("hf-"): + for arm in ("routed", "allopus"): + write_case(os.path.join(repo, "plugins", "dojo-flow", "evals", name + "-" + arm), hf_case(task, arm), scaffold) + else: + print("skipped %s: not hr- or hf-" % name, file=sys.stderr) + continue + print("built %s (%s)" % (name, task["kind"])) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/case.json b/evals/routing-2026-10-23/cases/hf-gradebook/case.json new file mode 100644 index 0000000..3d640af --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/case.json @@ -0,0 +1,30 @@ +{ + "name": "hf-gradebook", + "family": "hf", + "kind": "standard", + "description": "Two parallel tracks build a gradebook: statistics helpers, and CSV report plus command line using them.", + "prompt": "", + "goal": "Build a small Python package `gradebook` (standard library only) that turns a class's scores, given as CSV text with a name column and a score column, into a ranked plain-text table with letter grades and a class summary line, plus a command line front end. The directory `gradebook/` and its `__init__.py` already exist, and `tests/test_gradebook.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `stats` writes `gradebook/stats.py`; `report` writes `gradebook/report.py` and `gradebook/cli.py`. The shared interface is `stats.letter_grade(score)` and `stats.summarize(values)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.", + "tracks": [ + { + "name": "stats", + "prompt": "Write the file `gradebook/stats.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `gradebook/report.py` and `gradebook/cli.py` at the same time; they will call `stats.letter_grade(score)` and `stats.summarize(values)` exactly as described below, so keep those names, arguments and return shapes.\n\nProvide exactly these functions:\n- `mean(values)` -> float: the arithmetic mean of a non-empty sequence of numbers. Raise `ValueError` if it is empty.\n- `median(values)` -> float: the middle value of the numbers in sorted order; for an even count, the mean of the two middle values. Raise `ValueError` if empty. It must not modify the sequence passed in.\n- `letter_grade(score)` -> str: `\"A\"` when score >= 90, `\"B\"` when >= 80, `\"C\"` when >= 70, `\"D\"` when >= 60, otherwise `\"F\"`. Scores are floats, so 89.99 is `\"B\"`, 90.0 is `\"A\"` and 104.5 is `\"A\"`.\n- `summarize(values)` -> dict with exactly the keys `count` (int), `mean`, `median`, `min`, `max` for a sequence of numbers. For an empty sequence do not raise: return `{\"count\": 0, \"mean\": None, \"median\": None, \"min\": None, \"max\": None}`.", + "files": [ + "gradebook/stats.py" + ] + }, + { + "name": "report", + "prompt": "Write the files `gradebook/report.py` and `gradebook/cli.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `gradebook/stats.py` at the same time; do not write that file. Import it with `from gradebook import stats` and use only these two functions from it, which will behave as follows:\n- `stats.letter_grade(score)` -> str: `\"A\"` for score >= 90, `\"B\"` for >= 80, `\"C\"` for >= 70, `\"D\"` for >= 60, otherwise `\"F\"`.\n- `stats.summarize(values)` -> dict with the keys `count` (int), `mean`, `median`, `min`, `max` for a list of numbers (for an empty list the count is 0 and the others are `None`; you will not need that case).\n\n`gradebook/report.py` provides:\n- `parse_rows(text)` -> list of `(name, score)` tuples with `name` a str and `score` a float. `text` is CSV with two columns, name and score; read it with the `csv` module. Ignore lines that are empty or contain only whitespace. The first remaining line is a header, and is skipped, when it has at least two fields and its second field (stripped) is not a number that `float()` accepts, for example `name,score`; any other line is a data row. Every data row must have exactly two fields, a non-empty name once stripped, and a stripped score that `float()` accepts; otherwise raise `ValueError`. Names are returned stripped. Rows keep their input order. A header line is only recognised as the first remaining line, so a later line with a non-numeric score is an error.\n- `render(rows)` -> str. If `rows` is empty return exactly `No scores`. Otherwise return these lines joined with `\"\\n\"` and no trailing newline: first `Name | Score | Grade`; then one line per row, ordered by score from highest to lowest and, for equal scores, by name in plain string order (case-sensitive, so `Zed` comes before `bob`), each formatted `{name} | {score:.1f} | {grade}` where grade is `stats.letter_grade(score)` applied to the unrounded score (so a score of 89.96 is printed as `90.0` but still gets `B`); then a last line `Class: {count} students, mean {mean:.1f}, median {median:.1f}` with the numbers taken from `stats.summarize` of all the scores, using `1 student` instead of `1 students` when the count is 1.\n\n`gradebook/cli.py` provides:\n- `main(argv=None, stdin=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdin`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. With more than one item in `argv`, write `usage: python -m gradebook.cli [file.csv]` and a newline to stderr and return 2. With one item, it is the path of a CSV file: read it as UTF-8 (open it with `newline=\"\"`). With none, read all of `stdin`. Then write `report.render(report.parse_rows(text))` followed by a newline to stdout and return 0. If reading the file raises `OSError` or parsing raises `ValueError`, write `error: ` and a newline to stderr, write nothing to stdout, and return 2.\n- End the file with `if __name__ == \"__main__\": raise SystemExit(main())`.", + "files": [ + "gradebook/report.py", + "gradebook/cli.py" + ] + } + ], + "gates": [ + "python3 -m unittest discover -s tests -q", + "python3 -c \"import gradebook.stats, gradebook.report, gradebook.cli\"" + ], + "expected_outcome": "gradebook/stats.py, gradebook/report.py and gradebook/cli.py exist and agree on the shared stats interface, and the gates pass: the tests pass and all three modules import." +} diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hf-gradebook/hidden/test_hidden.py new file mode 100644 index 0000000..e77427c --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/hidden/test_hidden.py @@ -0,0 +1,215 @@ +import io +import os +import tempfile +import unittest + +from gradebook import cli, report, stats + + +class MeanAndMedian(unittest.TestCase): + def test_mean_values(self): + self.assertEqual(stats.mean([1, 2, 3, 4]), 2.5) + self.assertEqual(stats.mean([7]), 7) + self.assertAlmostEqual(stats.mean([0.1, 0.2, 0.3]), 0.2) + + def test_mean_returns_float(self): + self.assertIsInstance(stats.mean([2, 4]), float) + + def test_mean_accepts_any_sequence(self): + self.assertEqual(stats.mean((1, 2, 3)), 2) + + def test_mean_of_nothing_raises(self): + with self.assertRaises(ValueError): + stats.mean([]) + + def test_median_odd(self): + self.assertEqual(stats.median([9, 1, 5]), 5) + + def test_median_even_averages_the_middle_two(self): + self.assertEqual(stats.median([10, 2, 8, 4]), 6) + self.assertEqual(stats.median([1, 2]), 1.5) + + def test_median_single(self): + self.assertEqual(stats.median([42]), 42) + + def test_median_does_not_change_the_input(self): + data = [3, 1, 2] + stats.median(data) + self.assertEqual(data, [3, 1, 2]) + + def test_median_of_nothing_raises(self): + with self.assertRaises(ValueError): + stats.median([]) + + +class LetterGrades(unittest.TestCase): + def test_boundaries(self): + cases = [ + (100, "A"), (90.0, "A"), (89.99, "B"), (80, "B"), (79.99, "C"), + (70, "C"), (69.5, "D"), (60, "D"), (59.99, "F"), (0, "F"), + ] + for score, grade in cases: + self.assertEqual(stats.letter_grade(score), grade, score) + + def test_above_one_hundred_is_still_an_a(self): + self.assertEqual(stats.letter_grade(104.5), "A") + + +class Summarize(unittest.TestCase): + def test_keys_and_values(self): + summary = stats.summarize([90, 70, 80, 100]) + self.assertEqual( + summary, + {"count": 4, "mean": 85, "median": 85, "min": 70, "max": 100}, + ) + + def test_exact_keys(self): + self.assertEqual( + sorted(stats.summarize([1])), ["count", "max", "mean", "median", "min"] + ) + + def test_empty(self): + self.assertEqual( + stats.summarize([]), + {"count": 0, "mean": None, "median": None, "min": None, "max": None}, + ) + + +class ParseRows(unittest.TestCase): + def test_header_is_skipped(self): + self.assertEqual(report.parse_rows("name,score\nAda,95\n"), [("Ada", 95.0)]) + + def test_no_header(self): + self.assertEqual( + report.parse_rows("Ada,95\nBob,82.5"), [("Ada", 95.0), ("Bob", 82.5)] + ) + + def test_blank_lines_and_whitespace(self): + text = "\n name , score \n\n Ada , 95 \n \nBob,70\n\n" + self.assertEqual(report.parse_rows(text), [("Ada", 95.0), ("Bob", 70.0)]) + + def test_scores_are_floats(self): + ((_, score),) = report.parse_rows("Ada,95") + self.assertIsInstance(score, float) + + def test_header_only_and_empty_text(self): + self.assertEqual(report.parse_rows("name,score\n"), []) + self.assertEqual(report.parse_rows(""), []) + self.assertEqual(report.parse_rows("\n\n"), []) + + def test_quoted_names(self): + self.assertEqual( + report.parse_rows('"Lovelace, Ada",99\n'), [("Lovelace, Ada", 99.0)] + ) + + def test_bad_score_after_the_first_line_raises(self): + with self.assertRaises(ValueError): + report.parse_rows("Ada,95\nBob,absent\n") + + def test_wrong_field_count_raises(self): + with self.assertRaises(ValueError): + report.parse_rows("Ada,95,extra\n") + with self.assertRaises(ValueError): + report.parse_rows("Ada,95\nBob\n") + + def test_missing_name_raises(self): + with self.assertRaises(ValueError): + report.parse_rows("Ada,95\n,80\n") + + def test_only_the_first_line_may_be_a_header(self): + with self.assertRaises(ValueError): + report.parse_rows("Ada,95\nname,score\n") + + +class Render(unittest.TestCase): + def test_nothing(self): + self.assertEqual(report.render([]), "No scores") + + def test_full_table(self): + text = report.render([("Bob", 82.5), ("Ada", 95.0), ("Cy", 55.0)]) + self.assertEqual( + text, + "\n".join( + [ + "Name | Score | Grade", + "Ada | 95.0 | A", + "Bob | 82.5 | B", + "Cy | 55.0 | F", + "Class: 3 students, mean 77.5, median 82.5", + ] + ), + ) + + def test_no_trailing_newline(self): + self.assertFalse(report.render([("Ada", 95.0)]).endswith("\n")) + + def test_singular(self): + text = report.render([("Ada", 95.0)]) + self.assertEqual(text.split("\n")[-1], "Class: 1 student, mean 95.0, median 95.0") + + def test_ties_sort_by_name_in_plain_string_order(self): + text = report.render([("bob", 80.0), ("Ada", 80.0), ("Zed", 80.0), ("Cy", 95.0)]) + names = [line.split(" | ")[0] for line in text.split("\n")[1:5]] + self.assertEqual(names, ["Cy", "Ada", "Zed", "bob"]) + + def test_grade_uses_the_unrounded_score(self): + text = report.render([("Eve", 89.96)]) + self.assertEqual(text.split("\n")[1], "Eve | 90.0 | B") + + def test_scores_sorted_high_to_low(self): + text = report.render([("A", 10.0), ("B", 99.0), ("C", 55.5)]) + scores = [line.split(" | ")[1] for line in text.split("\n")[1:4]] + self.assertEqual(scores, ["99.0", "55.5", "10.0"]) + + +class CommandLine(unittest.TestCase): + CSV = "name,score\nAda,95\nBob,82.5\n" + + def run_main(self, argv, stdin_text=""): + out, err = io.StringIO(), io.StringIO() + code = cli.main(argv, io.StringIO(stdin_text), out, err) + return code, out.getvalue(), err.getvalue() + + def test_stdin(self): + code, out, err = self.run_main([], self.CSV) + self.assertEqual(code, 0) + self.assertEqual(err, "") + self.assertTrue(out.endswith("\n")) + self.assertEqual(out, report.render(report.parse_rows(self.CSV)) + "\n") + + def test_file_argument(self): + with tempfile.TemporaryDirectory() as tmp: + path = os.path.join(tmp, "scores.csv") + with open(path, "w", encoding="utf-8") as handle: + handle.write(self.CSV) + code, out, err = self.run_main([path], "ignored") + self.assertEqual(code, 0) + self.assertIn("Bob | 82.5 | B", out) + self.assertEqual(err, "") + + def test_empty_input(self): + code, out, _ = self.run_main([], "") + self.assertEqual((code, out), (0, "No scores\n")) + + def test_bad_data(self): + code, out, err = self.run_main([], "Ada,95\nBob,absent\n") + self.assertEqual(code, 2) + self.assertEqual(out, "") + self.assertTrue(err.startswith("error: ")) + self.assertTrue(err.endswith("\n")) + + def test_missing_file(self): + code, out, err = self.run_main(["/no/such/dir/scores.csv"]) + self.assertEqual(code, 2) + self.assertEqual(out, "") + self.assertTrue(err.startswith("error: ")) + + def test_too_many_arguments(self): + code, out, err = self.run_main(["a.csv", "b.csv"]) + self.assertEqual(code, 2) + self.assertEqual(out, "") + self.assertTrue(err.startswith("usage:")) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/cli.py b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/cli.py new file mode 100644 index 0000000..c7d0fa9 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/cli.py @@ -0,0 +1,33 @@ +"""Command line front end: python -m gradebook.cli [file.csv]""" + +import sys + +from gradebook import report + +USAGE = "usage: python -m gradebook.cli [file.csv]" + + +def main(argv=None, stdin=None, stdout=None, stderr=None): + argv = sys.argv[1:] if argv is None else argv + stdin = sys.stdin if stdin is None else stdin + stdout = sys.stdout if stdout is None else stdout + stderr = sys.stderr if stderr is None else stderr + if len(argv) > 1: + stderr.write(USAGE + "\n") + return 2 + try: + if argv: + with open(argv[0], encoding="utf-8", newline="") as handle: + text = handle.read() + else: + text = stdin.read() + output = report.render(report.parse_rows(text)) + except (OSError, ValueError) as exc: + stderr.write(f"error: {exc}\n") + return 2 + stdout.write(output + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/report.py b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/report.py new file mode 100644 index 0000000..3a2fc42 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/report.py @@ -0,0 +1,53 @@ +"""CSV in, text table out.""" + +import csv +import io + +from gradebook import stats + + +def _is_number(text): + try: + float(text) + except ValueError: + return False + return True + + +def parse_rows(text): + rows = [] + seen_first = False + for fields in csv.reader(io.StringIO(text)): + if not fields or (len(fields) == 1 and not fields[0].strip()): + continue + if not seen_first: + seen_first = True + if len(fields) >= 2 and not _is_number(fields[1].strip()): + continue # header row + if len(fields) != 2: + raise ValueError(f"expected 2 fields, got {len(fields)}: {fields!r}") + name = fields[0].strip() + if not name: + raise ValueError("missing name") + try: + score = float(fields[1].strip()) + except ValueError: + raise ValueError(f"bad score for {name!r}: {fields[1]!r}") from None + rows.append((name, score)) + return rows + + +def render(rows): + if not rows: + return "No scores" + ordered = sorted(rows, key=lambda row: (-row[1], row[0])) + lines = ["Name | Score | Grade"] + for name, score in ordered: + lines.append(f"{name} | {score:.1f} | {stats.letter_grade(score)}") + summary = stats.summarize([score for _, score in rows]) + noun = "student" if summary["count"] == 1 else "students" + lines.append( + f"Class: {summary['count']} {noun}, " + f"mean {summary['mean']:.1f}, median {summary['median']:.1f}" + ) + return "\n".join(lines) diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/stats.py b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/stats.py new file mode 100644 index 0000000..17f42db --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/reference/gradebook/stats.py @@ -0,0 +1,44 @@ +"""Numbers for the gradebook.""" + + +def mean(values): + values = list(values) + if not values: + raise ValueError("mean of empty data") + return sum(values) / len(values) + + +def median(values): + ordered = sorted(values) + count = len(ordered) + if count == 0: + raise ValueError("median of empty data") + mid = count // 2 + if count % 2: + return float(ordered[mid]) + return (ordered[mid - 1] + ordered[mid]) / 2 + + +def letter_grade(score): + if score >= 90: + return "A" + if score >= 80: + return "B" + if score >= 70: + return "C" + if score >= 60: + return "D" + return "F" + + +def summarize(values): + values = list(values) + if not values: + return {"count": 0, "mean": None, "median": None, "min": None, "max": None} + return { + "count": len(values), + "mean": mean(values), + "median": median(values), + "min": min(values), + "max": max(values), + } diff --git a/evals/routing-2026-10-23/cases/hf-gradebook/scaffold.sh b/evals/routing-2026-10-23/cases/hf-gradebook/scaffold.sh new file mode 100644 index 0000000..bcf22d9 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-gradebook/scaffold.sh @@ -0,0 +1,96 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# gradebook + +Turns a class's scores (CSV text) into a ranked table with letter grades. + +Layout: + +- `gradebook/stats.py` numbers: mean, median, letter grades, summary +- `gradebook/report.py` CSV parsing and the text table +- `gradebook/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'gradebook' +cat > 'gradebook/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""gradebook: scores in, ranked report out.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for gradebook.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_gradebook.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from gradebook import cli, report, stats + + +class Stats(unittest.TestCase): + def test_mean(self): + self.assertEqual(stats.mean([1, 2, 3, 4]), 2.5) + + def test_median_odd_and_even(self): + self.assertEqual(stats.median([3, 1, 2]), 2) + self.assertEqual(stats.median([4, 1, 3, 2]), 2.5) + + def test_letter_grades(self): + self.assertEqual(stats.letter_grade(95), "A") + self.assertEqual(stats.letter_grade(85), "B") + self.assertEqual(stats.letter_grade(75), "C") + self.assertEqual(stats.letter_grade(65), "D") + self.assertEqual(stats.letter_grade(10), "F") + + def test_summarize(self): + summary = stats.summarize([50, 100]) + self.assertEqual(summary["count"], 2) + self.assertEqual(summary["mean"], 75) + self.assertEqual(summary["min"], 50) + self.assertEqual(summary["max"], 100) + + +class Report(unittest.TestCase): + def test_parse_rows(self): + rows = report.parse_rows("name,score\nAda,95\nBob,82.5\n") + self.assertEqual(rows, [("Ada", 95.0), ("Bob", 82.5)]) + + def test_render_has_header_rows_and_summary(self): + text = report.render([("Bob", 82.5), ("Ada", 95.0)]) + lines = text.split("\n") + self.assertEqual(lines[0], "Name | Score | Grade") + self.assertEqual(lines[1], "Ada | 95.0 | A") + self.assertEqual(lines[2], "Bob | 82.5 | B") + self.assertTrue(lines[3].startswith("Class: 2 students")) + + def test_render_nothing(self): + self.assertEqual(report.render([]), "No scores") + + +class Cli(unittest.TestCase): + def test_reads_stdin(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main([], io.StringIO("name,score\nAda,95\n"), out, err) + self.assertEqual(code, 0) + self.assertIn("Ada | 95.0 | A", out.getvalue()) + self.assertEqual(err.getvalue(), "") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/case.json b/evals/routing-2026-10-23/cases/hf-semver-ranges/case.json new file mode 100644 index 0000000..61070bb --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/case.json @@ -0,0 +1,30 @@ +{ + "name": "hf-semver-ranges", + "family": "hf", + "kind": "careful", + "description": "Two parallel tracks build version comparison (full Semantic Versioning precedence rules, many edge cases) and range checking on top of it.", + "prompt": "", + "goal": "Build a small Python package `pkgcheck` (standard library only) that compares package versions and checks them against version ranges, with a command line front end. The directory `pkgcheck/` and its `__init__.py` already exist, and `tests/test_visible.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `semver` writes `pkgcheck/semver.py`; `constraints` writes `pkgcheck/constraints.py` and `pkgcheck/cli.py`. The shared interface is `semver.compare(a, b)` and `semver.parse_core(version)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.", + "tracks": [ + { + "name": "semver", + "prompt": "Write the file `pkgcheck/semver.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/constraints.py` and `pkgcheck/cli.py` at the same time and will import your two public functions exactly as described here, so keep their names, arguments and return types.\n\nVersion strings look like `MAJOR.MINOR.PATCH`, optionally followed by a prerelease part introduced by `-` and/or a build part introduced by `+`: `1.4.2`, `2.0.0-rc.1`, `1.0.0+build.7`, `1.0.0-beta.2+exp.sha.5114f85`.\n\nHow to read a version string:\n1. Split off the build part first: it starts at the first `+` and runs to the end. It is ignored completely, whatever it contains (even nothing, so `1.2.3+` is valid). A `-` inside the build part is not a prerelease marker: `1.0.0+build-5` has no prerelease.\n2. In what remains, the core is the text before the first `-` (or all of it if there is no `-`). It must be exactly three dot-separated, non-empty strings made only of the ASCII digits 0-9 (`1.2.3` is fine; `1.2`, `1.2.3.4`, `v1.2.3`, `1.x.3`, `1..3` and the empty string are malformed).\n3. If there is a `-`, everything after the first `-` (up to the build part) is the prerelease. It may itself contain `-` characters: `1.0.0-x-y` has the prerelease `x-y`. Split it on `.` into identifiers. No identifier may be empty, so `1.2.3-`, `1.2.3-a..b`, `1.2.3-.a` and `1.2.3-a.` are malformed.\n4. A malformed string (including a value that is not a str) raises `ValueError`.\n\nProvide these two public functions:\n- `parse_core(version)` -> `tuple[int, int, int]`: the `(major, minor, patch)` integers. It checks the whole string by the rules above and raises `ValueError` if it is malformed. Example: `parse_core(\"1.2.3-rc.1+b.5\") == (1, 2, 3)`.\n- `compare(a, b)` -> int: returns `-1`, `0` or `1` when `a` has lower, equal or higher precedence than `b`. It raises `ValueError` if either string is malformed. Precedence works like this:\n - Compare the cores as numbers, part by part from the left (`1.10.0` is higher than `1.9.0`, `2.0.0` is lower than `10.0.0`).\n - If the cores are equal, a version without a prerelease is higher than one with a prerelease (`1.0.0` is higher than `1.0.0-rc.1`).\n - If both have prereleases, compare the identifiers pairwise from left to right and stop at the first pair that differs. Two identifiers made only of digits compare as integers (`2` is lower than `10`). An identifier made only of digits is lower than any other identifier (`1` is lower than `a`, and `2` is lower than `1a`). Two identifiers that are not both all digits compare as plain strings by character code (`alpha` is lower than `beta`, `Beta` is lower than `alpha`, and `rc2` is higher than `rc10`).\n - If every compared pair is equal, the prerelease with more identifiers is higher (`alpha` is lower than `alpha.1`).\n - Build parts never matter.\n\nFor orientation, this list is in increasing order: `1.0.0-alpha`, `1.0.0-alpha.1`, `1.0.0-alpha.beta`, `1.0.0-beta`, `1.0.0-beta.2`, `1.0.0-beta.11`, `1.0.0-rc.1`, `1.0.0`. Equal versions compare as 0, and `compare(b, a)` is always the negative of `compare(a, b)`.", + "files": [ + "pkgcheck/semver.py" + ] + }, + { + "name": "constraints", + "prompt": "Write the files `pkgcheck/constraints.py` and `pkgcheck/cli.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/semver.py` at the same time; do not write that file. Import it with `from pkgcheck import semver` and use only these two functions from it, which will behave as follows:\n- `semver.compare(a, b)` -> int: `-1`, `0` or `1` as version string `a` has lower, equal or higher precedence than `b` (Semantic Versioning precedence; the build part after `+` is ignored; a prerelease version is lower than the same core without one). Raises `ValueError` if either string is malformed.\n- `semver.parse_core(version)` -> `tuple[int, int, int]`: `(major, minor, patch)` as ints, for example `parse_core(\"1.2.3-rc.1+b.5\") == (1, 2, 3)`. Raises `ValueError` if the string is malformed.\nDo not re-implement version comparison yourself.\n\n`pkgcheck/constraints.py` provides:\n- `satisfies(version, spec)` -> bool (a real `bool`). `spec` is one or more clauses separated by commas, and the version satisfies the spec only if it satisfies every clause. Whitespace around a clause, and between its operator and its version, is ignored. A clause is an operator followed by a version string. The operators are `==`, `!=`, `>=`, `<=`, `>`, `<`, `^` and `~` (match the two-character ones before the one-character ones), and a clause with no operator means `==`. The version in a clause may carry a prerelease and/or build part.\n Meaning, with `c = semver.compare(version, clause_version)`: `==` is `c == 0`, `!=` is `c != 0`, `>=` is `c >= 0`, `<=` is `c <= 0`, `>` is `c > 0`, `<` is `c < 0`.\n `~X.Y.Z` (tilde) means: at least that version, and below `X.(Y+1).0`.\n `^X.Y.Z` (caret) means: at least that version, and below an upper bound taken from the first non-zero part: if X > 0 the bound is `(X+1).0.0`; otherwise if Y > 0 it is `0.(Y+1).0`; otherwise it is `0.0.(Z+1)`. So `^1.2.3` is below `2.0.0`, `^0.2.3` is below `0.3.0` and `^0.0.3` is below `0.0.4`. Take X, Y, Z from `semver.parse_core` of the clause version (any prerelease or build part on it still counts for the \"at least\" test), and make both tests with `semver.compare`.\n There is no special rule for prereleases: a version satisfies a bound exactly when `compare` says so, so `2.0.0-rc.1` does satisfy `^1.2.3` because it is below `2.0.0`. That is intended.\n Raise `ValueError` for: an empty spec, an empty clause (`\">=1.0.0,\"`, `\">=1.0.0,,<2.0.0\"`), a clause with an unknown operator or no version (`\"=>1.0.0\"`, `\"*\"`, `\"~\"`), a malformed clause version (`\">=1.0\"`), or a malformed `version` argument. Every clause must be checked for errors before any clause is evaluated, so an invalid clause raises even when an earlier clause would already have failed.\n- `newest(versions, spec)` -> str or None: `versions` is any iterable of version strings. Return the satisfying version with the highest precedence, as the original string, or `None` if none satisfies `spec`. When two satisfying versions have equal precedence (for example `1.0.0+a` and `1.0.0+b`) return the one that comes first. A malformed entry in `versions` raises `ValueError`.\n\n`pkgcheck/cli.py` provides:\n- `main(argv=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. `argv[0]` is a spec and the rest are versions. With fewer than two items write `usage: python -m pkgcheck.cli SPEC VERSION [VERSION ...]` and a newline to stderr and return 2. Otherwise call `constraints.newest(argv[1:], argv[0])`: if it raises `ValueError`, write `error: ` and a newline to stderr and return 2; if it returns `None`, write `none` and a newline to stdout and return 1; otherwise write the version and a newline to stdout and return 0.\n- End the file with `if __name__ == \"__main__\": raise SystemExit(main())`.", + "files": [ + "pkgcheck/constraints.py", + "pkgcheck/cli.py" + ] + } + ], + "gates": [ + "python3 -m unittest discover -s tests -q", + "python3 -c \"from pkgcheck import semver, constraints, cli\"" + ], + "expected_outcome": "pkgcheck/semver.py implements the full precedence rules (numeric core, numeric versus alphanumeric prerelease identifiers, length tie-break, build metadata ignored, strict validation) and pkgcheck/constraints.py and pkgcheck/cli.py use it as specified; the gates pass." +} diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hf-semver-ranges/hidden/test_hidden.py new file mode 100644 index 0000000..26f0df2 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/hidden/test_hidden.py @@ -0,0 +1,330 @@ +import functools +import io +import itertools +import unittest + +from pkgcheck import cli, constraints, semver + +SEMVER_ORG_CHAIN = [ + "1.0.0-alpha", + "1.0.0-alpha.1", + "1.0.0-alpha.beta", + "1.0.0-beta", + "1.0.0-beta.2", + "1.0.0-beta.11", + "1.0.0-rc.1", + "1.0.0", +] + + +class CompareOrdering(unittest.TestCase): + def test_the_published_example_chain(self): + for i, j in itertools.combinations(range(len(SEMVER_ORG_CHAIN)), 2): + low, high = SEMVER_ORG_CHAIN[i], SEMVER_ORG_CHAIN[j] + self.assertEqual(semver.compare(low, high), -1, (low, high)) + self.assertEqual(semver.compare(high, low), 1, (high, low)) + + def test_equal_to_itself(self): + for version in SEMVER_ORG_CHAIN + ["0.0.0", "10.20.30"]: + self.assertEqual(semver.compare(version, version), 0, version) + + def test_core_is_numeric(self): + self.assertEqual(semver.compare("1.10.0", "1.9.0"), 1) + self.assertEqual(semver.compare("2.0.0", "10.0.0"), -1) + self.assertEqual(semver.compare("1.0.10", "1.0.9"), 1) + self.assertEqual(semver.compare("0.0.1", "0.0.0"), 1) + + def test_prerelease_never_beats_a_higher_core(self): + self.assertEqual(semver.compare("1.0.0-zzz.99", "1.0.1"), -1) + self.assertEqual(semver.compare("2.0.0-alpha", "1.99.99"), 1) + + def test_numeric_identifiers_compare_as_numbers(self): + self.assertEqual(semver.compare("1.0.0-2", "1.0.0-10"), -1) + self.assertEqual(semver.compare("1.0.0-alpha.2", "1.0.0-alpha.11"), -1) + self.assertEqual(semver.compare("1.0.0-rc.9", "1.0.0-rc.10"), -1) + + def test_numeric_is_lower_than_alphanumeric(self): + self.assertEqual(semver.compare("1.0.0-1", "1.0.0-a"), -1) + self.assertEqual(semver.compare("1.0.0-999", "1.0.0-a"), -1) + self.assertEqual(semver.compare("1.0.0-a.1", "1.0.0-a.b"), -1) + self.assertEqual(semver.compare("1.0.0-1a", "1.0.0-2"), 1) + + def test_alphanumeric_identifiers_compare_as_text(self): + self.assertEqual(semver.compare("1.0.0-alpha", "1.0.0-beta"), -1) + self.assertEqual(semver.compare("1.0.0-Beta", "1.0.0-alpha"), -1) + self.assertEqual(semver.compare("1.0.0-rc1", "1.0.0-rc10"), -1) + self.assertEqual(semver.compare("1.0.0-rc2", "1.0.0-rc10"), 1) + + def test_more_identifiers_win_when_the_rest_is_equal(self): + self.assertEqual(semver.compare("1.0.0-a", "1.0.0-a.0"), -1) + self.assertEqual(semver.compare("1.0.0-a.1.2", "1.0.0-a.1"), 1) + self.assertEqual(semver.compare("1.0.0-a.1", "1.0.0-a.1"), 0) + + def test_first_differing_identifier_decides(self): + self.assertEqual(semver.compare("1.0.0-a.5.z", "1.0.0-a.6.a"), -1) + self.assertEqual(semver.compare("1.0.0-b.1", "1.0.0-a.99.99"), 1) + + def test_antisymmetric_over_a_mixed_list(self): + versions = SEMVER_ORG_CHAIN + [ + "1.0.0-1", "1.0.0-1a", "1.0.0-x-y", "0.9.9", "1.0.1", "1.0.0-alpha.0", + ] + for a in versions: + for b in versions: + self.assertEqual( + semver.compare(a, b), -semver.compare(b, a), (a, b) + ) + + def test_transitive_over_a_mixed_list(self): + versions = SEMVER_ORG_CHAIN + ["1.0.0-1", "1.0.0-1a", "1.0.0-x-y", "0.9.9", "1.0.1"] + ordered = sorted(versions, key=functools.cmp_to_key(semver.compare)) + for i, j in itertools.combinations(range(len(ordered)), 2): + self.assertLessEqual(semver.compare(ordered[i], ordered[j]), 0) + + +class BuildMetadataAndHyphens(unittest.TestCase): + def test_build_is_ignored(self): + self.assertEqual(semver.compare("1.0.0+a", "1.0.0+b"), 0) + self.assertEqual(semver.compare("1.0.0+a", "1.0.0"), 0) + self.assertEqual(semver.compare("1.0.0-rc.1+x", "1.0.0-rc.1"), 0) + self.assertEqual(semver.compare("1.0.0-rc.1+zzz", "1.0.0-rc.2+aaa"), -1) + self.assertEqual(semver.compare("1.0.1+a", "1.0.0+z"), 1) + + def test_empty_build_is_allowed(self): + self.assertEqual(semver.compare("1.2.3+", "1.2.3"), 0) + + def test_hyphen_in_build_is_not_a_prerelease(self): + self.assertEqual(semver.compare("1.0.0+build-5", "1.0.0"), 0) + self.assertEqual(semver.compare("1.0.0+exp-sha-5114f85", "1.0.0-alpha"), 1) + + def test_hyphen_inside_prerelease(self): + self.assertEqual(semver.compare("1.0.0-x-y", "1.0.0-x-z"), -1) + self.assertEqual(semver.compare("1.0.0-alpha-1", "1.0.0"), -1) + self.assertEqual(semver.compare("1.0.0-a-1", "1.0.0-a-1"), 0) + self.assertEqual(semver.compare("1.0.0-a-1.2", "1.0.0-a-1.10"), -1) + + def test_plus_inside_the_build_part(self): + self.assertEqual(semver.compare("1.0.0+a+b", "1.0.0"), 0) + + +class Malformed(unittest.TestCase): + BAD = [ + "", "1", "1.2", "1.2.3.4", "a.b.c", "1.2.x", "1..3", ".1.2", "1.2.", + "v1.2.3", "-1.2.3", "1.2.3-", "1.2.3-a..b", "1.2.3-.a", "1.2.3-a.", + "1.2.-3", "+build", "1.2.3 4", + ] + + def test_compare_rejects_bad_versions_on_either_side(self): + for bad in self.BAD: + with self.assertRaises(ValueError, msg=repr(bad)): + semver.compare(bad, "1.0.0") + with self.assertRaises(ValueError, msg=repr(bad)): + semver.compare("1.0.0", bad) + + def test_parse_core_rejects_the_same(self): + for bad in self.BAD: + with self.assertRaises(ValueError, msg=repr(bad)): + semver.parse_core(bad) + + +class ParseCore(unittest.TestCase): + def test_values(self): + self.assertEqual(semver.parse_core("1.2.3"), (1, 2, 3)) + self.assertEqual(semver.parse_core("0.0.0"), (0, 0, 0)) + self.assertEqual(semver.parse_core("10.20.30"), (10, 20, 30)) + self.assertEqual(semver.parse_core("1.2.3-rc.1+b.5"), (1, 2, 3)) + self.assertEqual(semver.parse_core("4.5.6-x-y"), (4, 5, 6)) + self.assertEqual(semver.parse_core("4.5.6+build-1"), (4, 5, 6)) + + def test_results_are_ints(self): + for part in semver.parse_core("7.8.9"): + self.assertIsInstance(part, int) + + +class Operators(unittest.TestCase): + def test_each_operator(self): + cases = [ + ("1.2.3", "==1.2.3", True), ("1.2.3", "==1.2.4", False), + ("1.2.3", "1.2.3", True), ("1.2.3", "1.2.4", False), + ("1.2.3", "!=1.2.3", False), ("1.2.3", "!=1.2.4", True), + ("1.2.3", ">=1.2.3", True), ("1.2.2", ">=1.2.3", False), + ("1.2.3", ">1.2.3", False), ("1.2.4", ">1.2.3", True), + ("1.2.3", "<=1.2.3", True), ("1.2.4", "<=1.2.3", False), + ("1.2.3", "<1.2.3", False), ("1.2.2", "<1.2.3", True), + ] + for version, spec, expected in cases: + self.assertIs(constraints.satisfies(version, spec), expected, (version, spec)) + + def test_numbers_not_text(self): + self.assertTrue(constraints.satisfies("1.10.0", ">1.9.0")) + self.assertTrue(constraints.satisfies("10.0.0", ">=9.0.0")) + + def test_whitespace(self): + self.assertTrue(constraints.satisfies("1.2.3", ">= 1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", " >=1.0.0 , < 2.0.0 ")) + self.assertTrue(constraints.satisfies("1.2.3", " 1.2.3 ")) + + def test_prerelease_ordering_is_used(self): + self.assertTrue(constraints.satisfies("1.0.0-rc.1", ">=1.0.0-beta")) + self.assertTrue(constraints.satisfies("1.0.0-rc.1", "<1.0.0")) + self.assertTrue(constraints.satisfies("1.0.0-alpha.2", "<1.0.0-alpha.11")) + self.assertFalse(constraints.satisfies("1.0.0-beta.11", "<1.0.0-beta.2")) + + def test_build_metadata_is_ignored(self): + self.assertTrue(constraints.satisfies("1.2.3+abc", "==1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "==1.2.3+zzz")) + + def test_several_clauses_are_anded(self): + spec = ">=1.0.0, !=1.3.0, <2.0.0" + self.assertFalse(constraints.satisfies("1.3.0", spec)) + self.assertTrue(constraints.satisfies("1.4.0", spec)) + self.assertTrue(constraints.satisfies("1.0.0", spec)) + self.assertFalse(constraints.satisfies("2.0.0", spec)) + self.assertFalse(constraints.satisfies("0.9.9", spec)) + + +class CaretAndTilde(unittest.TestCase): + def check(self, spec, inside, outside): + for version in inside: + self.assertTrue(constraints.satisfies(version, spec), (spec, version)) + for version in outside: + self.assertFalse(constraints.satisfies(version, spec), (spec, version)) + + def test_caret_major(self): + self.check("^1.2.3", ["1.2.3", "1.2.4", "1.9.9", "1.10.0"], ["1.2.2", "2.0.0", "0.9.0"]) + + def test_caret_zero_minor(self): + self.check("^0.2.3", ["0.2.3", "0.2.9"], ["0.2.2", "0.3.0", "1.0.0"]) + + def test_caret_zero_zero(self): + self.check("^0.0.3", ["0.0.3"], ["0.0.2", "0.0.4", "0.1.0"]) + + def test_caret_all_zero(self): + self.check("^0.0.0", ["0.0.0"], ["0.0.1", "0.1.0"]) + + def test_tilde(self): + self.check("~1.2.3", ["1.2.3", "1.2.9"], ["1.2.2", "1.3.0", "2.0.0"]) + self.check("~0.2.0", ["0.2.0", "0.2.5"], ["0.3.0", "0.1.9"]) + self.check("~1.0.0", ["1.0.0", "1.0.7"], ["1.1.0"]) + + def test_two_digit_parts(self): + self.check("^1.9.0", ["1.10.0", "1.19.2"], ["2.0.0"]) + self.check("~1.9.0", ["1.9.5"], ["1.10.0"]) + self.check("^0.9.0", ["0.9.7"], ["0.10.0"]) + + def test_lower_bound_may_carry_a_prerelease(self): + self.check( + "^1.2.3-beta.1", + ["1.2.3-beta.2", "1.2.3-beta.11", "1.2.3", "1.5.0"], + ["1.2.3-alpha", "1.2.3-beta", "2.0.0"], + ) + self.check("~1.2.3-rc.1", ["1.2.3-rc.1", "1.2.3"], ["1.2.3-rc.0", "1.3.0"]) + + def test_prerelease_of_the_next_major_is_inside_the_range(self): + # No special prerelease rule: it is below the upper bound 2.0.0. + self.assertTrue(constraints.satisfies("2.0.0-rc.1", "^1.2.3")) + self.assertFalse(constraints.satisfies("2.0.0", "^1.2.3")) + + def test_caret_combined_with_another_clause(self): + self.assertTrue(constraints.satisfies("1.4.0", "^1.2.3, !=1.3.0")) + self.assertFalse(constraints.satisfies("1.3.0", "^1.2.3, !=1.3.0")) + + def test_build_metadata_in_the_bound(self): + self.assertTrue(constraints.satisfies("1.2.4", "^1.2.3+build")) + + +class SpecErrors(unittest.TestCase): + def test_bad_specs(self): + for spec in [ + "", " ", ">=1.0.0,", ",>=1.0.0", ">=1.0.0,,<2.0.0", "=>1.0.0", "=1.0.0", + "*", "~", "^", ">=", ">=1.0", "1.0", "latest", ">= ", "~1.2", "^x.y.z", + ]: + with self.assertRaises(ValueError, msg=repr(spec)): + constraints.satisfies("1.0.0", spec) + + def test_bad_version(self): + for version in ["1.2", "", "x", "1.2.3-"]: + with self.assertRaises(ValueError, msg=repr(version)): + constraints.satisfies(version, ">=1.0.0") + + def test_every_clause_is_checked_even_after_one_fails(self): + with self.assertRaises(ValueError): + constraints.satisfies("1.0.0", "<0.5.0,garbage") + with self.assertRaises(ValueError): + constraints.satisfies("1.0.0", "<0.5.0, >=1.x") + + +class Newest(unittest.TestCase): + def test_picks_the_highest_match(self): + versions = ["1.0.0", "1.10.0", "1.9.0", "2.0.0"] + self.assertEqual(constraints.newest(versions, "^1.0.0"), "1.10.0") + self.assertEqual(constraints.newest(versions, ">=1.0.0"), "2.0.0") + self.assertEqual(constraints.newest(versions, "<2.0.0"), "1.10.0") + self.assertEqual( + constraints.newest(["1.0.0", "2.0.0-rc.1", "2.0.0"], "<2.0.0"), "2.0.0-rc.1" + ) + + def test_prerelease_ordering(self): + self.assertEqual( + constraints.newest(["1.0.0-beta.2", "1.0.0-beta.11"], ">=1.0.0-alpha"), + "1.0.0-beta.11", + ) + self.assertEqual( + constraints.newest(["1.0.0-alpha", "1.0.0-beta", "1.0.0-alpha.1"], ">=1.0.0-alpha"), + "1.0.0-beta", + ) + + def test_nothing_matches(self): + self.assertIsNone(constraints.newest(["1.0.0", "2.0.0"], "<1.0.0")) + self.assertIsNone(constraints.newest([], ">=0.0.0")) + + def test_ties_keep_the_first(self): + self.assertEqual(constraints.newest(["1.0.0+a", "1.0.0+b"], "==1.0.0"), "1.0.0+a") + self.assertEqual(constraints.newest(["1.0.0+b", "1.0.0+a"], ">=1.0.0"), "1.0.0+b") + + def test_returns_the_original_string(self): + self.assertEqual(constraints.newest(["1.2.3+meta"], ">=1.0.0"), "1.2.3+meta") + + def test_accepts_any_iterable(self): + self.assertEqual(constraints.newest(iter(["1.0.0", "1.1.0"]), "^1.0.0"), "1.1.0") + + def test_malformed_entry_raises(self): + with self.assertRaises(ValueError): + constraints.newest(["1.0.0", "oops"], ">=1.0.0") + + +class CommandLine(unittest.TestCase): + def run_main(self, argv): + out, err = io.StringIO(), io.StringIO() + code = cli.main(argv, out, err) + return code, out.getvalue(), err.getvalue() + + def test_prints_the_newest_match(self): + code, out, err = self.run_main(["^1.0.0", "1.0.0", "1.10.0", "1.9.0", "2.0.0"]) + self.assertEqual((code, out, err), (0, "1.10.0\n", "")) + + def test_prerelease_order_in_output(self): + code, out, _ = self.run_main([">=1.0.0-alpha", "1.0.0-beta.2", "1.0.0-beta.11"]) + self.assertEqual((code, out), (0, "1.0.0-beta.11\n")) + + def test_no_match(self): + code, out, err = self.run_main(["<1.0.0", "1.0.0", "2.0.0"]) + self.assertEqual((code, out, err), (1, "none\n", "")) + + def test_bad_spec_or_version(self): + for argv in (["=>1.0.0", "1.0.0"], [">=1.0.0", "1.0"]): + code, out, err = self.run_main(argv) + self.assertEqual(code, 2, argv) + self.assertEqual(out, "", argv) + self.assertTrue(err.startswith("error: "), argv) + self.assertTrue(err.endswith("\n"), argv) + + def test_usage(self): + for argv in ([], [">=1.0.0"]): + code, out, err = self.run_main(argv) + self.assertEqual(code, 2, argv) + self.assertEqual(out, "", argv) + self.assertTrue(err.startswith("usage:"), argv) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/cli.py b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/cli.py new file mode 100644 index 0000000..e340c99 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/cli.py @@ -0,0 +1,30 @@ +"""Command line front end: python -m pkgcheck.cli SPEC VERSION [VERSION ...]""" + +import sys + +from pkgcheck import constraints + +USAGE = "usage: python -m pkgcheck.cli SPEC VERSION [VERSION ...]" + + +def main(argv=None, stdout=None, stderr=None): + argv = sys.argv[1:] if argv is None else argv + stdout = sys.stdout if stdout is None else stdout + stderr = sys.stderr if stderr is None else stderr + if len(argv) < 2: + stderr.write(USAGE + "\n") + return 2 + try: + best = constraints.newest(argv[1:], argv[0]) + except ValueError as exc: + stderr.write(f"error: {exc}\n") + return 2 + if best is None: + stdout.write("none\n") + return 1 + stdout.write(best + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/constraints.py b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/constraints.py new file mode 100644 index 0000000..69b9b7c --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/constraints.py @@ -0,0 +1,62 @@ +"""Version ranges built on top of pkgcheck.semver.""" + +from pkgcheck import semver + +_OPERATORS = ("==", "!=", ">=", "<=", ">", "<", "^", "~") + + +def _parse_clause(text): + text = text.strip() + if not text: + raise ValueError("empty clause") + for operator in _OPERATORS: + if text.startswith(operator): + bound = text[len(operator):].strip() + break + else: + operator, bound = "==", text + semver.parse_core(bound) # raises ValueError when malformed + return operator, bound + + +def _upper_bound(operator, bound): + major, minor, patch = semver.parse_core(bound) + if operator == "~": + return f"{major}.{minor + 1}.0" + if major > 0: + return f"{major + 1}.0.0" + if minor > 0: + return f"0.{minor + 1}.0" + return f"0.0.{patch + 1}" + + +def _holds(version, operator, bound): + if operator in ("^", "~"): + upper = _upper_bound(operator, bound) + return semver.compare(version, bound) >= 0 and semver.compare(version, upper) < 0 + result = semver.compare(version, bound) + if operator == "==": + return result == 0 + if operator == "!=": + return result != 0 + if operator == ">=": + return result >= 0 + if operator == "<=": + return result <= 0 + if operator == ">": + return result > 0 + return result < 0 + + +def satisfies(version, spec): + clauses = [_parse_clause(part) for part in spec.split(",")] + semver.parse_core(version) + return all(_holds(version, operator, bound) for operator, bound in clauses) + + +def newest(versions, spec): + best = None + for version in versions: + if satisfies(version, spec) and (best is None or semver.compare(version, best) > 0): + best = version + return best diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/semver.py b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/semver.py new file mode 100644 index 0000000..3783512 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/reference/pkgcheck/semver.py @@ -0,0 +1,61 @@ +"""Version precedence, following the Semantic Versioning rules.""" + +import re + +_CORE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+") +_NUMERIC = re.compile(r"[0-9]+") + + +def _split(version): + """Return ((major, minor, patch), identifiers-or-None). Raises ValueError.""" + if not isinstance(version, str): + raise ValueError(f"malformed version: {version!r}") + main, _plus, _build = version.partition("+") + core, dash, prerelease = main.partition("-") + if not _CORE.fullmatch(core): + raise ValueError(f"malformed version: {version!r}") + numbers = tuple(int(part) for part in core.split(".")) + identifiers = None + if dash: + identifiers = prerelease.split(".") + if any(ident == "" for ident in identifiers): + raise ValueError(f"malformed version: {version!r}") + return numbers, identifiers + + +def parse_core(version): + """Return (major, minor, patch) as ints. Raises ValueError if malformed.""" + return _split(version)[0] + + +def _compare_identifier(x, y): + x_num = bool(_NUMERIC.fullmatch(x)) + y_num = bool(_NUMERIC.fullmatch(y)) + if x_num and y_num: + a, b = int(x), int(y) + elif x_num: + return -1 + elif y_num: + return 1 + else: + a, b = x, y + return (a > b) - (a < b) + + +def compare(a, b): + """Return -1, 0 or 1 as version `a` has lower, equal or higher precedence than `b`.""" + core_a, pre_a = _split(a) + core_b, pre_b = _split(b) + if core_a != core_b: + return -1 if core_a < core_b else 1 + if pre_a is None and pre_b is None: + return 0 + if pre_a is None: + return 1 + if pre_b is None: + return -1 + for x, y in zip(pre_a, pre_b): + result = _compare_identifier(x, y) + if result: + return result + return (len(pre_a) > len(pre_b)) - (len(pre_a) < len(pre_b)) diff --git a/evals/routing-2026-10-23/cases/hf-semver-ranges/scaffold.sh b/evals/routing-2026-10-23/cases/hf-semver-ranges/scaffold.sh new file mode 100644 index 0000000..fd48e5a --- /dev/null +++ b/evals/routing-2026-10-23/cases/hf-semver-ranges/scaffold.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# pkgcheck + +Version precedence and version-range checks for package versions such as +`1.4.2`, `2.0.0-rc.1` or `1.0.0+build.7`. + +Layout: + +- `pkgcheck/semver.py` comparing two versions +- `pkgcheck/constraints.py` `satisfies(version, spec)` and `newest(versions, spec)` +- `pkgcheck/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'pkgcheck' +cat > 'pkgcheck/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""pkgcheck: compare versions and check them against ranges.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for pkgcheck.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_visible.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from pkgcheck import cli, constraints, semver + + +class Compare(unittest.TestCase): + def test_plain_ordering(self): + self.assertEqual(semver.compare("1.2.3", "1.2.4"), -1) + self.assertEqual(semver.compare("2.0.0", "1.9.9"), 1) + self.assertEqual(semver.compare("1.2.3", "1.2.3"), 0) + + def test_numbers_are_not_compared_as_text(self): + self.assertEqual(semver.compare("1.10.0", "1.9.0"), 1) + + def test_prerelease_is_lower_than_release(self): + self.assertEqual(semver.compare("1.0.0-alpha", "1.0.0"), -1) + self.assertEqual(semver.compare("1.0.0", "1.0.0-rc.1"), 1) + + def test_core(self): + self.assertEqual(semver.parse_core("3.4.5"), (3, 4, 5)) + + def test_malformed(self): + with self.assertRaises(ValueError): + semver.compare("1.2", "1.2.3") + + +class Satisfies(unittest.TestCase): + def test_comparison_operators(self): + self.assertTrue(constraints.satisfies("1.2.3", ">=1.2.0")) + self.assertFalse(constraints.satisfies("1.2.3", "<1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "==1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "!=1.2.4")) + + def test_caret_and_tilde(self): + self.assertTrue(constraints.satisfies("1.9.0", "^1.2.3")) + self.assertFalse(constraints.satisfies("2.0.0", "^1.2.3")) + self.assertTrue(constraints.satisfies("1.2.9", "~1.2.3")) + self.assertFalse(constraints.satisfies("1.3.0", "~1.2.3")) + + def test_several_clauses(self): + self.assertTrue(constraints.satisfies("1.4.0", ">=1.2.0, <1.5.0")) + self.assertFalse(constraints.satisfies("1.5.0", ">=1.2.0, <1.5.0")) + + def test_newest(self): + versions = ["1.0.0", "1.10.0", "1.9.0", "2.0.0"] + self.assertEqual(constraints.newest(versions, "^1.0.0"), "1.10.0") + self.assertIsNone(constraints.newest(versions, "<1.0.0")) + + +class Cli(unittest.TestCase): + def test_prints_the_newest_match(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main(["^1.0.0", "1.0.0", "1.1.0", "2.0.0"], out, err) + self.assertEqual(code, 0) + self.assertEqual(out.getvalue(), "1.1.0\n") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/evals/routing-2026-10-23/cases/hr-limiter-starve/case.json b/evals/routing-2026-10-23/cases/hr-limiter-starve/case.json new file mode 100644 index 0000000..0047f50 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-limiter-starve/case.json @@ -0,0 +1,8 @@ +{ + "name": "hr-limiter-starve", + "family": "hr", + "kind": "careful", + "description": "Subtle token-bucket bug: whole-token refill math throws away fractional progress whenever the bucket is looked at more often than once per token interval; naive fixes bank idle time.", + "prompt": "Our rate limiter has a nasty problem. With a policy of 2 requests per 4 seconds, a client that gets a 429 and keeps retrying every quarter of a second is never let back in, even after waiting far longer than the two seconds it should take for a slot to free up. A client that goes quiet for a few seconds and then tries once gets through fine, and people polling the remaining-quota status seem to cause the same trouble. The tests in tests/ all pass, so they don't cover it. Ask a subagent to review the token bucket in `ratelimit/bucket.py` carefully and work out what actually causes this (not a guess), then fix it so those clients get a slot back on time, without changing how the limiter behaves for clients that never hit the problem.", + "expected_outcome": "The bucket keeps fractional refill progress across frequent calls so retrying or polling clients regain a token on schedule, does not bank time spent full, and leaves all existing bucket and limiter behaviour unchanged." +} diff --git a/evals/routing-2026-10-23/cases/hr-limiter-starve/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hr-limiter-starve/hidden/test_hidden.py new file mode 100644 index 0000000..9cb034e --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-limiter-starve/hidden/test_hidden.py @@ -0,0 +1,180 @@ +import unittest + +from ratelimit.bucket import TokenBucket +from ratelimit.clock import ManualClock +from ratelimit.limiter import Limiter +from ratelimit.middleware import guard +from ratelimit.policy import Policy + +# All times below are multiples of 0.25 and all rates are binary fractions, +# so the arithmetic is exact and the expected step counts are not rounding luck. + + +def make(capacity=2, rate=0.5): + clock = ManualClock() + return clock, TokenBucket(capacity, rate, clock) + + +def drain(bucket): + while bucket.allow(): + pass + + +class PollingDoesNotStarveRefill(unittest.TestCase): + def test_status_polls_between_requests(self): + clock, bucket = make() + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + for _ in range(8): + clock.advance(0.25) + bucket.available() + self.assertTrue(bucket.allow()) + + def test_rejected_retries_get_in_on_time(self): + clock, bucket = make() + drain(bucket) + first = None + for step in range(1, 41): + clock.advance(0.25) + if bucket.allow(): + first = step + break + self.assertEqual(first, 8) + + def test_faster_rate_gets_in_on_time(self): + clock, bucket = make(capacity=3, rate=1.5) + drain(bucket) + first = None + for step in range(1, 41): + clock.advance(0.25) + if bucket.allow(): + first = step + break + self.assertEqual(first, 3) + + def test_steady_client_under_the_limit_is_never_refused(self): + clock, bucket = make() + for i in range(20): + clock.advance(1.0) + bucket.available() + clock.advance(1.0) + self.assertTrue(bucket.allow(), f"request {i}") + + def test_throughput_of_a_client_that_retries_constantly(self): + clock, bucket = make() + allowed = 0 + for _ in range(160): # 40 seconds in quarter-second steps + clock.advance(0.25) + if bucket.allow(): + allowed += 1 + # 2 from the full bucket plus one token every 2 seconds + self.assertIn(allowed, (21, 22)) + + def test_partial_progress_is_remembered(self): + clock, bucket = make() + drain(bucket) + clock.advance(1.0) + self.assertFalse(bucket.allow()) + clock.advance(1.0) + self.assertTrue(bucket.allow()) + + def test_available_counts_whole_tokens(self): + clock, bucket = make() + drain(bucket) + clock.advance(1.0) + self.assertEqual(int(bucket.available()), 0) + clock.advance(1.0) + self.assertEqual(int(bucket.available()), 1) + clock.advance(100.0) + self.assertEqual(int(bucket.available()), 2) + + +class IdleTimeWhileFullIsNotBanked(unittest.TestCase): + def test_refill_clock_starts_when_a_token_is_taken(self): + clock, bucket = make() + clock.advance(1.75) # bucket has been full the whole time + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + clock.advance(0.25) # t = 2.0: only a quarter second since the bucket emptied + self.assertFalse(bucket.allow()) + clock.advance(1.5) # t = 3.5 + self.assertFalse(bucket.allow()) + clock.advance(0.25) # t = 3.75: two seconds since it emptied + self.assertTrue(bucket.allow()) + + def test_a_look_at_a_full_bucket_changes_nothing(self): + clock, bucket = make() + clock.advance(1.75) + self.assertEqual(int(bucket.available()), 2) + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + clock.advance(0.25) + self.assertFalse(bucket.allow()) + + def test_never_more_than_capacity_after_long_idle(self): + clock, bucket = make() + clock.advance(1000) + self.assertEqual(int(bucket.available()), 2) + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + +class OtherBehaviourUnchanged(unittest.TestCase): + def test_burst_then_refused(self): + _, bucket = make() + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_asking_for_several_tokens(self): + clock, bucket = make(capacity=5, rate=1.0) + self.assertTrue(bucket.allow(3)) + self.assertFalse(bucket.allow(3)) + clock.advance(1.0) + self.assertTrue(bucket.allow(3)) + + def test_refused_request_takes_nothing(self): + clock, bucket = make(capacity=5, rate=1.0) + self.assertTrue(bucket.allow(4)) + self.assertFalse(bucket.allow(4)) + self.assertTrue(bucket.allow(1)) + + +class ThroughTheLimiter(unittest.TestCase): + def setUp(self): + self.clock = ManualClock() + self.limiter = Limiter(Policy(limit=2, window=4.0), self.clock) + + def test_status_polling_does_not_block_the_client(self): + self.assertTrue(self.limiter.allow("alice")) + self.assertTrue(self.limiter.allow("alice")) + for _ in range(8): + self.clock.advance(0.25) + self.limiter.status("alice") + self.assertTrue(self.limiter.allow("alice")) + + def test_other_clients_are_not_affected(self): + self.limiter.allow("alice") + self.limiter.allow("alice") + self.clock.advance(0.25) + self.assertEqual(self.limiter.status("bob"), {"remaining": 2, "limit": 2}) + self.assertTrue(self.limiter.allow("bob")) + + def test_guard_lets_a_retrying_client_back_in(self): + for _ in range(2): + self.assertEqual(guard(self.limiter, "carol", lambda: "ok")[0], 200) + first_ok = None + for step in range(1, 41): + self.clock.advance(0.25) + status, body = guard(self.limiter, "carol", lambda: "ok") + if status == 200: + first_ok = step + self.assertEqual(body, "ok") + break + self.assertEqual((status, body), (429, "slow down")) + self.assertEqual(first_ok, 8) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hr-limiter-starve/reference/ratelimit/bucket.py b/evals/routing-2026-10-23/cases/hr-limiter-starve/reference/ratelimit/bucket.py new file mode 100644 index 0000000..f638076 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-limiter-starve/reference/ratelimit/bucket.py @@ -0,0 +1,34 @@ +"""Token bucket. + +The bucket holds up to `capacity` tokens and gains `rate` tokens per second. +A request takes one token, or is refused when the bucket is empty. +""" + + +class TokenBucket: + def __init__(self, capacity, rate, clock): + self.capacity = capacity + self.rate = rate + self.clock = clock + self.tokens = float(capacity) # tokens currently in the bucket (may be fractional) + self.last = clock.now() # when we last looked at the clock + + def _refill(self): + now = self.clock.now() + elapsed = now - self.last + self.last = now + if elapsed > 0: + self.tokens = min(self.capacity, self.tokens + elapsed * self.rate) + + def available(self): + """Whole tokens that could be taken right now.""" + self._refill() + return int(self.tokens) + + def allow(self, amount=1): + """Take `amount` tokens if they are there. Returns True when taken.""" + self._refill() + if self.tokens >= amount: + self.tokens -= amount + return True + return False diff --git a/evals/routing-2026-10-23/cases/hr-limiter-starve/scaffold.sh b/evals/routing-2026-10-23/cases/hr-limiter-starve/scaffold.sh new file mode 100644 index 0000000..51a97ba --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-limiter-starve/scaffold.sh @@ -0,0 +1,252 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# ratelimit + +A small per-client rate limiter. + +- `ratelimit/policy.py` `Policy(limit, window)` +- `ratelimit/bucket.py` `TokenBucket` +- `ratelimit/limiter.py` `Limiter`: one bucket per client +- `ratelimit/middleware.py` `guard(limiter, client_id, handler)` +- `ratelimit/clock.py` `SystemClock` and a `ManualClock` for tests +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""ratelimit: per-client token bucket limiter.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/bucket.py' <<'__SCAFFOLD_FILE_END__' +"""Token bucket. + +The bucket holds up to `capacity` tokens and gains `rate` tokens per second. +A request takes one token, or is refused when the bucket is empty. +""" + + +class TokenBucket: + def __init__(self, capacity, rate, clock): + self.capacity = capacity + self.rate = rate + self.clock = clock + self.tokens = capacity # whole tokens currently in the bucket + self.last = clock.now() # when we last looked at the clock + + def _refill(self): + now = self.clock.now() + elapsed = now - self.last + self.last = now + self.tokens = min(self.capacity, self.tokens + int(elapsed * self.rate)) + + def available(self): + """Whole tokens that could be taken right now.""" + self._refill() + return self.tokens + + def allow(self, amount=1): + """Take `amount` tokens if they are there. Returns True when taken.""" + self._refill() + if self.tokens >= amount: + self.tokens -= amount + return True + return False +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/clock.py' <<'__SCAFFOLD_FILE_END__' +"""Clocks. The limiter only ever asks a clock for the current time in seconds.""" + +import time + + +class SystemClock: + def now(self): + return time.monotonic() + + +class ManualClock: + """A clock that only moves when told to (used by the tests).""" + + def __init__(self, start=0.0): + self._now = float(start) + + def now(self): + return self._now + + def advance(self, seconds): + self._now += seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/limiter.py' <<'__SCAFFOLD_FILE_END__' +"""One bucket per client, all following the same policy.""" + +from ratelimit.bucket import TokenBucket + + +class Limiter: + def __init__(self, policy, clock): + self.policy = policy + self.clock = clock + self._buckets = {} + + def _bucket(self, client_id): + bucket = self._buckets.get(client_id) + if bucket is None: + bucket = TokenBucket(self.policy.limit, self.policy.rate, self.clock) + self._buckets[client_id] = bucket + return bucket + + def allow(self, client_id): + return self._bucket(client_id).allow() + + def status(self, client_id): + """What a quota dashboard shows for this client.""" + return { + "remaining": self._bucket(client_id).available(), + "limit": self.policy.limit, + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/middleware.py' <<'__SCAFFOLD_FILE_END__' +"""Glue between the limiter and a request handler.""" + + +def guard(limiter, client_id, handler): + """Run `handler()` unless the client is over its limit. + + Returns (status_code, body). + """ + if not limiter.allow(client_id): + return 429, "slow down" + return 200, handler() +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/policy.py' <<'__SCAFFOLD_FILE_END__' +"""How many requests a client may make in a time window.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Policy: + limit: int # requests allowed per window (also the burst size) + window: float # seconds + + @property + def rate(self): + """Tokens gained per second.""" + return self.limit / self.window +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for ratelimit.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_bucket.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from ratelimit.bucket import TokenBucket +from ratelimit.clock import ManualClock + + +def make(capacity=2, rate=0.5): + clock = ManualClock() + return clock, TokenBucket(capacity, rate, clock) + + +class BucketBasics(unittest.TestCase): + def test_starts_full(self): + _, bucket = make() + self.assertEqual(bucket.available(), 2) + + def test_burst_then_refused(self): + _, bucket = make() + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_long_wait_refills_to_capacity(self): + clock, bucket = make() + bucket.allow() + bucket.allow() + clock.advance(100) + self.assertEqual(bucket.available(), 2) + + def test_never_holds_more_than_capacity(self): + clock, bucket = make() + clock.advance(1000) + self.assertEqual(bucket.available(), 2) + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_one_token_after_two_seconds(self): + clock, bucket = make() + bucket.allow() + bucket.allow() + clock.advance(2.0) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_asking_for_several_tokens(self): + clock, bucket = make(capacity=5, rate=1.0) + self.assertTrue(bucket.allow(3)) + self.assertFalse(bucket.allow(3)) + clock.advance(1.0) + self.assertTrue(bucket.allow(3)) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_limiter.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from ratelimit.clock import ManualClock +from ratelimit.limiter import Limiter +from ratelimit.middleware import guard +from ratelimit.policy import Policy + + +class LimiterBasics(unittest.TestCase): + def setUp(self): + self.clock = ManualClock() + self.limiter = Limiter(Policy(limit=2, window=4.0), self.clock) + + def test_clients_are_independent(self): + self.assertTrue(self.limiter.allow("a")) + self.assertTrue(self.limiter.allow("a")) + self.assertFalse(self.limiter.allow("a")) + self.assertTrue(self.limiter.allow("b")) + + def test_status_shows_remaining(self): + self.limiter.allow("a") + status = self.limiter.status("a") + self.assertEqual(status, {"remaining": 1, "limit": 2}) + + def test_guard_returns_429_when_over(self): + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (200, "hi")) + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (200, "hi")) + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (429, "slow down")) + + def test_waiting_quietly_lets_a_client_back_in(self): + self.limiter.allow("a") + self.limiter.allow("a") + self.clock.advance(2.0) + self.assertTrue(self.limiter.allow("a")) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/case.json b/evals/routing-2026-10-23/cases/hr-note-cache/case.json new file mode 100644 index 0000000..fd89507 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/case.json @@ -0,0 +1,8 @@ +{ + "name": "hr-note-cache", + "family": "hr", + "kind": "standard", + "description": "Call-graph lookup: find every function that ends up writing to the store (some indirectly, some in other modules) and add the missing cache invalidation.", + "prompt": "In the notes package (`notesvc`), reads of a note go through a cache, and some of the code that changes notes doesn't invalidate it, so people see stale data after an edit. Use subagents to find every function in the package that writes to the store (anything that ends up calling `store.save` or `store.delete`, directly or through another function) and work out which of them already drop that note's cache entry. Then fix the ones that don't: invalidate just that note's entry, and keep the cache working for everything else.", + "expected_outcome": "edit_body, archive/unarchive, delete_note, tags.add_tag and importer.import_notes (and restore, through delete_note) all invalidate only the affected note's cache entry so reads are fresh, while other cached notes and normal read caching are untouched." +} diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hr-note-cache/hidden/test_hidden.py new file mode 100644 index 0000000..a88c3e4 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/hidden/test_hidden.py @@ -0,0 +1,175 @@ +import unittest + +from notesvc import backup, importer, tags +from notesvc.cache import ReadCache +from notesvc.service import NoteService +from notesvc.store import Store + + +def make(): + store = Store() + cache = ReadCache() + return store, cache, NoteService(store, cache) + + +def prime(svc, *ids): + for note_id in ids: + svc.get_note(note_id) + + +class StaleReadsAreGone(unittest.TestCase): + def test_edit_body(self): + _, _, svc = make() + a = svc.create_note("A", "old") + prime(svc, a) + svc.edit_body(a, "new") + self.assertEqual(svc.get_note(a).body, "new") + + def test_archive_and_unarchive(self): + _, _, svc = make() + a = svc.create_note("A") + prime(svc, a) + svc.archive_note(a) + self.assertTrue(svc.get_note(a).archived) + svc.unarchive_note(a) + self.assertFalse(svc.get_note(a).archived) + + def test_delete(self): + _, _, svc = make() + a = svc.create_note("A") + prime(svc, a) + svc.delete_note(a) + self.assertIsNone(svc.get_note(a)) + + def test_add_tag(self): + _, _, svc = make() + a = svc.create_note("A") + prime(svc, a) + tags.add_tag(svc, a, "urgent") + self.assertIn("urgent", svc.get_note(a).tags) + + def test_import_replaces_an_existing_note(self): + _, _, svc = make() + a = svc.create_note("Old title", "old body") + prime(svc, a) + count = importer.import_notes(svc, [{"id": a, "title": "New title", "body": "new body"}]) + self.assertEqual(count, 1) + note = svc.get_note(a) + self.assertEqual((note.title, note.body), ("New title", "new body")) + + def test_import_adds_new_notes(self): + _, _, svc = make() + importer.import_notes(svc, [{"id": "n50", "title": "Fresh", "tags": ["t"]}]) + note = svc.get_note("n50") + self.assertEqual((note.title, note.tags), ("Fresh", ["t"])) + + def test_restore_brings_back_old_content_and_removes_extras(self): + _, _, svc = make() + a = svc.create_note("A", "original") + snap = backup.snapshot(svc) + svc.edit_body(a, "changed") + extra = svc.create_note("Extra") + prime(svc, a, extra) + backup.restore(svc, snap) + self.assertEqual(svc.get_note(a).body, "original") + self.assertIsNone(svc.get_note(extra)) + + def test_changes_made_after_the_read_are_all_seen(self): + _, _, svc = make() + a = svc.create_note("A", "b0") + prime(svc, a) + svc.edit_body(a, "b1") + self.assertEqual(svc.get_note(a).body, "b1") + svc.edit_body(a, "b2") + self.assertEqual(svc.get_note(a).body, "b2") + svc.rename_note(a, "A2") + note = svc.get_note(a) + self.assertEqual((note.title, note.body), ("A2", "b2")) + + +class CacheStillWorks(unittest.TestCase): + def test_reads_are_still_cached(self): + store, _, svc = make() + a = svc.create_note("A") + before = store.reads + svc.get_note(a) + svc.get_note(a) + svc.get_note(a) + self.assertEqual(store.reads - before, 1) + + def test_cache_is_used_again_after_a_write(self): + store, _, svc = make() + a = svc.create_note("A") + prime(svc, a) + svc.edit_body(a, "x") + before = store.reads + svc.get_note(a) + svc.get_note(a) + self.assertEqual(store.reads - before, 1) + + def test_other_notes_stay_cached_after_edit(self): + store, _, svc = make() + a = svc.create_note("A") + b = svc.create_note("B") + prime(svc, a, b) + svc.edit_body(a, "x") + before = store.reads + svc.get_note(b) + self.assertEqual(store.reads, before) + + def test_other_notes_stay_cached_after_archive(self): + store, _, svc = make() + a = svc.create_note("A") + b = svc.create_note("B") + prime(svc, a, b) + svc.archive_note(a) + before = store.reads + svc.get_note(b) + self.assertEqual(store.reads, before) + + def test_other_notes_stay_cached_after_delete(self): + store, _, svc = make() + a = svc.create_note("A") + b = svc.create_note("B") + prime(svc, a, b) + svc.delete_note(a) + before = store.reads + svc.get_note(b) + self.assertEqual(store.reads, before) + + def test_other_notes_stay_cached_after_add_tag(self): + store, _, svc = make() + a = svc.create_note("A") + b = svc.create_note("B") + prime(svc, a, b) + tags.add_tag(svc, a, "t") + before = store.reads + svc.get_note(b) + self.assertEqual(store.reads, before) + + def test_import_only_drops_the_imported_ids(self): + store, _, svc = make() + a = svc.create_note("A") + b = svc.create_note("B") + c = svc.create_note("C") + prime(svc, a, b, c) + importer.import_notes(svc, [{"id": a, "title": "A2"}]) + before = store.reads + svc.get_note(b) + svc.get_note(c) + self.assertEqual(store.reads, before) + self.assertEqual(svc.get_note(a).title, "A2") + + def test_missing_note_edit_still_raises(self): + _, _, svc = make() + with self.assertRaises(KeyError): + svc.edit_body("nope", "x") + + def test_deleting_a_missing_note_is_quiet(self): + _, _, svc = make() + svc.delete_note("nope") + self.assertIsNone(svc.get_note("nope")) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/importer.py b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/importer.py new file mode 100644 index 0000000..b8ee7a6 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/importer.py @@ -0,0 +1,21 @@ +"""Bulk import from plain dictionaries.""" + +from notesvc.note import Note + + +def import_notes(service, rows): + """Create or replace one note per row. Returns how many rows were imported. + + Each row looks like {"id": "n7", "title": "...", "body": "...", "tags": [...]}. + """ + count = 0 + for row in rows: + note = Note( + id=row["id"], + title=row["title"], + body=row.get("body", ""), + tags=list(row.get("tags", [])), + ) + service._write(note) + count += 1 + return count diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/service.py b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/service.py new file mode 100644 index 0000000..cc4952b --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/service.py @@ -0,0 +1,67 @@ +"""The note service: reads go through the cache, writes go to the store.""" + +from notesvc.note import Note + + +class NoteService: + def __init__(self, store, cache): + self.store = store + self.cache = cache + self._next = 1 + + # ---- reads ------------------------------------------------------- + + def get_note(self, note_id): + cached = self.cache.get(note_id) + if cached is not None: + return cached + note = self.store.get(note_id) + if note is not None: + self.cache.put(note_id, note) + return note + + def _load(self, note_id): + """Read straight from the store (used before changing a note).""" + note = self.store.get(note_id) + if note is None: + raise KeyError(note_id) + return note + + # ---- writes ------------------------------------------------------ + + def _write(self, note): + self.store.save(note) + self.cache.invalidate(note.id) + + def create_note(self, title, body="", tags=()): + while self.store.exists(f"n{self._next}"): + self._next += 1 + note = Note(id=f"n{self._next}", title=title, body=body, tags=list(tags)) + self._next += 1 + self._write(note) + return note.id + + def rename_note(self, note_id, title): + note = self._load(note_id) + note.title = title + self._write(note) + + def edit_body(self, note_id, body): + note = self._load(note_id) + note.body = body + self._write(note) + + def archive_note(self, note_id): + self._set_flag(note_id, "archived", True) + + def unarchive_note(self, note_id): + self._set_flag(note_id, "archived", False) + + def _set_flag(self, note_id, name, value): + note = self._load(note_id) + setattr(note, name, value) + self._write(note) + + def delete_note(self, note_id): + self.store.delete(note_id) + self.cache.invalidate(note_id) diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/tags.py b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/tags.py new file mode 100644 index 0000000..52950a3 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/reference/notesvc/tags.py @@ -0,0 +1,12 @@ +"""Tag helpers.""" + + +def add_tag(service, note_id, tag): + note = service._load(note_id) + if tag not in note.tags: + note.tags.append(tag) + service._write(note) + + +def notes_with_tag(service, tag): + return sorted(n.id for n in service.store.all() if tag in n.tags) diff --git a/evals/routing-2026-10-23/cases/hr-note-cache/scaffold.sh b/evals/routing-2026-10-23/cases/hr-note-cache/scaffold.sh new file mode 100644 index 0000000..f3f0ab5 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-note-cache/scaffold.sh @@ -0,0 +1,329 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +mkdir -p 'notesvc' +cat > 'notesvc/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""notesvc: a small note service with an in-memory store and a read cache.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/backup.py' <<'__SCAFFOLD_FILE_END__' +"""Snapshot and restore the whole note set.""" + + +def snapshot(service): + return service.store.all() + + +def restore(service, notes): + """Make the store hold exactly `notes`.""" + wanted = {n.id for n in notes} + for existing in service.store.all(): + if existing.id not in wanted: + service.delete_note(existing.id) + for note in notes: + service._write(note) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/cache.py' <<'__SCAFFOLD_FILE_END__' +"""A read cache keyed by note id.""" + +import copy + + +class ReadCache: + def __init__(self): + self._items = {} + self.hits = 0 + self.misses = 0 + + def get(self, key): + if key in self._items: + self.hits += 1 + return copy.deepcopy(self._items[key]) + self.misses += 1 + return None + + def put(self, key, value): + self._items[key] = copy.deepcopy(value) + + def invalidate(self, key): + self._items.pop(key, None) + + def clear(self): + self._items.clear() + + def __contains__(self, key): + return key in self._items +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/importer.py' <<'__SCAFFOLD_FILE_END__' +"""Bulk import from plain dictionaries.""" + +from notesvc.note import Note + + +def import_notes(service, rows): + """Create or replace one note per row. Returns how many rows were imported. + + Each row looks like {"id": "n7", "title": "...", "body": "...", "tags": [...]}. + """ + count = 0 + for row in rows: + note = Note( + id=row["id"], + title=row["title"], + body=row.get("body", ""), + tags=list(row.get("tags", [])), + ) + service.store.save(note) + count += 1 + return count +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/note.py' <<'__SCAFFOLD_FILE_END__' +"""The Note record.""" + +from dataclasses import dataclass, field + + +@dataclass +class Note: + id: str + title: str + body: str = "" + tags: list = field(default_factory=list) + archived: bool = False +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/report.py' <<'__SCAFFOLD_FILE_END__' +"""Read-only summary numbers.""" + + +def summary(service): + notes = service.store.all() + return { + "total": len(notes), + "archived": sum(1 for n in notes if n.archived), + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/search.py' <<'__SCAFFOLD_FILE_END__' +"""Read-only search helpers.""" + + +def find_by_text(service, text): + text = text.lower() + return sorted( + n.id + for n in service.store.all() + if text in n.title.lower() or text in n.body.lower() + ) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/service.py' <<'__SCAFFOLD_FILE_END__' +"""The note service: reads go through the cache, writes go to the store.""" + +from notesvc.note import Note + + +class NoteService: + def __init__(self, store, cache): + self.store = store + self.cache = cache + self._next = 1 + + # ---- reads ------------------------------------------------------- + + def get_note(self, note_id): + cached = self.cache.get(note_id) + if cached is not None: + return cached + note = self.store.get(note_id) + if note is not None: + self.cache.put(note_id, note) + return note + + def _load(self, note_id): + """Read straight from the store (used before changing a note).""" + note = self.store.get(note_id) + if note is None: + raise KeyError(note_id) + return note + + # ---- writes ------------------------------------------------------ + + def _write(self, note): + self.store.save(note) + self.cache.invalidate(note.id) + + def create_note(self, title, body="", tags=()): + while self.store.exists(f"n{self._next}"): + self._next += 1 + note = Note(id=f"n{self._next}", title=title, body=body, tags=list(tags)) + self._next += 1 + self._write(note) + return note.id + + def rename_note(self, note_id, title): + note = self._load(note_id) + note.title = title + self._write(note) + + def edit_body(self, note_id, body): + note = self._load(note_id) + note.body = body + self.store.save(note) + + def archive_note(self, note_id): + self._set_flag(note_id, "archived", True) + + def unarchive_note(self, note_id): + self._set_flag(note_id, "archived", False) + + def _set_flag(self, note_id, name, value): + note = self._load(note_id) + setattr(note, name, value) + self.store.save(note) + + def delete_note(self, note_id): + self.store.delete(note_id) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/store.py' <<'__SCAFFOLD_FILE_END__' +"""In-memory storage. Keeps its own copies so callers cannot change stored notes.""" + +import copy + + +class Store: + def __init__(self): + self._notes = {} + self.reads = 0 # number of get() calls, handy for checking cache hits + + def save(self, note): + self._notes[note.id] = copy.deepcopy(note) + + def get(self, note_id): + self.reads += 1 + note = self._notes.get(note_id) + return copy.deepcopy(note) if note is not None else None + + def exists(self, note_id): + return note_id in self._notes + + def delete(self, note_id): + self._notes.pop(note_id, None) + + def all(self): + return [copy.deepcopy(n) for n in self._notes.values()] +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/tags.py' <<'__SCAFFOLD_FILE_END__' +"""Tag helpers.""" + + +def add_tag(service, note_id, tag): + note = service._load(note_id) + if tag not in note.tags: + note.tags.append(tag) + service.store.save(note) + + +def notes_with_tag(service, tag): + return sorted(n.id for n in service.store.all() if tag in n.tags) +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for notesvc.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_service.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from notesvc import backup, importer, report, search, tags +from notesvc.cache import ReadCache +from notesvc.service import NoteService +from notesvc.store import Store + + +def make(): + store = Store() + cache = ReadCache() + return store, cache, NoteService(store, cache) + + +class Basics(unittest.TestCase): + def test_create_and_get(self): + _, _, svc = make() + note_id = svc.create_note("Groceries", "milk", tags=["home"]) + note = svc.get_note(note_id) + self.assertEqual((note.title, note.body, note.tags), ("Groceries", "milk", ["home"])) + + def test_missing_note_is_none(self): + _, _, svc = make() + self.assertIsNone(svc.get_note("nope")) + + def test_second_read_comes_from_the_cache(self): + store, _, svc = make() + note_id = svc.create_note("A") + before = store.reads + svc.get_note(note_id) + svc.get_note(note_id) + self.assertEqual(store.reads - before, 1) + + def test_rename_is_visible_after_a_read(self): + _, _, svc = make() + note_id = svc.create_note("Old") + svc.get_note(note_id) + svc.rename_note(note_id, "New") + self.assertEqual(svc.get_note(note_id).title, "New") + + def test_edit_of_a_missing_note_raises(self): + _, _, svc = make() + with self.assertRaises(KeyError): + svc.edit_body("nope", "x") + + +class Helpers(unittest.TestCase): + def test_search_and_report(self): + _, _, svc = make() + a = svc.create_note("Red apples", "fruit") + svc.create_note("Blue", "sky") + svc.archive_note(a) + self.assertEqual(search.find_by_text(svc, "apple"), [a]) + self.assertEqual(report.summary(svc), {"total": 2, "archived": 1}) + + def test_tags_and_import(self): + _, _, svc = make() + a = svc.create_note("A") + tags.add_tag(svc, a, "x") + self.assertEqual(tags.notes_with_tag(svc, "x"), [a]) + self.assertEqual(importer.import_notes(svc, [{"id": "n9", "title": "Imported"}]), 1) + self.assertEqual(svc.get_note("n9").title, "Imported") + + def test_snapshot_and_restore_round_trip(self): + _, _, svc = make() + a = svc.create_note("A", "one") + snap = backup.snapshot(svc) + extra = svc.create_note("Extra") + backup.restore(svc, snap) + self.assertIsNotNone(svc.get_note(a)) + self.assertEqual(report.summary(svc)["total"], 1) + self.assertNotEqual(extra, a) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/evals/routing-2026-10-23/cases/hr-page-size/case.json b/evals/routing-2026-10-23/cases/hr-page-size/case.json new file mode 100644 index 0000000..a040fb3 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-page-size/case.json @@ -0,0 +1,8 @@ +{ + "name": "hr-page-size", + "family": "hr", + "kind": "standard", + "description": "Where a value is defined and used: change the default page size and cap, notice the hard-coded 20 in the renderer, and leave the unrelated printed-sheet 20 alone.", + "prompt": "The product listing returns 20 products per page by default and lets callers ask for up to 50 with `page_size`. I want the default to be 50, and callers should be able to ask for up to 100 (anything bigger is cut down to 100; zero, negative or non-numeric values fall back to the default). Use subagents first to find where the page size is defined and every place that relies on it or assumes the old number, and to pick out anything that just happens to use the number 20 for something else, because that should stay as it is. Then make the change.", + "expected_outcome": "The default listing page size is 50 with a cap of 100, the 'Showing a-b of n' text follows the actual page size, and the printed-sheet export still uses 20 per sheet." +} diff --git a/evals/routing-2026-10-23/cases/hr-page-size/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hr-page-size/hidden/test_hidden.py new file mode 100644 index 0000000..d2d0abe --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-page-size/hidden/test_hidden.py @@ -0,0 +1,126 @@ +import unittest + +from shop import api, catalog, export_pdf, paging, web + + +def ids(result): + return [p["id"] for p in result["items"]] + + +class Defaults(unittest.TestCase): + def test_default_page_is_fifty(self): + result = api.list_products({}) + self.assertEqual(len(result["items"]), 50) + self.assertEqual( + result["meta"], {"page": 1, "page_size": 50, "total": 130, "pages": 3} + ) + + def test_last_page_has_the_rest(self): + result = api.list_products({"page": "3"}) + self.assertEqual(ids(result), list(range(101, 131))) + self.assertEqual(result["meta"]["page_size"], 50) + + def test_no_query_at_all(self): + self.assertEqual(len(api.list_products()["items"]), 50) + + def test_category_filter_uses_the_new_default(self): + result = api.list_products({"category": "tools"}) + self.assertEqual(result["meta"]["total"], 65) + self.assertEqual(result["meta"]["pages"], 2) + self.assertEqual(len(result["items"]), 50) + + +class RequestedSizes(unittest.TestCase): + def test_smaller_size_is_honoured(self): + result = api.list_products({"page_size": "25"}) + self.assertEqual(len(result["items"]), 25) + self.assertEqual(result["meta"]["pages"], 6) + + def test_fifty_one_is_now_allowed(self): + self.assertEqual(len(api.list_products({"page_size": "51"})["items"]), 51) + + def test_one_hundred_is_allowed(self): + result = api.list_products({"page_size": "100"}) + self.assertEqual(len(result["items"]), 100) + self.assertEqual(result["meta"]["page_size"], 100) + + def test_over_the_cap_is_clamped(self): + for raw in ("101", "500", "100000"): + result = api.list_products({"page_size": raw}) + self.assertEqual(result["meta"]["page_size"], 100, raw) + self.assertEqual(len(result["items"]), 100, raw) + + def test_bad_values_fall_back_to_the_default(self): + for raw in ("0", "-5", "abc", "", "2.5"): + result = api.list_products({"page_size": raw}) + self.assertEqual(result["meta"]["page_size"], 50, raw) + + def test_bad_page_numbers_mean_page_one(self): + for raw in ("0", "-1", "x", ""): + result = api.list_products({"page": raw}) + self.assertEqual(result["meta"]["page"], 1, raw) + self.assertEqual(ids(result)[0], 1, raw) + + def test_second_page_of_a_custom_size(self): + result = api.list_products({"page": "2", "page_size": "60"}) + self.assertEqual(ids(result), list(range(61, 121))) + + +class PagingFunctions(unittest.TestCase): + def test_paginate_default(self): + page = paging.paginate(list(range(120))) + self.assertEqual(len(page["items"]), 50) + self.assertEqual(page["pages"], 3) + + def test_paginate_cap(self): + page = paging.paginate(list(range(300)), 1, 150) + self.assertEqual(len(page["items"]), 100) + + def test_clamp_values(self): + self.assertEqual(paging.clamp_page_size(None), 50) + self.assertEqual(paging.clamp_page_size(75), 75) + self.assertEqual(paging.clamp_page_size("100"), 100) + self.assertEqual(paging.clamp_page_size(101), 100) + self.assertEqual(paging.clamp_page_size(0), 50) + + +class RenderedText(unittest.TestCase): + def test_first_page_header_and_rows(self): + text = web.page("") + lines = text.split("\n") + self.assertEqual(lines[0], "Showing 1-50 of 130") + self.assertEqual(len(lines), 51) + self.assertEqual(lines[1], "- Product 001") + self.assertEqual(lines[-1], "- Product 050") + + def test_last_page_header(self): + text = web.page("page=3") + self.assertEqual(text.split("\n")[0], "Showing 101-130 of 130") + + def test_custom_size_header(self): + text = web.page("page_size=25&page=2") + self.assertEqual(text.split("\n")[0], "Showing 26-50 of 130") + + def test_big_size_header(self): + text = web.page("page_size=500&page=2") + self.assertEqual(text.split("\n")[0], "Showing 101-130 of 130") + + def test_category_header(self): + text = web.page("category=tools") + self.assertEqual(text.split("\n")[0], "Showing 1-50 of 65") + + def test_page_past_the_end(self): + text = web.page("page=9") + self.assertEqual(text, "Showing 0-0 of 130") + + +class PrintedSheetsUnchanged(unittest.TestCase): + def test_sheets_still_hold_twenty(self): + sheets = export_pdf.paginate_for_pdf(catalog.all_products()) + self.assertEqual(len(sheets[0]), 20) + self.assertEqual(len(sheets), 7) + self.assertEqual(len(sheets[-1]), 10) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/config.py b/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/config.py new file mode 100644 index 0000000..1c843b6 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/config.py @@ -0,0 +1,8 @@ +"""Settings for the shop code.""" + +# Product listings (API and web pages) +DEFAULT_PAGE_SIZE = 50 +MAX_PAGE_SIZE = 100 + +# Rows that fit on one printed catalogue sheet +PDF_PAGE_SIZE = 20 diff --git a/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/render.py b/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/render.py new file mode 100644 index 0000000..0167110 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-page-size/reference/shop/render.py @@ -0,0 +1,15 @@ +"""Plain-text rendering of a listing result.""" + + +def render_listing(result): + meta = result["meta"] + total = meta["total"] + size = meta["page_size"] + if not result["items"]: + header = f"Showing 0-0 of {total}" + else: + start = (meta["page"] - 1) * size + 1 + end = min(meta["page"] * size, total) + header = f"Showing {start}-{end} of {total}" + lines = [header] + [f"- {p['name']}" for p in result["items"]] + return "\n".join(lines) diff --git a/evals/routing-2026-10-23/cases/hr-page-size/scaffold.sh b/evals/routing-2026-10-23/cases/hr-page-size/scaffold.sh new file mode 100644 index 0000000..692fc58 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-page-size/scaffold.sh @@ -0,0 +1,153 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# shop + +Small product catalogue. + +- `shop/config.py` settings +- `shop/catalog.py` the products +- `shop/paging.py` page arithmetic +- `shop/api.py` `list_products(query)` returns items plus paging info +- `shop/render.py` text rendering of a listing +- `shop/web.py` `page(query_string)` ties api and render together +- `shop/export_pdf.py` printed sheets +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""shop: a tiny product catalogue with paged listings.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/api.py' <<'__SCAFFOLD_FILE_END__' +"""The listing endpoint, as a plain function.""" + +from shop import catalog, paging + + +def list_products(query=None): + """`query` holds the raw (string) URL parameters: category, page, page_size.""" + query = query or {} + items = catalog.all_products() + category = query.get("category") + if category: + items = [p for p in items if p["category"] == category] + result = paging.paginate(items, query.get("page"), query.get("page_size")) + return { + "items": result["items"], + "meta": {key: result[key] for key in ("page", "page_size", "total", "pages")}, + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/catalog.py' <<'__SCAFFOLD_FILE_END__' +"""The product data (generated, 130 products).""" + +_PRODUCTS = [ + {"id": i, "name": f"Product {i:03d}", "category": "tools" if i % 2 else "garden"} + for i in range(1, 131) +] + + +def all_products(): + return [dict(p) for p in _PRODUCTS] +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/config.py' <<'__SCAFFOLD_FILE_END__' +"""Settings for the shop code.""" + +# Product listings (API and web pages) +DEFAULT_PAGE_SIZE = 20 +MAX_PAGE_SIZE = 50 + +# Rows that fit on one printed catalogue sheet +PDF_PAGE_SIZE = 20 +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/export_pdf.py' <<'__SCAFFOLD_FILE_END__' +"""Split the catalogue into printed sheets.""" + +from shop import config + + +def paginate_for_pdf(items): + size = config.PDF_PAGE_SIZE + return [items[i:i + size] for i in range(0, len(items), size)] +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/paging.py' <<'__SCAFFOLD_FILE_END__' +"""Page arithmetic shared by the listing code.""" + +from shop import config + + +def clamp_page_size(value): + """Turn a requested page size (any type) into a usable one.""" + try: + size = int(value) + except (TypeError, ValueError): + return config.DEFAULT_PAGE_SIZE + if size < 1: + return config.DEFAULT_PAGE_SIZE + return min(size, config.MAX_PAGE_SIZE) + + +def clamp_page(value): + try: + page = int(value) + except (TypeError, ValueError): + return 1 + return max(page, 1) + + +def paginate(items, page=None, page_size=None): + size = clamp_page_size(page_size) + page = clamp_page(page) + total = len(items) + start = (page - 1) * size + return { + "items": items[start:start + size], + "page": page, + "page_size": size, + "total": total, + "pages": -(-total // size), + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/render.py' <<'__SCAFFOLD_FILE_END__' +"""Plain-text rendering of a listing result.""" + + +def render_listing(result): + meta = result["meta"] + total = meta["total"] + if not result["items"]: + header = f"Showing 0-0 of {total}" + else: + start = (meta["page"] - 1) * 20 + 1 + end = min(meta["page"] * 20, total) + header = f"Showing {start}-{end} of {total}" + lines = [header] + [f"- {p['name']}" for p in result["items"]] + return "\n".join(lines) +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/web.py' <<'__SCAFFOLD_FILE_END__' +"""The web page: query string in, text out.""" + +from urllib.parse import parse_qsl + +from shop import api, render + + +def page(query_string=""): + query = dict(parse_qsl(query_string)) + return render.render_listing(api.list_products(query)) +__SCAFFOLD_FILE_END__ diff --git a/evals/routing-2026-10-23/cases/hr-reminder-every/case.json b/evals/routing-2026-10-23/cases/hr-reminder-every/case.json new file mode 100644 index 0000000..898b54d --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-reminder-every/case.json @@ -0,0 +1,8 @@ +{ + "name": "hr-reminder-every", + "family": "hr", + "kind": "standard", + "description": "Which module handles each reminder kind: the recurring handler has its own broken unit table and rejects compound intervals; visible tests fail until fixed.", + "prompt": "Reminders written like `stretch every 2h` go off every two minutes instead of every two hours. Please use subagents to find which module handles each kind of reminder line (`in`, `at` and `every`) and where each one turns the time text into seconds, then fix the `every` case. There are tests in tests/; run them when you're done.", + "expected_outcome": "Every-reminders convert hours and days correctly and also accept compound intervals such as 1h30m (for example by reusing the shared duration parser), zero or malformed intervals still raise ValueError, in/at reminders are unchanged, and the tests in tests/ pass." +} diff --git a/evals/routing-2026-10-23/cases/hr-reminder-every/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hr-reminder-every/hidden/test_hidden.py new file mode 100644 index 0000000..2c2ca9e --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-reminder-every/hidden/test_hidden.py @@ -0,0 +1,114 @@ +import contextlib +import io +import unittest + +from remind import cli +from remind.parser import parse_line + + +class EveryIntervals(unittest.TestCase): + def test_hours(self): + self.assertEqual(parse_line("stretch every 2h").seconds, 7200) + + def test_one_hour(self): + self.assertEqual(parse_line("drink water every 1h").seconds, 3600) + + def test_days(self): + self.assertEqual(parse_line("backup every 1d").seconds, 86400) + + def test_minutes(self): + self.assertEqual(parse_line("check mail every 10m").seconds, 600) + + def test_many_minutes(self): + self.assertEqual(parse_line("walk every 90m").seconds, 5400) + + def test_seconds(self): + self.assertEqual(parse_line("ping every 45s").seconds, 45) + + def test_hours_and_minutes(self): + self.assertEqual(parse_line("water plants every 1h30m").seconds, 5400) + + def test_days_and_hours(self): + self.assertEqual(parse_line("rotate logs every 2d12h").seconds, 2 * 86400 + 12 * 3600) + + def test_three_parts(self): + self.assertEqual(parse_line("big job every 1d2h3m").seconds, 86400 + 7200 + 180) + + def test_kind_and_text(self): + reminder = parse_line("stretch every 2h") + self.assertEqual(reminder.kind, "every") + self.assertEqual(reminder.text, "stretch") + + def test_text_may_contain_the_word_every(self): + reminder = parse_line("check every door every 3h") + self.assertEqual(reminder.text, "check every door") + self.assertEqual(reminder.seconds, 3 * 3600) + + +class EveryRejects(unittest.TestCase): + def test_zero_seconds(self): + with self.assertRaises(ValueError): + parse_line("nothing every 0s") + + def test_zero_compound(self): + with self.assertRaises(ValueError): + parse_line("nothing every 0h0m") + + def test_no_unit(self): + with self.assertRaises(ValueError): + parse_line("nothing every 5") + + def test_garbage(self): + with self.assertRaises(ValueError): + parse_line("nothing every banana") + + def test_trailing_junk(self): + with self.assertRaises(ValueError): + parse_line("nothing every 1h30") + + +class OtherKindsUnchanged(unittest.TestCase): + def test_in_hours(self): + self.assertEqual(parse_line("laundry in 2h").seconds, 7200) + + def test_in_compound(self): + self.assertEqual(parse_line("standup in 1h30m").seconds, 5400) + + def test_in_zero_rejected(self): + with self.assertRaises(ValueError): + parse_line("nothing in 0m") + + def test_at_same_day(self): + self.assertEqual(parse_line("coffee at 09:15", now=8 * 3600).seconds, 4500) + + def test_at_next_day(self): + self.assertEqual(parse_line("coffee at 07:00", now=8 * 3600).seconds, 23 * 3600) + + def test_at_bad_time(self): + with self.assertRaises(ValueError): + parse_line("coffee at 25:00") + + def test_unparseable_line(self): + with self.assertRaises(ValueError): + parse_line("just some words") + + +class CliOutput(unittest.TestCase): + def test_describe_every(self): + self.assertEqual( + cli.describe(parse_line("stretch every 2h")), "stretch: every 7200s" + ) + + def test_main_prints_one_line_each(self): + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = cli.main(["stretch every 1h30m", "call mom in 10m"]) + self.assertEqual(code, 0) + self.assertEqual( + out.getvalue().splitlines(), + ["stretch: every 5400s", "call mom: fires in 600s"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hr-reminder-every/reference/remind/recurring.py b/evals/routing-2026-10-23/cases/hr-reminder-every/reference/remind/recurring.py new file mode 100644 index 0000000..9b9d8ff --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-reminder-every/reference/remind/recurring.py @@ -0,0 +1,11 @@ +"""Handles lines of the form ' every '.""" + +from remind.durations import parse_duration + + +def parse(rest): + """`rest` is the text after the word 'every'. Returns the interval in seconds.""" + seconds = parse_duration(rest) + if seconds <= 0: + raise ValueError("interval must be positive") + return seconds diff --git a/evals/routing-2026-10-23/cases/hr-reminder-every/scaffold.sh b/evals/routing-2026-10-23/cases/hr-reminder-every/scaffold.sh new file mode 100644 index 0000000..d273d7d --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-reminder-every/scaffold.sh @@ -0,0 +1,207 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +mkdir -p 'remind' +cat > 'remind/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""remind: parse one-line reminders such as 'stretch every 2h'.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/absolute.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' at HH:MM' (24 hour clock).""" + +DAY = 24 * 3600 + + +def parse(rest, now=0): + """`rest` is 'HH:MM'. `now` is seconds since midnight. + + Returns the number of seconds until the next occurrence of that time. + """ + hours, sep, minutes = rest.partition(":") + if not sep or not hours.isdigit() or not minutes.isdigit(): + raise ValueError(f"bad time of day: {rest!r}") + hh, mm = int(hours), int(minutes) + if hh > 23 or mm > 59: + raise ValueError(f"bad time of day: {rest!r}") + target = hh * 3600 + mm * 60 + return (target - now) % DAY +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/cli.py' <<'__SCAFFOLD_FILE_END__' +"""Tiny command line front end: prints one line per reminder.""" + +from remind.parser import parse_line + + +def describe(reminder): + if reminder.kind == "every": + return f"{reminder.text}: every {reminder.seconds}s" + return f"{reminder.text}: fires in {reminder.seconds}s" + + +def main(lines, now=0): + for line in lines: + print(describe(parse_line(line, now))) + return 0 +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/durations.py' <<'__SCAFFOLD_FILE_END__' +"""Shared duration parsing: '90s', '2h', '1h30m' -> whole seconds.""" + +import re + +UNIT_SECONDS = {"s": 1, "m": 60, "h": 3600, "d": 86400} +_CHUNK = re.compile(r"(\d+)([smhd])") + + +def parse_duration(text): + text = text.strip().lower() + if not text: + raise ValueError("empty duration") + total = 0 + pos = 0 + for match in _CHUNK.finditer(text): + if match.start() != pos: + raise ValueError(f"bad duration: {text!r}") + total += int(match.group(1)) * UNIT_SECONDS[match.group(2)] + pos = match.end() + if pos != len(text): + raise ValueError(f"bad duration: {text!r}") + return total +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/parser.py' <<'__SCAFFOLD_FILE_END__' +"""Split a reminder line and hand the time part to the right module.""" + +import re + +from remind import absolute, recurring, relative +from remind.reminder import Reminder + +_LINE = re.compile(r"^(?P.+?)\s+(?Pin|at|every)\s+(?P\S+)$") + + +def parse_line(line, now=0): + """Parse ' in|at|every ' into a Reminder. + + `now` (seconds since midnight) is only used for 'at' reminders. + """ + match = _LINE.match(line.strip()) + if not match: + raise ValueError(f"cannot parse reminder: {line!r}") + kind = match.group("kind") + rest = match.group("rest") + if kind == "in": + seconds = relative.parse(rest) + elif kind == "at": + seconds = absolute.parse(rest, now) + else: + seconds = recurring.parse(rest) + return Reminder(match.group("text"), kind, seconds) +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/recurring.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' every '.""" + +import re + +_UNITS = {"s": 1, "m": 60, "h": 60, "d": 24 * 60} +_SINGLE = re.compile(r"(\d+)([smhd])") + + +def _to_seconds(text): + match = _SINGLE.fullmatch(text.strip().lower()) + if not match: + raise ValueError(f"bad interval: {text!r}") + return int(match.group(1)) * _UNITS[match.group(2)] + + +def parse(rest): + """`rest` is the text after the word 'every'. Returns the interval in seconds.""" + seconds = _to_seconds(rest) + if seconds <= 0: + raise ValueError("interval must be positive") + return seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/relative.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' in '.""" + +from remind.durations import parse_duration + + +def parse(rest): + """`rest` is the text after the word 'in'. Returns the delay in seconds.""" + seconds = parse_duration(rest) + if seconds <= 0: + raise ValueError("delay must be positive") + return seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/reminder.py' <<'__SCAFFOLD_FILE_END__' +"""The parsed form of a reminder line.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Reminder: + text: str + kind: str # "in", "at" or "every" + seconds: int # delay (in/at) or interval (every) +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for remind.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_parser.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from remind.parser import parse_line + + +class RelativeAndAbsolute(unittest.TestCase): + def test_in_minutes(self): + self.assertEqual(parse_line("call mom in 10m").seconds, 600) + + def test_in_compound(self): + self.assertEqual(parse_line("standup in 1h30m").seconds, 5400) + + def test_at_later_today(self): + self.assertEqual(parse_line("lunch at 12:00", now=0).seconds, 12 * 3600) + + def test_at_wraps_to_tomorrow(self): + self.assertEqual(parse_line("lunch at 12:00", now=13 * 3600).seconds, 23 * 3600) + + +class Every(unittest.TestCase): + def test_every_seconds(self): + reminder = parse_line("ping every 45s") + self.assertEqual(reminder.kind, "every") + self.assertEqual(reminder.seconds, 45) + + def test_every_hours(self): + self.assertEqual(parse_line("stretch every 2h").seconds, 7200) + + def test_every_compound(self): + self.assertEqual(parse_line("water plants every 1h30m").seconds, 5400) + + def test_every_zero_is_rejected(self): + with self.assertRaises(ValueError): + parse_line("nothing every 0s") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/case.json b/evals/routing-2026-10-23/cases/hr-retry-limit/case.json new file mode 100644 index 0000000..1f18751 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/case.json @@ -0,0 +1,8 @@ +{ + "name": "hr-retry-limit", + "family": "hr", + "kind": "standard", + "description": "Setting trace: a retry count of 0 is dropped in three different files of a small fetch client.", + "prompt": "I can't turn retries off in our little fetch client (the fetchkit package). `--retries 0` on the command line, or `max_retries` set to 0 in code, still ends up making four attempts when the server keeps failing; I want 0 to mean one attempt and then give up. Please use subagents to trace where the retry count is read and set across the package, starting from the CLI flag and from the Client, and report every place that touches it. Then fix it so 0 really means no retries, without changing what happens when no retry count is given.", + "expected_outcome": "A retry count of 0 (from the --retries flag, from settings overrides, from the FETCHKIT_MAX_RETRIES environment variable, or passed to the client directly) results in exactly one attempt, while the default of three retries and other explicit counts behave as before." +} diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/hidden/test_hidden.py b/evals/routing-2026-10-23/cases/hr-retry-limit/hidden/test_hidden.py new file mode 100644 index 0000000..735c7a2 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/hidden/test_hidden.py @@ -0,0 +1,127 @@ +import contextlib +import io +import unittest + +from fetchkit import cli +from fetchkit.client import Client +from fetchkit.errors import RetriesExhausted +from fetchkit.settings import load_settings +from fetchkit.transport import FlakyTransport + + +def nosleep(seconds): + pass + + +class ZeroRetries(unittest.TestCase): + def test_settings_keep_zero(self): + self.assertEqual(load_settings({"max_retries": 0})["max_retries"], 0) + + def test_env_zero_reaches_settings(self): + settings = load_settings(env={"FETCHKIT_MAX_RETRIES": "0"}) + self.assertEqual(settings["max_retries"], 0) + + def test_client_zero_means_one_attempt(self): + transport = FlakyTransport(fail_times=10) + client = Client(transport, load_settings({"max_retries": 0}), sleep=nosleep) + with self.assertRaises(RetriesExhausted) as ctx: + client.fetch("http://x/a") + self.assertEqual(transport.calls, 1) + self.assertEqual(ctx.exception.attempts, 1) + + def test_client_zero_from_env(self): + transport = FlakyTransport(fail_times=10) + settings = load_settings(env={"FETCHKIT_MAX_RETRIES": "0"}) + client = Client(transport, settings, sleep=nosleep) + with self.assertRaises(RetriesExhausted): + client.fetch("http://x/a") + self.assertEqual(transport.calls, 1) + + def test_client_given_a_plain_dict_with_zero(self): + transport = FlakyTransport(fail_times=10) + settings = dict(load_settings()) + settings["max_retries"] = 0 + client = Client(transport, settings, sleep=nosleep) + with self.assertRaises(RetriesExhausted): + client.fetch("http://x/a") + self.assertEqual(transport.calls, 1) + + def test_cli_retries_zero(self): + transport = FlakyTransport(fail_times=10) + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = cli.run(["--retries", "0", "http://x/a"], transport) + self.assertEqual(code, 1) + self.assertEqual(transport.calls, 1) + + def test_cli_retries_zero_beats_env(self): + transport = FlakyTransport(fail_times=10) + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = cli.run( + ["--retries", "0", "http://x/a"], + transport, + env={"FETCHKIT_MAX_RETRIES": "5"}, + ) + self.assertEqual(code, 1) + self.assertEqual(transport.calls, 1) + + def test_cli_zero_succeeds_when_first_attempt_works(self): + transport = FlakyTransport(fail_times=0) + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = cli.run(["--retries", "0", "http://x/a"], transport) + self.assertEqual(code, 0) + self.assertIn("ok:http://x/a", out.getvalue()) + + +class OtherCountsStillWork(unittest.TestCase): + def test_default_is_three_retries(self): + transport = FlakyTransport(fail_times=10) + client = Client(transport, sleep=nosleep) + with self.assertRaises(RetriesExhausted): + client.fetch("http://x/a") + self.assertEqual(transport.calls, 4) + + def test_default_recovers_after_two_failures(self): + transport = FlakyTransport(fail_times=2) + client = Client(transport, sleep=nosleep) + self.assertEqual(client.fetch("http://x/a"), "ok:http://x/a") + self.assertEqual(transport.calls, 3) + + def test_two_retries_means_three_attempts(self): + transport = FlakyTransport(fail_times=10) + client = Client(transport, load_settings({"max_retries": 2}), sleep=nosleep) + with self.assertRaises(RetriesExhausted): + client.fetch("http://x/a") + self.assertEqual(transport.calls, 3) + + def test_cli_without_flag_uses_default(self): + transport = FlakyTransport(fail_times=10) + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = cli.run(["http://x/a"], transport) + self.assertEqual(code, 1) + self.assertEqual(transport.calls, 4) + + def test_cli_flag_overrides_env(self): + transport = FlakyTransport(fail_times=10) + out = io.StringIO() + with contextlib.redirect_stdout(out): + cli.run( + ["--retries", "1", "http://x/a"], + transport, + env={"FETCHKIT_MAX_RETRIES": "5"}, + ) + self.assertEqual(transport.calls, 2) + + def test_env_used_when_no_flag(self): + transport = FlakyTransport(fail_times=10) + out = io.StringIO() + with contextlib.redirect_stdout(out): + cli.run(["http://x/a"], transport, env={"FETCHKIT_MAX_RETRIES": "1"}) + self.assertEqual(transport.calls, 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/cli.py b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/cli.py new file mode 100644 index 0000000..b28a60c --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/cli.py @@ -0,0 +1,35 @@ +"""Command line entry point: `fetchkit URL [--retries N] [--timeout S]`.""" + +import argparse + +from fetchkit.client import Client +from fetchkit.errors import RetriesExhausted +from fetchkit.settings import load_settings + + +def parse_args(argv): + parser = argparse.ArgumentParser(prog="fetchkit") + parser.add_argument("url") + parser.add_argument( + "--retries", type=int, default=None, help="extra attempts after the first one" + ) + parser.add_argument("--timeout", type=float, default=None) + return parser.parse_args(argv) + + +def run(argv, transport, env=None, sleep=lambda seconds: None): + """Run one fetch. Returns the process exit code.""" + args = parse_args(argv) + overrides = {} + if args.retries is not None: + overrides["max_retries"] = args.retries + if args.timeout is not None: + overrides["timeout"] = args.timeout + settings = load_settings(overrides, env) + client = Client(transport, settings, sleep=sleep) + try: + print(client.fetch(args.url)) + except RetriesExhausted as exc: + print(f"error: {exc}") + return 1 + return 0 diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/client.py b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/client.py new file mode 100644 index 0000000..04f486d --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/client.py @@ -0,0 +1,28 @@ +"""The client: send a request, retry on transport errors.""" + +import time + +from fetchkit.defaults import DEFAULTS +from fetchkit.errors import RetriesExhausted, TransportError +from fetchkit.settings import load_settings + + +class Client: + def __init__(self, transport, settings=None, sleep=time.sleep): + self.transport = transport + self.settings = settings if settings is not None else load_settings() + self.sleep = sleep + # number of extra attempts allowed after the first one + retries = self.settings.get("max_retries") + self.max_retries = DEFAULTS["max_retries"] if retries is None else retries + + def fetch(self, url): + attempts = 0 + while True: + attempts += 1 + try: + return self.transport.send(url, timeout=self.settings["timeout"]) + except TransportError as exc: + if attempts > self.max_retries: + raise RetriesExhausted(url, attempts) from exc + self.sleep(self.settings["backoff"] * attempts) diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/settings.py b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/settings.py new file mode 100644 index 0000000..cce1542 --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/reference/fetchkit/settings.py @@ -0,0 +1,35 @@ +"""Build the effective settings from defaults, environment and explicit overrides.""" + +from fetchkit.defaults import DEFAULTS + +ENV_PREFIX = "FETCHKIT_" + + +def _from_env(env): + found = {} + if not env: + return found + raw = env.get(ENV_PREFIX + "MAX_RETRIES") + if raw not in (None, ""): + found["max_retries"] = int(raw) + raw = env.get(ENV_PREFIX + "TIMEOUT") + if raw not in (None, ""): + found["timeout"] = float(raw) + raw = env.get(ENV_PREFIX + "BACKOFF") + if raw not in (None, ""): + found["backoff"] = float(raw) + return found + + +def load_settings(overrides=None, env=None): + """Return the effective settings. + + Explicit overrides win over the environment, which wins over the defaults. + """ + merged = dict(DEFAULTS) + merged.update(_from_env(env)) + if overrides: + for key in ("timeout", "backoff", "max_retries"): + if overrides.get(key) is not None: + merged[key] = overrides[key] + return merged diff --git a/evals/routing-2026-10-23/cases/hr-retry-limit/scaffold.sh b/evals/routing-2026-10-23/cases/hr-retry-limit/scaffold.sh new file mode 100644 index 0000000..db8c44a --- /dev/null +++ b/evals/routing-2026-10-23/cases/hr-retry-limit/scaffold.sh @@ -0,0 +1,181 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# fetchkit + +A tiny fetch client with retries. The transport is a stand-in that fails a set +number of times, so everything can be exercised without a network. + +- `fetchkit/defaults.py` built-in defaults +- `fetchkit/settings.py` merges defaults, environment (`FETCHKIT_*`) and overrides +- `fetchkit/client.py` `Client.fetch(url)` with retry and backoff +- `fetchkit/transport.py` `FlakyTransport` +- `fetchkit/cli.py` `run(argv, transport)` +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""fetchkit: a tiny fetch client with retries, driven by a fake transport.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/cli.py' <<'__SCAFFOLD_FILE_END__' +"""Command line entry point: `fetchkit URL [--retries N] [--timeout S]`.""" + +import argparse + +from fetchkit.client import Client +from fetchkit.errors import RetriesExhausted +from fetchkit.settings import load_settings + + +def parse_args(argv): + parser = argparse.ArgumentParser(prog="fetchkit") + parser.add_argument("url") + parser.add_argument( + "--retries", type=int, default=None, help="extra attempts after the first one" + ) + parser.add_argument("--timeout", type=float, default=None) + return parser.parse_args(argv) + + +def run(argv, transport, env=None, sleep=lambda seconds: None): + """Run one fetch. Returns the process exit code.""" + args = parse_args(argv) + overrides = {} + if args.retries: + overrides["max_retries"] = args.retries + if args.timeout is not None: + overrides["timeout"] = args.timeout + settings = load_settings(overrides, env) + client = Client(transport, settings, sleep=sleep) + try: + print(client.fetch(args.url)) + except RetriesExhausted as exc: + print(f"error: {exc}") + return 1 + return 0 +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/client.py' <<'__SCAFFOLD_FILE_END__' +"""The client: send a request, retry on transport errors.""" + +import time + +from fetchkit.defaults import DEFAULTS +from fetchkit.errors import RetriesExhausted, TransportError +from fetchkit.settings import load_settings + + +class Client: + def __init__(self, transport, settings=None, sleep=time.sleep): + self.transport = transport + self.settings = settings if settings is not None else load_settings() + self.sleep = sleep + # number of extra attempts allowed after the first one + self.max_retries = self.settings.get("max_retries") or DEFAULTS["max_retries"] + + def fetch(self, url): + attempts = 0 + while True: + attempts += 1 + try: + return self.transport.send(url, timeout=self.settings["timeout"]) + except TransportError as exc: + if attempts > self.max_retries: + raise RetriesExhausted(url, attempts) from exc + self.sleep(self.settings["backoff"] * attempts) +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/defaults.py' <<'__SCAFFOLD_FILE_END__' +"""Built-in defaults for the fetch client.""" + +DEFAULTS = { + "timeout": 5.0, + "max_retries": 3, + "backoff": 0.5, +} +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/errors.py' <<'__SCAFFOLD_FILE_END__' +"""Exceptions raised by fetchkit.""" + + +class TransportError(Exception): + """One attempt to reach the server failed.""" + + +class RetriesExhausted(Exception): + """Every allowed attempt failed.""" + + def __init__(self, url, attempts): + super().__init__(f"gave up on {url} after {attempts} attempt(s)") + self.url = url + self.attempts = attempts +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/settings.py' <<'__SCAFFOLD_FILE_END__' +"""Build the effective settings from defaults, environment and explicit overrides.""" + +from fetchkit.defaults import DEFAULTS + +ENV_PREFIX = "FETCHKIT_" + + +def _from_env(env): + found = {} + if not env: + return found + raw = env.get(ENV_PREFIX + "MAX_RETRIES") + if raw not in (None, ""): + found["max_retries"] = int(raw) + raw = env.get(ENV_PREFIX + "TIMEOUT") + if raw not in (None, ""): + found["timeout"] = float(raw) + raw = env.get(ENV_PREFIX + "BACKOFF") + if raw not in (None, ""): + found["backoff"] = float(raw) + return found + + +def load_settings(overrides=None, env=None): + """Return the effective settings. + + Explicit overrides win over the environment, which wins over the defaults. + """ + merged = dict(DEFAULTS) + merged.update(_from_env(env)) + if overrides: + for key in ("timeout", "backoff"): + if overrides.get(key) is not None: + merged[key] = overrides[key] + merged["max_retries"] = overrides.get("max_retries") or merged["max_retries"] + return merged +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/transport.py' <<'__SCAFFOLD_FILE_END__' +"""A fake transport used by the tests and the demo CLI.""" + +from fetchkit.errors import TransportError + + +class FlakyTransport: + """Fails the first `fail_times` calls, then succeeds.""" + + def __init__(self, fail_times=0): + self.fail_times = fail_times + self.calls = 0 + + def send(self, url, timeout=None): + self.calls += 1 + if self.calls <= self.fail_times: + raise TransportError(f"simulated failure {self.calls}") + return "ok:" + url +__SCAFFOLD_FILE_END__ diff --git a/evals/routing-2026-10-23/grade_hidden.py b/evals/routing-2026-10-23/grade_hidden.py new file mode 100644 index 0000000..bd76395 --- /dev/null +++ b/evals/routing-2026-10-23/grade_hidden.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Hidden acceptance tests for the routing eval (PREREG-2026-10-23-routing.md). + +The agent never sees these tests: they live outside the plugin directories and are not scaffolded. +After a run, the work directory the eval kept is copied to a temporary directory, the case's +hidden/test_hidden.py is copied into its root, and `python -m unittest test_hidden` runs there. + + grade_hidden.py grade + PASS (exit 0), FAIL (exit 1), or COULD NOT RUN (exit 2). A run over 120 s is a FAIL. + grade_hidden.py selfcheck + Known-bad and known-good check: the scaffold as written must FAIL and the scaffold with + reference/ laid over it must PASS. Exit 0 when both hold, 1 when either doesn't. + + holds hidden/test_hidden.py and reference/. Standard library only. +""" +import os +import shutil +import subprocess +import sys +import tempfile + +TIMEOUT_S = 120 +IGNORE = shutil.ignore_patterns(".git", "__pycache__", "test_hidden.py") + + +def run_hidden(case_dir, work_dir): + """Return (verdict, detail): verdict is 'PASS', 'FAIL' or 'COULD NOT RUN'.""" + test = os.path.join(case_dir, "hidden", "test_hidden.py") + if not os.path.isfile(test): + return "COULD NOT RUN", "no hidden/test_hidden.py in " + case_dir + if not os.path.isdir(work_dir): + return "COULD NOT RUN", "no work directory at " + work_dir + with tempfile.TemporaryDirectory(prefix="routing-hidden-", ignore_cleanup_errors=True) as tmp: + root = os.path.join(tmp, "w") + shutil.copytree(work_dir, root, ignore=IGNORE, symlinks=False) + shutil.copyfile(test, os.path.join(root, "test_hidden.py")) + # -E and -s keep the operator's PYTHON* variables and user site out; the work directory + # itself must stay importable, so -I (which drops it from sys.path) is not used. + cmd = [sys.executable, "-E", "-s", "-m", "unittest", "test_hidden", "-q"] + try: + p = subprocess.run(cmd, cwd=root, capture_output=True, text=True, timeout=TIMEOUT_S) + except subprocess.TimeoutExpired: + return "FAIL", "timed out after %d s" % TIMEOUT_S + tail = (p.stdout + p.stderr).strip().splitlines()[-3:] + return ("PASS" if p.returncode == 0 else "FAIL"), " | ".join(tail) + + +def scaffold_into(scaffold, dest): + os.makedirs(dest) + bash = shutil.which("bash") or "bash" + p = subprocess.run([bash, os.path.abspath(scaffold)], cwd=dest, capture_output=True, text=True, timeout=120) + if p.returncode != 0: + raise RuntimeError("scaffold exited %d: %s" % (p.returncode, (p.stderr or p.stdout).strip()[-300:])) + + +def selfcheck(case_dir, scaffold): + ok = True + with tempfile.TemporaryDirectory(prefix="routing-selfcheck-", ignore_cleanup_errors=True) as tmp: + bad = os.path.join(tmp, "bad") + scaffold_into(scaffold, bad) + v, d = run_hidden(case_dir, bad) + print("known-bad (scaffold as written): %s %s" % (v, d)) + ok = ok and v == "FAIL" + good = os.path.join(tmp, "good") + scaffold_into(scaffold, good) + ref = os.path.join(case_dir, "reference") + if not os.path.isdir(ref): + print("known-good: COULD NOT RUN no reference/ directory") + return False + shutil.copytree(ref, good, dirs_exist_ok=True) + v, d = run_hidden(case_dir, good) + print("known-good (reference laid over): %s %s" % (v, d)) + ok = ok and v == "PASS" + return ok + + +def main(argv): + if len(argv) != 4 or argv[1] not in ("grade", "selfcheck"): + print(__doc__.strip(), file=sys.stderr) + return 2 + if argv[1] == "grade": + v, d = run_hidden(argv[2], argv[3]) + print("%s %s" % (v, d)) + return {"PASS": 0, "FAIL": 1}.get(v, 2) + try: + return 0 if selfcheck(argv[2], argv[3]) else 1 + except Exception as e: # a scaffold that won't run is a broken case, not a failing one + print("COULD NOT RUN %s" % e) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/routing-2026-10-23/held-out-brief.md b/evals/routing-2026-10-23/held-out-brief.md new file mode 100644 index 0000000..b93a25b --- /dev/null +++ b/evals/routing-2026-10-23/held-out-brief.md @@ -0,0 +1,93 @@ +# Brief for the held-out task author + +You are writing small, self-contained coding tasks for an evaluation of coding agents (Claude Code). You did not +build the software being evaluated and you must not look at it: write from this brief alone. Do not read any +repository, plugin, eval file or anything outside the output directory given in your instructions. + +There are two families of tasks. Seven tasks in all. + +## Family 1: five tasks named `hr-*` (one agent that hands work to helpers) + +Each task sets up a tiny Python project and gives the agent a prompt in a user's voice. The prompt asks the agent +to hand part of the work to subagents (helper agents it can start, each with its own instructions) and then to +finish the job itself. Phrase it naturally, for example "use subagents to find ... and ..., then fix it" or "have a +helper agent look through ... first". The agent decides how to brief the helpers. + +- **Four standard tasks.** The part handed to helpers is reading and finding: where a value is defined, which + functions call another, how a setting flows through several files, which of several modules handles a case. The + project should have roughly 5 to 10 small files so the lookup is real. After the helpers report, the agent makes + a small code change (a bug fix or a small feature). Vary them: some have a visible test file the agent can run, + some don't; at least one where the obvious fix is wrong and only careful reading or running the tests shows it. +- **One task where careful reasoning matters.** The part handed to a helper needs judgment, not lookup: for + example, a helper reviews one module and finds the cause of a described wrong result. Make the cause subtle, so a + quick read misses it (an off-by-one in a rare branch, an ordering assumption, a rounding or boundary case, state + shared between calls). The visible tests, if any, must not catch it. Your hidden tests must. + +## Family 2: two tasks named `hf-*` (a parallel build tool) + +These tasks are run through a build tool that takes a goal, a list of tracks and a list of gate commands. It +starts one builder per track at the same time, each builder writes only the files its track owns, and at the end +the gate commands must pass. The builders cannot talk to each other. So: + +- Two tracks per task. Each track has a `name` (short, lowercase), a `prompt` (everything a builder who has read + nothing else needs: the files to write, function names, signatures and data shapes it must use), and `files` + (repository-relative paths only that track writes; no globs; the two tracks' files must not overlap). +- Any name or shape one track uses from the other must be stated in both prompts, so neither builder has to guess. +- `gates`: one to three shell commands that must pass at the end (for example `python3 -m unittest discover -s tests -q`). + The scaffold may include visible tests for the gates, or a track may own the test file. +- **One standard task** of modest size. +- **One task where careful reasoning matters** in one track: an algorithm or rule with an edge case that a quick + implementation gets wrong (the visible gate tests may pass while your hidden tests catch it). + +The scaffold for family 2 must also make the directory a git repository with one commit: +`git init -q && git add -A && git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold`. + +## Rules for every task + +- Python 3 standard library only. No network, no package installs, nothing outside the project directory. +- The prompt (and for family 2 the goal and both track prompts) stands alone: no absolute paths, no home-directory + paths, nothing outside the scaffolded files. +- Do not use any of these words, in any form, in a prompt, goal or track prompt: model, haiku, sonnet, opus, cheap, + cost, expensive, route, routing, tier, dojo, verify, protocol, claim, evidence. +- Keep each family 1 task small enough for an agent to finish in under 16 turns, including the helpers' work. +- Write `expected_outcome` as one sentence describing a correct finish. + +## What to write for each task + +A folder `//` (lowercase letters, digits and hyphens; family 1 names start with `hr-`, family 2 +with `hf-`) containing: + +1. `scaffold.sh`: a POSIX shell script that writes the project's files into the current directory (`printf` or + here-documents; `set -e` at the top). It must not write the hidden tests or the reference solution. +2. `case.json`: + ```json + { + "name": "hr-example", + "family": "hr", + "kind": "standard", + "description": "One sentence.", + "prompt": "...", + "expected_outcome": "..." + } + ``` + `kind` is `standard` or `careful`. Family 2 tasks also carry `"goal": "..."`, `"tracks": [{"name": "...", + "prompt": "...", "files": ["..."]}, ...]` and `"gates": ["..."]`; their `prompt` field is left empty (the + evaluator writes the instruction that starts the build tool). +3. `hidden/test_hidden.py`: `unittest` acceptance tests for a correct finish, run from the project root after the + agent is done (`python3 -m unittest test_hidden` with the file copied into the root). Standard library only. + Test behaviour through the project's own modules and functions, not file text. They must fail on the scaffold as + written and pass on your reference solution. +4. `reference/`: the files of one correct finish, at their project-relative paths (only the files that change or + are added). + +## Check your work, and report it + +For every task, in a fresh empty temporary directory: + +1. run `scaffold.sh` and list the files it wrote; +2. copy `hidden/test_hidden.py` into the directory and run `python3 -m unittest test_hidden -q`: it must fail; +3. copy `reference/` over the directory and run the hidden tests again: they must pass; for family 2, also run every + gate command and confirm it passes on the reference; +4. for any task with visible tests, run them on the scaffold and say whether they pass or fail before a fix. + +Report each command you ran and its exit code. Report anything you could not make work. diff --git a/evals/routing-2026-10-23/judge_files.py b/evals/routing-2026-10-23/judge_files.py new file mode 100644 index 0000000..4961959 --- /dev/null +++ b/evals/routing-2026-10-23/judge_files.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""The `quality-judge` reference, run after the eval (PREREG-2026-10-23-routing.md, amendment 1). + +On this Windows machine the eval's `focus: files` view reaches the judge empty ("(no file changes)") even when the +agent changed files, so the in-eval judge can't see anything. This runs the same grader outside the eval: the same +criteria string (read from the case.yaml), the same judge (Opus), three votes with the majority deciding, and the +same view, files only and never the trace. The files are the run's changes against a fresh scaffold: every added or +modified text file in full, plus the names of deleted ones. + + judge_files.py + Prints one JSON line: {"verdict": "PASS"|"FAIL"|"COULD NOT RUN", "votes": [...], "cost_usd": ...}. + +Each vote is `claude -p --model opus` on the logged-in account (the session's claude.ai login; this script +never reads or sets an API key) with no tools, no setting sources, no MCP servers and no slash commands, so no +user hook, plugin or memory reaches the judge. +""" +import filecmp +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile + +VOTES = 3 +SKIP_DIRS = {".git", "__pycache__"} +MAX_FILE = 20000 +MAX_TOTAL = 80000 +SYSTEM = ( + "You are an evaluation judge. You are shown the files a coding agent added or changed in a small project, " + "and a grading criterion. Apply the criterion to the files only. End your reply with one line that is exactly " + "VERDICT: PASS or VERDICT: FAIL." +) + + +def criteria_of(case_yaml): + text = open(case_yaml, encoding="utf-8").read() + body = text[text.index("{"):] # the case files are JSON after a comment header + case = json.loads(body) + return [g for g in case["graders"] if g["name"] == "quality-judge"][0]["criteria"] + + +def tree(root): + out = {} + for dp, dns, fns in os.walk(root): + dns[:] = [d for d in dns if d not in SKIP_DIRS] + for fn in fns: + full = os.path.join(dp, fn) + out[os.path.relpath(full, root).replace(os.sep, "/")] = full + return out + + +def changes(scaffold, work): + with tempfile.TemporaryDirectory(prefix="routing-judge-", ignore_cleanup_errors=True) as tmp: + base = os.path.join(tmp, "base") + os.makedirs(base) + bash = shutil.which("bash") or "bash" + subprocess.run([bash, os.path.abspath(scaffold)], cwd=base, capture_output=True, text=True, timeout=120, check=True) + before, after = tree(base), tree(work) + added = sorted(p for p in after if p not in before) + modified = sorted(p for p in after if p in before and not filecmp.cmp(before[p], after[p], shallow=False)) + deleted = sorted(p for p in before if p not in after) + parts, total = [], 0 + for label, paths in (("added", added), ("modified", modified)): + for p in paths: + try: + body = open(after[p], encoding="utf-8").read() + except (UnicodeDecodeError, OSError): + body = "(not a text file)" + if len(body) > MAX_FILE: + body = body[:MAX_FILE] + "\n... (cut at %d characters)" % MAX_FILE + chunk = "=== %s: %s ===\n%s\n" % (label, p, body) + if total + len(chunk) > MAX_TOTAL: + parts.append("=== %s: %s === (not shown: the shown files already reach %d characters)\n" % (label, p, MAX_TOTAL)) + continue + parts.append(chunk) + total += len(chunk) + for p in deleted: + parts.append("=== deleted: %s ===\n" % p) + return "".join(parts) + + +def vote(prompt, cwd): + # The prompt goes in on stdin: a Windows command line stops at about 32K characters. + cmd = ["claude", "-p", "--model", "opus", "--system-prompt", SYSTEM, "--tools", "", + "--setting-sources", "project", "--strict-mcp-config", "--disable-slash-commands", + "--no-session-persistence", "--max-turns", "1", "--output-format", "json"] + p = subprocess.run(cmd, cwd=cwd, input=prompt, capture_output=True, text=True, encoding="utf-8", timeout=600) + d = json.loads(p.stdout) + text = d.get("result") or "" + m = re.findall(r"VERDICT:\s*(PASS|FAIL)", text) + return (m[-1] == "PASS") if m else None, float(d.get("total_cost_usd") or 0.0), list((d.get("modelUsage") or {}).keys()) + + +def main(argv): + if len(argv) != 4: + print(__doc__.strip(), file=sys.stderr) + return 2 + case_yaml, scaffold, work = argv[1], argv[2], argv[3] + if not os.path.isdir(work): + print(json.dumps({"verdict": "COULD NOT RUN", "reason": "no work directory", "votes": [], "cost_usd": 0.0})) + return 2 + files = changes(scaffold, work) + prompt = (criteria_of(case_yaml) + "\n\nAgent output (files):\n" + (files or "(no file changes)")) + votes, cost, models = [], 0.0, set() + with tempfile.TemporaryDirectory(prefix="routing-judge-cwd-", ignore_cleanup_errors=True) as cwd: + for _ in range(VOTES): + v, c, ms = vote(prompt, cwd) + votes.append(v) + cost += c + models.update(ms) + passes = sum(1 for v in votes if v is True) + verdict = "PASS" if passes * 2 > VOTES else "FAIL" + print(json.dumps({"verdict": verdict, "votes": votes, "cost_usd": round(cost, 6), "models": sorted(models), + "changed_chars": len(files)})) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/evals/routing-2026-10-23/judge_runs.py b/evals/routing-2026-10-23/judge_runs.py new file mode 100644 index 0000000..5374ff5 --- /dev/null +++ b/evals/routing-2026-10-23/judge_runs.py @@ -0,0 +1,27 @@ +"""Run judge_files.py on every run of an eval JSON (amendment 1); append one JSON line per run. + + judge_runs.py +""" +import json, os, subprocess, sys +from concurrent.futures import ThreadPoolExecutor +E = os.path.dirname(os.path.abspath(__file__)) +d = json.load(open(sys.argv[1], encoding="utf-8")) +plugin, out = sys.argv[2], sys.argv[3] +jobs = [] +for c in d["cases"]: + for arm, runs in c["arms"].items(): + for i, r in enumerate(runs): + root = os.path.dirname(os.path.dirname(r["tracePath"])) + jobs.append((c["name"], arm, i + 1, os.path.join(root, "home", "cwd"))) +def one(j): + name, arm, run, work = j + cy = os.path.join(plugin, "evals", name, "case.yaml"); sc = os.path.join(plugin, "evals", name, "scaffold.sh") + p = subprocess.run([sys.executable, os.path.join(E, "judge_files.py"), cy, sc, work], capture_output=True, text=True, encoding="utf-8") + try: res = json.loads(p.stdout.strip().splitlines()[-1]) + except Exception: res = {"verdict": "COULD NOT RUN", "reason": (p.stderr or p.stdout)[-300:]} + res.update(case=name, arm=arm, run=run, work=work) + return res +with ThreadPoolExecutor(3) as ex, open(out, "a", encoding="utf-8") as f: + for res in ex.map(one, jobs): + f.write(json.dumps(res) + "\n"); f.flush() + print(res["case"], res["arm"], res["run"], res["verdict"], res.get("cost_usd")) diff --git a/evals/routing-2026-10-23/results-2026-10-23-main.json b/evals/routing-2026-10-23/results-2026-10-23-main.json new file mode 100644 index 0000000..315c674 --- /dev/null +++ b/evals/routing-2026-10-23/results-2026-10-23-main.json @@ -0,0 +1,2259 @@ +{ + "note": "Distilled from the eval JSON, the hidden tests and the files judge by analyze.py (PREREG-2026-10-23-routing.md). agent_cost = cost_usd - judge_cost_in_eval. 'judge' is judge_files.py (amendment 1); the in-eval quality-judge is void. Kept traces and transcripts stay on the machine that ran them.", + "studies": { + "R": { + "summary": { + "study": "R", + "claude_version": "2.1.305", + "started_at": "2026-10-23T21:38:50.944Z", + "eval_cost_usd": 21.858342055, + "partial": false, + "cells": { + "hr-limiter-starve|routed": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.3878359775, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 7.5, + "agents": 11, + "agents_non_opus": 7, + "models": [ + "claude-haiku-5-5", + "claude-opus-5-5", + "claude-sonnet-5-5" + ] + }, + "hr-limiter-starve|allopus": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.44102263333333336, + "hidden_pass": 6, + "judge_pass": 5, + "mean_turns": 8.666666666666666, + "agents": 7, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hr-note-cache|routed": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.31078108, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 10.833333333333334, + "agents": 12, + "agents_non_opus": 12, + "models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + "hr-note-cache|allopus": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.4942568666666667, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 9.833333333333334, + "agents": 13, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hr-page-size|routed": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.18771163166666668, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 7.833333333333333, + "agents": 11, + "agents_non_opus": 11, + "models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + "hr-page-size|allopus": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.35549830000000004, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 6.0, + "agents": 12, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hr-reminder-every|routed": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.23331246166666667, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 8.333333333333334, + "agents": 19, + "agents_non_opus": 19, + "models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + "hr-reminder-every|allopus": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.47663553333333325, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 6.166666666666667, + "agents": 24, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hr-retry-limit|routed": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.24325749166666666, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 7.166666666666667, + "agents": 14, + "agents_non_opus": 14, + "models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + "hr-retry-limit|allopus": { + "planned_or_seen": 6, + "counted": 6, + "mean_agent_cost": 0.4401583666666667, + "hidden_pass": 6, + "judge_pass": 6, + "mean_turns": 5.5, + "agents": 13, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + } + }, + "runs_planned_or_seen": { + "routed": 30, + "allopus": 30 + }, + "runs_counted": { + "routed": 30, + "allopus": 30 + }, + "pooled_mean_agent_cost": { + "routed": 0.2725797285, + "allopus": 0.44151434 + }, + "cost_ratio": [ + 0.617374576100971, + 0.5704435212103913, + 0.6651667714348987 + ], + "cost_outcome": "savings detected", + "quality_tasks": [ + "hr-note-cache", + "hr-page-size", + "hr-reminder-every", + "hr-retry-limit" + ], + "hidden": { + "routed": [ + 24, + 24 + ], + "allopus": [ + 24, + 24 + ], + "dq_newcombe": [ + 0.0, + -0.13797620467498017, + 0.13797620467498017 + ] + }, + "judge": "opus, 3 votes (judge_files.py)", + "quality_outcome": "non-inferior", + "quality_gain_unexpected": false, + "careful": "hr-limiter-starve", + "careful_harm": false, + "careful_routed_wins_by_2": false, + "agents_non_opus": { + "routed": [ + 63, + 67 + ], + "allopus": [ + 0, + 69 + ], + "share_routed": 0.9402985074626866, + "share_allopus": 0.0 + }, + "manipulation": "took effect", + "verdict": "saves, no measured loss", + "cap_usd": 28.93, + "started_utc": "2026-10-23T21:38:50Z", + "executor": "opus (claude-opus-5-5)", + "plugin": "dojo-router 0.1.1" + }, + "runs": [ + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 1, + "cost_usd": 0.453505, + "judge_cost_in_eval": 0.01036, + "agent_cost_usd": 0.443145, + "turns": 6, + "duration_s": 127, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.118887, + "helper_models": [ + "claude-haiku-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 2, + "cost_usd": 0.519803, + "judge_cost_in_eval": 0.010436, + "agent_cost_usd": 0.509367, + "turns": 6, + "duration_s": 132, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.136935, + "helper_models": [ + "claude-sonnet-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 3, + "cost_usd": 0.38851, + "judge_cost_in_eval": 0.006736, + "agent_cost_usd": 0.381774, + "turns": 7, + "duration_s": 111, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.136613, + "helper_models": [ + "claude-haiku-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 4, + "cost_usd": 0.286291, + "judge_cost_in_eval": 0.006976, + "agent_cost_usd": 0.279315, + "turns": 8, + "duration_s": 94, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.144482, + "helper_models": [ + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 5, + "cost_usd": 0.449328, + "judge_cost_in_eval": 0.006876, + "agent_cost_usd": 0.442452, + "turns": 11, + "duration_s": 120, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.084558, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "routed", + "run": 6, + "cost_usd": 0.277519, + "judge_cost_in_eval": 0.006556, + "agent_cost_usd": 0.270963, + "turns": 7, + "duration_s": 83, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.116832, + "helper_models": [ + "claude-sonnet-5-5", + "claude-sonnet-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 1, + "cost_usd": 0.516124, + "judge_cost_in_eval": 0.006956, + "agent_cost_usd": 0.509168, + "turns": 9, + "duration_s": 145, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.144156, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 2, + "cost_usd": 0.457275, + "judge_cost_in_eval": 0.007196, + "agent_cost_usd": 0.450079, + "turns": 9, + "duration_s": 141, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.117037, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 3, + "cost_usd": 0.447199, + "judge_cost_in_eval": 0.008508, + "agent_cost_usd": 0.438691, + "turns": 10, + "duration_s": 136, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.104147, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 4, + "cost_usd": 0.440129, + "judge_cost_in_eval": 0.007136, + "agent_cost_usd": 0.432993, + "turns": 8, + "duration_s": 112, + "error": null, + "hidden": "PASS", + "judge": "FAIL", + "judge_votes": [ + false, + true, + false + ], + "judge_cost_usd": 0.160061, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 5, + "cost_usd": 0.388201, + "judge_cost_in_eval": 0.006076, + "agent_cost_usd": 0.382125, + "turns": 7, + "duration_s": 112, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.117352, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-limiter-starve", + "task": "hr-limiter-starve", + "arm": "allopus", + "run": 6, + "cost_usd": 0.440156, + "judge_cost_in_eval": 0.007076, + "agent_cost_usd": 0.43308, + "turns": 9, + "duration_s": 123, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.133364, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 1, + "cost_usd": 0.347468, + "judge_cost_in_eval": 0.007004, + "agent_cost_usd": 0.340464, + "turns": 9, + "duration_s": 65, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.066021, + "helper_models": [ + "claude-sonnet-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 2, + "cost_usd": 0.374773, + "judge_cost_in_eval": 0.007584, + "agent_cost_usd": 0.367189, + "turns": 5, + "duration_s": 80, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.057973, + "helper_models": [ + "claude-sonnet-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 3, + "cost_usd": 0.332238, + "judge_cost_in_eval": 0.006504, + "agent_cost_usd": 0.325734, + "turns": 14, + "duration_s": 71, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.063765, + "helper_models": [ + "claude-sonnet-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 4, + "cost_usd": 0.242792, + "judge_cost_in_eval": 0.006684, + "agent_cost_usd": 0.236108, + "turns": 12, + "duration_s": 60, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061633, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 5, + "cost_usd": 0.275018, + "judge_cost_in_eval": 0.006844, + "agent_cost_usd": 0.268174, + "turns": 15, + "duration_s": 68, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061909, + "helper_models": [ + "claude-sonnet-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "routed", + "run": 6, + "cost_usd": 0.333981, + "judge_cost_in_eval": 0.006964, + "agent_cost_usd": 0.327018, + "turns": 10, + "duration_s": 63, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.066057, + "helper_models": [ + "claude-sonnet-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 1, + "cost_usd": 0.626779, + "judge_cost_in_eval": 0.008792, + "agent_cost_usd": 0.617987, + "turns": 8, + "duration_s": 95, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.064997, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 2, + "cost_usd": 0.506519, + "judge_cost_in_eval": 0.01114, + "agent_cost_usd": 0.495379, + "turns": 9, + "duration_s": 85, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.064741, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 3, + "cost_usd": 0.510551, + "judge_cost_in_eval": 0.010532, + "agent_cost_usd": 0.500019, + "turns": 12, + "duration_s": 94, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.096671, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 4, + "cost_usd": 0.521944, + "judge_cost_in_eval": 0.01336, + "agent_cost_usd": 0.508584, + "turns": 12, + "duration_s": 94, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.091684, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 5, + "cost_usd": 0.443239, + "judge_cost_in_eval": 0.01184, + "agent_cost_usd": 0.431399, + "turns": 9, + "duration_s": 73, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.064641, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-note-cache", + "task": "hr-note-cache", + "arm": "allopus", + "run": 6, + "cost_usd": 0.420905, + "judge_cost_in_eval": 0.008732, + "agent_cost_usd": 0.412173, + "turns": 9, + "duration_s": 66, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.062969, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 1, + "cost_usd": 0.196631, + "judge_cost_in_eval": 0.008372, + "agent_cost_usd": 0.188259, + "turns": 7, + "duration_s": 55, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.068692, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 2, + "cost_usd": 0.198896, + "judge_cost_in_eval": 0.01128, + "agent_cost_usd": 0.187616, + "turns": 7, + "duration_s": 56, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.062552, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 3, + "cost_usd": 0.162958, + "judge_cost_in_eval": 0.01024, + "agent_cost_usd": 0.152718, + "turns": 10, + "duration_s": 63, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.064476, + "helper_models": [ + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 4, + "cost_usd": 0.242117, + "judge_cost_in_eval": 0.008232, + "agent_cost_usd": 0.233885, + "turns": 10, + "duration_s": 56, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.0761, + "helper_models": [ + "claude-sonnet-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 5, + "cost_usd": 0.192936, + "judge_cost_in_eval": 0.006744, + "agent_cost_usd": 0.186192, + "turns": 7, + "duration_s": 49, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.068464, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "routed", + "run": 6, + "cost_usd": 0.183784, + "judge_cost_in_eval": 0.006184, + "agent_cost_usd": 0.1776, + "turns": 6, + "duration_s": 53, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.059876, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 1, + "cost_usd": 0.358756, + "judge_cost_in_eval": 0.006904, + "agent_cost_usd": 0.351852, + "turns": 6, + "duration_s": 52, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.062608, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 2, + "cost_usd": 0.367501, + "judge_cost_in_eval": 0.006384, + "agent_cost_usd": 0.361117, + "turns": 5, + "duration_s": 67, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.059536, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 3, + "cost_usd": 0.364141, + "judge_cost_in_eval": 0.006644, + "agent_cost_usd": 0.357497, + "turns": 7, + "duration_s": 51, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.0534, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 4, + "cost_usd": 0.353552, + "judge_cost_in_eval": 0.006204, + "agent_cost_usd": 0.347348, + "turns": 7, + "duration_s": 60, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.056164, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 5, + "cost_usd": 0.360542, + "judge_cost_in_eval": 0.006224, + "agent_cost_usd": 0.354318, + "turns": 6, + "duration_s": 51, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.052928, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-page-size", + "task": "hr-page-size", + "arm": "allopus", + "run": 6, + "cost_usd": 0.366943, + "judge_cost_in_eval": 0.006084, + "agent_cost_usd": 0.360859, + "turns": 5, + "duration_s": 53, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061916, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 1, + "cost_usd": 0.258895, + "judge_cost_in_eval": 0.00476, + "agent_cost_usd": 0.254135, + "turns": 9, + "duration_s": 59, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061784, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 2, + "cost_usd": 0.225778, + "judge_cost_in_eval": 0.00504, + "agent_cost_usd": 0.220738, + "turns": 7, + "duration_s": 49, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061904, + "helper_models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 3, + "cost_usd": 0.223074, + "judge_cost_in_eval": 0.00384, + "agent_cost_usd": 0.219234, + "turns": 7, + "duration_s": 62, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.063088, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 4, + "cost_usd": 0.240343, + "judge_cost_in_eval": 0.00484, + "agent_cost_usd": 0.235503, + "turns": 6, + "duration_s": 57, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.0592, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 5, + "cost_usd": 0.218765, + "judge_cost_in_eval": 0.00674, + "agent_cost_usd": 0.212025, + "turns": 13, + "duration_s": 73, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.064256, + "helper_models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "routed", + "run": 6, + "cost_usd": 0.262741, + "judge_cost_in_eval": 0.0045, + "agent_cost_usd": 0.258241, + "turns": 8, + "duration_s": 71, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.0682, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-haiku-5-5", + "claude-sonnet-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 1, + "cost_usd": 0.506171, + "judge_cost_in_eval": 0.00532, + "agent_cost_usd": 0.500851, + "turns": 7, + "duration_s": 63, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.05934, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 2, + "cost_usd": 0.467, + "judge_cost_in_eval": 0.00524, + "agent_cost_usd": 0.46176, + "turns": 7, + "duration_s": 71, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.057988, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 3, + "cost_usd": 0.459462, + "judge_cost_in_eval": 0.00498, + "agent_cost_usd": 0.454482, + "turns": 6, + "duration_s": 67, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.058916, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 4, + "cost_usd": 0.488469, + "judge_cost_in_eval": 0.0062, + "agent_cost_usd": 0.482269, + "turns": 6, + "duration_s": 78, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.060664, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 5, + "cost_usd": 0.492179, + "judge_cost_in_eval": 0.00498, + "agent_cost_usd": 0.487199, + "turns": 4, + "duration_s": 65, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.060216, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-reminder-every", + "task": "hr-reminder-every", + "arm": "allopus", + "run": 6, + "cost_usd": 0.477933, + "judge_cost_in_eval": 0.00468, + "agent_cost_usd": 0.473253, + "turns": 7, + "duration_s": 89, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.061556, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 1, + "cost_usd": 0.225256, + "judge_cost_in_eval": 0.008488, + "agent_cost_usd": 0.216768, + "turns": 7, + "duration_s": 53, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.081748, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 2, + "cost_usd": 0.222861, + "judge_cost_in_eval": 0.006568, + "agent_cost_usd": 0.216293, + "turns": 8, + "duration_s": 58, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.089524, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 3, + "cost_usd": 0.304616, + "judge_cost_in_eval": 0.007808, + "agent_cost_usd": 0.296808, + "turns": 12, + "duration_s": 94, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.082592, + "helper_models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 4, + "cost_usd": 0.216257, + "judge_cost_in_eval": 0.008148, + "agent_cost_usd": 0.208109, + "turns": 7, + "duration_s": 55, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.06582, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 5, + "cost_usd": 0.304426, + "judge_cost_in_eval": 0.007748, + "agent_cost_usd": 0.296678, + "turns": 8, + "duration_s": 93, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.082236, + "helper_models": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "routed", + "run": 6, + "cost_usd": 0.231737, + "judge_cost_in_eval": 0.006848, + "agent_cost_usd": 0.224889, + "turns": 1, + "duration_s": 53, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.081768, + "helper_models": [ + "claude-haiku-5-5", + "claude-haiku-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 1, + "cost_usd": 0.404374, + "judge_cost_in_eval": 0.007048, + "agent_cost_usd": 0.397326, + "turns": 7, + "duration_s": 64, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.068911, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 2, + "cost_usd": 0.418957, + "judge_cost_in_eval": 0.006968, + "agent_cost_usd": 0.411989, + "turns": 1, + "duration_s": 70, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.06938, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 3, + "cost_usd": 0.41353, + "judge_cost_in_eval": 0.007448, + "agent_cost_usd": 0.406082, + "turns": 7, + "duration_s": 82, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.071048, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 4, + "cost_usd": 0.420519, + "judge_cost_in_eval": 0.006848, + "agent_cost_usd": 0.413671, + "turns": 7, + "duration_s": 72, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.0708, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 5, + "cost_usd": 0.636516, + "judge_cost_in_eval": 0.007308, + "agent_cost_usd": 0.629208, + "turns": 2, + "duration_s": 129, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.089216, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5", + "claude-opus-5-5" + ] + }, + { + "case": "hr-retry-limit", + "task": "hr-retry-limit", + "arm": "allopus", + "run": 6, + "cost_usd": 0.389483, + "judge_cost_in_eval": 0.006808, + "agent_cost_usd": 0.382675, + "turns": 9, + "duration_s": 66, + "error": null, + "hidden": "PASS", + "judge": "PASS", + "judge_votes": [ + true, + true, + true + ], + "judge_cost_usd": 0.06738, + "helper_models": [ + "claude-opus-5-5", + "claude-opus-5-5" + ] + } + ] + }, + "F": { + "summary": { + "study": "F", + "claude_version": "2.1.305", + "started_at": "2026-10-23T22:18:46.491Z", + "eval_cost_usd": 15.751031999999999, + "partial": false, + "cells": { + "hf-gradebook|routed": { + "planned_or_seen": 5, + "counted": 5, + "mean_agent_cost": 0.72835308, + "hidden_pass": 0, + "judge_pass": 0, + "mean_turns": 2.0, + "agents": 5, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hf-gradebook|allopus": { + "planned_or_seen": 5, + "counted": 5, + "mean_agent_cost": 0.71823656, + "hidden_pass": 0, + "judge_pass": 0, + "mean_turns": 2.0, + "agents": 5, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hf-semver-ranges|routed": { + "planned_or_seen": 5, + "counted": 5, + "mean_agent_cost": 0.82904664, + "hidden_pass": 0, + "judge_pass": 0, + "mean_turns": 3.0, + "agents": 5, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + }, + "hf-semver-ranges|allopus": { + "planned_or_seen": 5, + "counted": 5, + "mean_agent_cost": 0.84312212, + "hidden_pass": 0, + "judge_pass": 0, + "mean_turns": 2.4, + "agents": 5, + "agents_non_opus": 0, + "models": [ + "claude-opus-5-5" + ] + } + }, + "runs_planned_or_seen": { + "routed": 10, + "allopus": 10 + }, + "runs_counted": { + "routed": 10, + "allopus": 10 + }, + "pooled_mean_agent_cost": { + "routed": 0.7786998599999999, + "allopus": 0.78067934 + }, + "cost_ratio": [ + 0.9974644134940216, + 0.920586560163663, + 1.0804471884874487 + ], + "cost_outcome": "no detectable difference", + "quality_tasks": [ + "hf-gradebook", + "hf-semver-ranges" + ], + "hidden": { + "routed": [ + 0, + 10 + ], + "allopus": [ + 0, + 10 + ], + "dq_newcombe": [ + 0.0, + -0.2775327998628892, + 0.2775327998628892 + ] + }, + "judge": "opus, 3 votes (judge_files.py)", + "quality_outcome": "not resolved", + "quality_gain_unexpected": false, + "careful": "hf-semver-ranges", + "careful_harm": false, + "careful_routed_wins_by_2": false, + "agents_non_opus": { + "routed": [ + 0, + 10 + ], + "allopus": [ + 0, + 10 + ], + "share_routed": 0.0, + "share_allopus": 0.0 + }, + "manipulation": "did not take effect", + "verdict": "inconclusive: no savings detected and routing did not take effect or was not measured", + "cap_usd": 20.72, + "started_utc": "2026-10-23T22:18:45Z", + "executor": "opus (claude-opus-5-5)", + "plugin": "dojo-flow 0.1.0" + }, + "runs": [ + { + "case": "hf-gradebook-allopus", + "task": "hf-gradebook", + "arm": "allopus", + "run": 1, + "cost_usd": 0.698023, + "judge_cost_in_eval": 0.01154, + "agent_cost_usd": 0.686483, + "turns": 2, + "duration_s": 187, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.065239, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-allopus", + "task": "hf-gradebook", + "arm": "allopus", + "run": 2, + "cost_usd": 0.788649, + "judge_cost_in_eval": 0.0136, + "agent_cost_usd": 0.775049, + "turns": 2, + "duration_s": 187, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.065711, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-allopus", + "task": "hf-gradebook", + "arm": "allopus", + "run": 3, + "cost_usd": 0.739588, + "judge_cost_in_eval": 0.0061, + "agent_cost_usd": 0.733488, + "turns": 2, + "duration_s": 198, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.056566, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-allopus", + "task": "hf-gradebook", + "arm": "allopus", + "run": 4, + "cost_usd": 0.69519, + "judge_cost_in_eval": 0.00678, + "agent_cost_usd": 0.68841, + "turns": 2, + "duration_s": 168, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.058236, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-allopus", + "task": "hf-gradebook", + "arm": "allopus", + "run": 5, + "cost_usd": 0.715154, + "judge_cost_in_eval": 0.0074, + "agent_cost_usd": 0.707754, + "turns": 2, + "duration_s": 167, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.060868, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-routed", + "task": "hf-gradebook", + "arm": "routed", + "run": 1, + "cost_usd": 0.718878, + "judge_cost_in_eval": 0.0059, + "agent_cost_usd": 0.712978, + "turns": 2, + "duration_s": 237, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.066269, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-routed", + "task": "hf-gradebook", + "arm": "routed", + "run": 2, + "cost_usd": 0.717568, + "judge_cost_in_eval": 0.00576, + "agent_cost_usd": 0.711808, + "turns": 2, + "duration_s": 181, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.059001, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-routed", + "task": "hf-gradebook", + "arm": "routed", + "run": 3, + "cost_usd": 0.833952, + "judge_cost_in_eval": 0.00646, + "agent_cost_usd": 0.827492, + "turns": 2, + "duration_s": 214, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.054722, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-routed", + "task": "hf-gradebook", + "arm": "routed", + "run": 4, + "cost_usd": 0.798681, + "judge_cost_in_eval": 0.00644, + "agent_cost_usd": 0.792241, + "turns": 2, + "duration_s": 215, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.064302, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-gradebook-routed", + "task": "hf-gradebook", + "arm": "routed", + "run": 5, + "cost_usd": 0.603668, + "judge_cost_in_eval": 0.00642, + "agent_cost_usd": 0.597248, + "turns": 2, + "duration_s": 140, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.055612, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-allopus", + "task": "hf-semver-ranges", + "arm": "allopus", + "run": 1, + "cost_usd": 0.944354, + "judge_cost_in_eval": 0.007472, + "agent_cost_usd": 0.936882, + "turns": 2, + "duration_s": 234, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.073545, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-allopus", + "task": "hf-semver-ranges", + "arm": "allopus", + "run": 2, + "cost_usd": 0.715274, + "judge_cost_in_eval": 0.007792, + "agent_cost_usd": 0.707482, + "turns": 2, + "duration_s": 170, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.059379, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-allopus", + "task": "hf-semver-ranges", + "arm": "allopus", + "run": 3, + "cost_usd": 0.955918, + "judge_cost_in_eval": 0.006692, + "agent_cost_usd": 0.949226, + "turns": 2, + "duration_s": 237, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.07786, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-allopus", + "task": "hf-semver-ranges", + "arm": "allopus", + "run": 4, + "cost_usd": 0.777085, + "judge_cost_in_eval": 0.007512, + "agent_cost_usd": 0.769573, + "turns": 3, + "duration_s": 194, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.07247, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-allopus", + "task": "hf-semver-ranges", + "arm": "allopus", + "run": 5, + "cost_usd": 0.861679, + "judge_cost_in_eval": 0.009232, + "agent_cost_usd": 0.852447, + "turns": 3, + "duration_s": 205, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.070428, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-routed", + "task": "hf-semver-ranges", + "arm": "routed", + "run": 1, + "cost_usd": 0.896438, + "judge_cost_in_eval": 0.008692, + "agent_cost_usd": 0.887746, + "turns": 4, + "duration_s": 217, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.075361, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-routed", + "task": "hf-semver-ranges", + "arm": "routed", + "run": 2, + "cost_usd": 0.784442, + "judge_cost_in_eval": 0.007652, + "agent_cost_usd": 0.77679, + "turns": 2, + "duration_s": 185, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.069693, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-routed", + "task": "hf-semver-ranges", + "arm": "routed", + "run": 3, + "cost_usd": 0.741418, + "judge_cost_in_eval": 0.008552, + "agent_cost_usd": 0.732866, + "turns": 2, + "duration_s": 173, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.068819, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-routed", + "task": "hf-semver-ranges", + "arm": "routed", + "run": 4, + "cost_usd": 0.848551, + "judge_cost_in_eval": 0.008692, + "agent_cost_usd": 0.839859, + "turns": 5, + "duration_s": 205, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.06848, + "helper_models": [ + "claude-opus-5-5" + ] + }, + { + "case": "hf-semver-ranges-routed", + "task": "hf-semver-ranges", + "arm": "routed", + "run": 5, + "cost_usd": 0.916524, + "judge_cost_in_eval": 0.008552, + "agent_cost_usd": 0.907972, + "turns": 2, + "duration_s": 225, + "error": null, + "hidden": "FAIL", + "judge": "FAIL", + "judge_votes": [ + false, + false, + false + ], + "judge_cost_usd": 0.071188, + "helper_models": [ + "claude-opus-5-5" + ] + } + ] + } + } +} diff --git a/evals/routing-2026-10-23/selfcheck-before-freeze.txt b/evals/routing-2026-10-23/selfcheck-before-freeze.txt new file mode 100644 index 0000000..7f1c9fc --- /dev/null +++ b/evals/routing-2026-10-23/selfcheck-before-freeze.txt @@ -0,0 +1,29 @@ +# grade_hidden.py selfcheck on every case, run 2026-10-23T21:14:06Z before the freeze, Python 3.14.2, Windows 11 / Git Bash +== hf-gradebook +known-bad (scaffold as written): FAIL Ran 1 test in 0.000s | | FAILED (errors=1) +known-good (reference laid over): PASS Ran 37 tests in 0.004s | | OK +exit 0 +== hf-semver-ranges +known-bad (scaffold as written): FAIL Ran 1 test in 0.000s | | FAILED (errors=1) +known-good (reference laid over): PASS Ran 51 tests in 0.004s | | OK +exit 0 +== hr-limiter-starve +known-bad (scaffold as written): FAIL Ran 16 tests in 0.003s | | FAILED (failures=10) +known-good (reference laid over): PASS Ran 16 tests in 0.000s | | OK +exit 0 +== hr-note-cache +known-bad (scaffold as written): FAIL Ran 17 tests in 0.004s | | FAILED (failures=9) +known-good (reference laid over): PASS Ran 17 tests in 0.002s | | OK +exit 0 +== hr-page-size +known-bad (scaffold as written): FAIL Ran 21 tests in 0.006s | | FAILED (failures=17) +known-good (reference laid over): PASS Ran 21 tests in 0.001s | | OK +exit 0 +== hr-reminder-every +known-bad (scaffold as written): FAIL Ran 25 tests in 0.004s | | FAILED (failures=5, errors=4) +known-good (reference laid over): PASS Ran 25 tests in 0.001s | | OK +exit 0 +== hr-retry-limit +known-bad (scaffold as written): FAIL Ran 14 tests in 0.004s | | FAILED (failures=6) +known-good (reference laid over): PASS Ran 14 tests in 0.002s | | OK +exit 0 diff --git a/evals/routing-2026-10-23/spend-ledger.md b/evals/routing-2026-10-23/spend-ledger.md new file mode 100644 index 0000000..08461bc --- /dev/null +++ b/evals/routing-2026-10-23/spend-ledger.md @@ -0,0 +1,21 @@ +# Spend ledger: routing eval, cap $100 + +Every `claude plugin eval` invocation for this eval, in order, with the cap it was given before it started and +the top-level `costUsd` it reported when it ended. Pilots, re-pilots, main runs and the judge all count. The runs +bill to the logged-in claude.ai subscription (auth method `claude.ai`, no API key in the environment), so these +are Claude Code's list-price equivalents of plan usage, not invoices. + +| # | when (UTC) | invocation | runs | cap given | costUsd | running total | left of $100 | +|---|---|---|---|---|---|---|---| +| 1 | 2026-10-23 21:15:42 | pilot R, attempt 1 (`--allow-tools Bash Edit Write`) | 10 (0 started: shell grant refused) | $5.00 | $0.1175 | $0.1175 | $99.88 | +| 2 | 2026-10-23 21:17:16 | pilot R, attempt 2 (`--allow-tools Edit Write`) | 10 | $4.85 | $3.7093 | $3.8268 | $96.17 | +| 3 | 2026-10-23 21:25-21:35 | files-judge probes: 1 Haiku isolation probe, planted good and bad on hr-retry-limit (`claude -p`, 7 calls) | n/a | n/a | $0.1195 | $3.9463 | $96.05 | +| 4 | 2026-10-23 21:28:20 | pilot F, attempt 1 (`--allow-tools Edit Write`; Workflow withheld, no workflow ran) | 4 | $12.00 | $0.5373 | $4.4836 | $95.52 | +| 5 | 2026-10-23 21:29-21:40 | files judge on pilot R attempt 2 (`judge_files.py`, 10 runs x 3 Opus votes) | 10 | n/a | $0.8095 | $5.2931 | $94.71 | +| 6 | 2026-10-23 21:29:44 | pilot F, attempt 2 (`--allow-tools Edit Write Workflow`) | 4 | $11.46 | $3.1872 | $8.4803 | $91.52 | +| 7 | 2026-10-23 21:38:50 | main R (frozen grid, amendment 1 flags) | 60 | $28.93 | $21.8583 | $30.3386 | $69.66 | +| 8 | 2026-10-23 22:19-22:35 | files judge on main R (`judge_runs.py`, 60 runs x 3 Opus votes) | 60 | n/a | $4.7417 | $35.0803 | $64.92 | +| 9 | 2026-10-23 22:18:45 | main F (frozen grid, amendment 1 flags) | 20 | $20.72 | $15.7510 | $50.8313 | $49.17 | +| 10 | 2026-10-23 22:52-23:00 | files judge on main F (`judge_runs.py`, 20 runs x 3 Opus votes) | 20 | n/a | $1.3137 | $52.1450 | $47.86 | + +Total: **$52.15** of the $100 cap ($8.48 pilots, excluded from analysis; $43.66 main runs and their judge). $47.86 unspent. No run was stopped by a cap. diff --git a/evals/routing-2026-10-23/stats.py b/evals/routing-2026-10-23/stats.py new file mode 100644 index 0000000..b02eb56 --- /dev/null +++ b/evals/routing-2026-10-23/stats.py @@ -0,0 +1,51 @@ +"""Interval methods named in section 4 of PREREG-2026-10-23-routing.md. Standard library only.""" +import math +import random + +Z95 = 1.959963984540054 + + +def wilson(x, n, z=Z95): + """Wilson score interval for x successes out of n.""" + if n == 0: + return (float("nan"), float("nan")) + p = x / n + den = 1 + z * z / n + centre = (p + z * z / (2 * n)) / den + half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den + return (centre - half, centre + half) + + +def newcombe_diff(x1, n1, x2, n2, z=Z95): + """p1 - p2 with Newcombe's hybrid score interval (1998, method 10).""" + p1, p2 = x1 / n1, x2 / n2 + l1, u1 = wilson(x1, n1, z) + l2, u2 = wilson(x2, n2, z) + d = p1 - p2 + lo = d - math.sqrt((p1 - l1) ** 2 + (u2 - p2) ** 2) + hi = d + math.sqrt((u1 - p1) ** 2 + (p2 - l2) ** 2) + return d, lo, hi + + +def pooled_mean(cells): + """cells: {case: [values]} -> mean over cases of each case's mean (every case weighs the same).""" + means = [sum(v) / len(v) for v in cells.values() if v] + return sum(means) / len(means) + + +def bootstrap_ratio(routed, base, resamples=10000, seed=20261023): + """Ratio of pooled means routed/base with a 95% percentile interval. + + routed, base: {case: [agent cost per run]}. Runs are resampled with replacement within each case and arm. + """ + rng = random.Random(seed) + point = pooled_mean(routed) / pooled_mean(base) + stats = [] + for _ in range(resamples): + r = {c: [rng.choice(v) for _ in v] for c, v in routed.items() if v} + b = {c: [rng.choice(v) for _ in v] for c, v in base.items() if v} + stats.append(pooled_mean(r) / pooled_mean(b)) + stats.sort() + lo = stats[int(0.025 * resamples)] + hi = stats[int(0.975 * resamples) - 1] + return point, lo, hi diff --git a/plugins/dojo-flow/evals/hf-gradebook-allopus/case.yaml b/plugins/dojo-flow/evals/hf-gradebook-allopus/case.yaml new file mode 100644 index 0000000..846e5a5 --- /dev/null +++ b/plugins/dojo-flow/evals/hf-gradebook-allopus/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hf-gradebook-allopus", + "description": "Two parallel tracks build a gradebook: statistics helpers, and CSV report plus command line using them. (every role on opus)", + "tags": [ + "held-out", + "routing", + "standard", + "allopus" + ], + "runs": 5, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow tool with name \"dojo-flow:build\"; if that name is not known, call it with scriptPath set to the plugin's workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not change the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to finish, then reply with its scorecard exactly as returned.\n\nargs (add repo yourself):\n{\n \"goal\": \"Build a small Python package `gradebook` (standard library only) that turns a class's scores, given as CSV text with a name column and a score column, into a ranked plain-text table with letter grades and a class summary line, plus a command line front end. The directory `gradebook/` and its `__init__.py` already exist, and `tests/test_gradebook.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `stats` writes `gradebook/stats.py`; `report` writes `gradebook/report.py` and `gradebook/cli.py`. The shared interface is `stats.letter_grade(score)` and `stats.summarize(values)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.\",\n \"tracks\": [\n {\n \"name\": \"stats\",\n \"prompt\": \"Write the file `gradebook/stats.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `gradebook/report.py` and `gradebook/cli.py` at the same time; they will call `stats.letter_grade(score)` and `stats.summarize(values)` exactly as described below, so keep those names, arguments and return shapes.\\n\\nProvide exactly these functions:\\n- `mean(values)` -> float: the arithmetic mean of a non-empty sequence of numbers. Raise `ValueError` if it is empty.\\n- `median(values)` -> float: the middle value of the numbers in sorted order; for an even count, the mean of the two middle values. Raise `ValueError` if empty. It must not modify the sequence passed in.\\n- `letter_grade(score)` -> str: `\\\"A\\\"` when score >= 90, `\\\"B\\\"` when >= 80, `\\\"C\\\"` when >= 70, `\\\"D\\\"` when >= 60, otherwise `\\\"F\\\"`. Scores are floats, so 89.99 is `\\\"B\\\"`, 90.0 is `\\\"A\\\"` and 104.5 is `\\\"A\\\"`.\\n- `summarize(values)` -> dict with exactly the keys `count` (int), `mean`, `median`, `min`, `max` for a sequence of numbers. For an empty sequence do not raise: return `{\\\"count\\\": 0, \\\"mean\\\": None, \\\"median\\\": None, \\\"min\\\": None, \\\"max\\\": None}`.\",\n \"files\": [\n \"gradebook/stats.py\"\n ]\n },\n {\n \"name\": \"report\",\n \"prompt\": \"Write the files `gradebook/report.py` and `gradebook/cli.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `gradebook/stats.py` at the same time; do not write that file. Import it with `from gradebook import stats` and use only these two functions from it, which will behave as follows:\\n- `stats.letter_grade(score)` -> str: `\\\"A\\\"` for score >= 90, `\\\"B\\\"` for >= 80, `\\\"C\\\"` for >= 70, `\\\"D\\\"` for >= 60, otherwise `\\\"F\\\"`.\\n- `stats.summarize(values)` -> dict with the keys `count` (int), `mean`, `median`, `min`, `max` for a list of numbers (for an empty list the count is 0 and the others are `None`; you will not need that case).\\n\\n`gradebook/report.py` provides:\\n- `parse_rows(text)` -> list of `(name, score)` tuples with `name` a str and `score` a float. `text` is CSV with two columns, name and score; read it with the `csv` module. Ignore lines that are empty or contain only whitespace. The first remaining line is a header, and is skipped, when it has at least two fields and its second field (stripped) is not a number that `float()` accepts, for example `name,score`; any other line is a data row. Every data row must have exactly two fields, a non-empty name once stripped, and a stripped score that `float()` accepts; otherwise raise `ValueError`. Names are returned stripped. Rows keep their input order. A header line is only recognised as the first remaining line, so a later line with a non-numeric score is an error.\\n- `render(rows)` -> str. If `rows` is empty return exactly `No scores`. Otherwise return these lines joined with `\\\"\\\\n\\\"` and no trailing newline: first `Name | Score | Grade`; then one line per row, ordered by score from highest to lowest and, for equal scores, by name in plain string order (case-sensitive, so `Zed` comes before `bob`), each formatted `{name} | {score:.1f} | {grade}` where grade is `stats.letter_grade(score)` applied to the unrounded score (so a score of 89.96 is printed as `90.0` but still gets `B`); then a last line `Class: {count} students, mean {mean:.1f}, median {median:.1f}` with the numbers taken from `stats.summarize` of all the scores, using `1 student` instead of `1 students` when the count is 1.\\n\\n`gradebook/cli.py` provides:\\n- `main(argv=None, stdin=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdin`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. With more than one item in `argv`, write `usage: python -m gradebook.cli [file.csv]` and a newline to stderr and return 2. With one item, it is the path of a CSV file: read it as UTF-8 (open it with `newline=\\\"\\\"`). With none, read all of `stdin`. Then write `report.render(report.parse_rows(text))` followed by a newline to stdout and return 0. If reading the file raises `OSError` or parsing raises `ValueError`, write `error: ` and a newline to stderr, write nothing to stdout, and return 2.\\n- End the file with `if __name__ == \\\"__main__\\\": raise SystemExit(main())`.\",\n \"files\": [\n \"gradebook/report.py\",\n \"gradebook/cli.py\"\n ]\n }\n ],\n \"gates\": [\n \"python3 -m unittest discover -s tests -q\",\n \"python3 -c \\\"import gradebook.stats, gradebook.report, gradebook.cli\\\"\"\n ],\n \"models\": {\n \"scout\": \"opus\",\n \"builder\": \"opus\",\n \"integrator\": \"opus\",\n \"reviewer\": \"opus\",\n \"checker\": \"opus\",\n \"judge\": \"opus\"\n }\n}", + "max_turns": 12, + "timeout_seconds": 1800, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Glob", + "Grep", + "Bash", + "Skill", + "Workflow" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: gradebook/stats.py, gradebook/report.py and gradebook/cli.py exist and agree on the shared stats interface, and the gates pass: the tests pass and all three modules import. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "workflow-ran", + "tool": "Workflow", + "min": 1 + }, + { + "type": "regex", + "name": "scorecard-relayed", + "target": "last_message", + "pattern": "verdict: (VERIFIED|UNVERIFIED|REFUSED|PLAN ONLY)" + } + ], + "expected_outcome": "gradebook/stats.py, gradebook/report.py and gradebook/cli.py exist and agree on the shared stats interface, and the gates pass: the tests pass and all three modules import." +} diff --git a/plugins/dojo-flow/evals/hf-gradebook-allopus/scaffold.sh b/plugins/dojo-flow/evals/hf-gradebook-allopus/scaffold.sh new file mode 100644 index 0000000..bcf22d9 --- /dev/null +++ b/plugins/dojo-flow/evals/hf-gradebook-allopus/scaffold.sh @@ -0,0 +1,96 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# gradebook + +Turns a class's scores (CSV text) into a ranked table with letter grades. + +Layout: + +- `gradebook/stats.py` numbers: mean, median, letter grades, summary +- `gradebook/report.py` CSV parsing and the text table +- `gradebook/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'gradebook' +cat > 'gradebook/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""gradebook: scores in, ranked report out.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for gradebook.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_gradebook.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from gradebook import cli, report, stats + + +class Stats(unittest.TestCase): + def test_mean(self): + self.assertEqual(stats.mean([1, 2, 3, 4]), 2.5) + + def test_median_odd_and_even(self): + self.assertEqual(stats.median([3, 1, 2]), 2) + self.assertEqual(stats.median([4, 1, 3, 2]), 2.5) + + def test_letter_grades(self): + self.assertEqual(stats.letter_grade(95), "A") + self.assertEqual(stats.letter_grade(85), "B") + self.assertEqual(stats.letter_grade(75), "C") + self.assertEqual(stats.letter_grade(65), "D") + self.assertEqual(stats.letter_grade(10), "F") + + def test_summarize(self): + summary = stats.summarize([50, 100]) + self.assertEqual(summary["count"], 2) + self.assertEqual(summary["mean"], 75) + self.assertEqual(summary["min"], 50) + self.assertEqual(summary["max"], 100) + + +class Report(unittest.TestCase): + def test_parse_rows(self): + rows = report.parse_rows("name,score\nAda,95\nBob,82.5\n") + self.assertEqual(rows, [("Ada", 95.0), ("Bob", 82.5)]) + + def test_render_has_header_rows_and_summary(self): + text = report.render([("Bob", 82.5), ("Ada", 95.0)]) + lines = text.split("\n") + self.assertEqual(lines[0], "Name | Score | Grade") + self.assertEqual(lines[1], "Ada | 95.0 | A") + self.assertEqual(lines[2], "Bob | 82.5 | B") + self.assertTrue(lines[3].startswith("Class: 2 students")) + + def test_render_nothing(self): + self.assertEqual(report.render([]), "No scores") + + +class Cli(unittest.TestCase): + def test_reads_stdin(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main([], io.StringIO("name,score\nAda,95\n"), out, err) + self.assertEqual(code, 0) + self.assertIn("Ada | 95.0 | A", out.getvalue()) + self.assertEqual(err.getvalue(), "") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/plugins/dojo-flow/evals/hf-gradebook-routed/case.yaml b/plugins/dojo-flow/evals/hf-gradebook-routed/case.yaml new file mode 100644 index 0000000..500bfa0 --- /dev/null +++ b/plugins/dojo-flow/evals/hf-gradebook-routed/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hf-gradebook-routed", + "description": "Two parallel tracks build a gradebook: statistics helpers, and CSV report plus command line using them. (routed: workflow defaults)", + "tags": [ + "held-out", + "routing", + "standard", + "routed" + ], + "runs": 5, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow tool with name \"dojo-flow:build\"; if that name is not known, call it with scriptPath set to the plugin's workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not change the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to finish, then reply with its scorecard exactly as returned.\n\nargs (add repo yourself):\n{\n \"goal\": \"Build a small Python package `gradebook` (standard library only) that turns a class's scores, given as CSV text with a name column and a score column, into a ranked plain-text table with letter grades and a class summary line, plus a command line front end. The directory `gradebook/` and its `__init__.py` already exist, and `tests/test_gradebook.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `stats` writes `gradebook/stats.py`; `report` writes `gradebook/report.py` and `gradebook/cli.py`. The shared interface is `stats.letter_grade(score)` and `stats.summarize(values)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.\",\n \"tracks\": [\n {\n \"name\": \"stats\",\n \"prompt\": \"Write the file `gradebook/stats.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `gradebook/report.py` and `gradebook/cli.py` at the same time; they will call `stats.letter_grade(score)` and `stats.summarize(values)` exactly as described below, so keep those names, arguments and return shapes.\\n\\nProvide exactly these functions:\\n- `mean(values)` -> float: the arithmetic mean of a non-empty sequence of numbers. Raise `ValueError` if it is empty.\\n- `median(values)` -> float: the middle value of the numbers in sorted order; for an even count, the mean of the two middle values. Raise `ValueError` if empty. It must not modify the sequence passed in.\\n- `letter_grade(score)` -> str: `\\\"A\\\"` when score >= 90, `\\\"B\\\"` when >= 80, `\\\"C\\\"` when >= 70, `\\\"D\\\"` when >= 60, otherwise `\\\"F\\\"`. Scores are floats, so 89.99 is `\\\"B\\\"`, 90.0 is `\\\"A\\\"` and 104.5 is `\\\"A\\\"`.\\n- `summarize(values)` -> dict with exactly the keys `count` (int), `mean`, `median`, `min`, `max` for a sequence of numbers. For an empty sequence do not raise: return `{\\\"count\\\": 0, \\\"mean\\\": None, \\\"median\\\": None, \\\"min\\\": None, \\\"max\\\": None}`.\",\n \"files\": [\n \"gradebook/stats.py\"\n ]\n },\n {\n \"name\": \"report\",\n \"prompt\": \"Write the files `gradebook/report.py` and `gradebook/cli.py` (Python 3, standard library only). The package `gradebook` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `gradebook/stats.py` at the same time; do not write that file. Import it with `from gradebook import stats` and use only these two functions from it, which will behave as follows:\\n- `stats.letter_grade(score)` -> str: `\\\"A\\\"` for score >= 90, `\\\"B\\\"` for >= 80, `\\\"C\\\"` for >= 70, `\\\"D\\\"` for >= 60, otherwise `\\\"F\\\"`.\\n- `stats.summarize(values)` -> dict with the keys `count` (int), `mean`, `median`, `min`, `max` for a list of numbers (for an empty list the count is 0 and the others are `None`; you will not need that case).\\n\\n`gradebook/report.py` provides:\\n- `parse_rows(text)` -> list of `(name, score)` tuples with `name` a str and `score` a float. `text` is CSV with two columns, name and score; read it with the `csv` module. Ignore lines that are empty or contain only whitespace. The first remaining line is a header, and is skipped, when it has at least two fields and its second field (stripped) is not a number that `float()` accepts, for example `name,score`; any other line is a data row. Every data row must have exactly two fields, a non-empty name once stripped, and a stripped score that `float()` accepts; otherwise raise `ValueError`. Names are returned stripped. Rows keep their input order. A header line is only recognised as the first remaining line, so a later line with a non-numeric score is an error.\\n- `render(rows)` -> str. If `rows` is empty return exactly `No scores`. Otherwise return these lines joined with `\\\"\\\\n\\\"` and no trailing newline: first `Name | Score | Grade`; then one line per row, ordered by score from highest to lowest and, for equal scores, by name in plain string order (case-sensitive, so `Zed` comes before `bob`), each formatted `{name} | {score:.1f} | {grade}` where grade is `stats.letter_grade(score)` applied to the unrounded score (so a score of 89.96 is printed as `90.0` but still gets `B`); then a last line `Class: {count} students, mean {mean:.1f}, median {median:.1f}` with the numbers taken from `stats.summarize` of all the scores, using `1 student` instead of `1 students` when the count is 1.\\n\\n`gradebook/cli.py` provides:\\n- `main(argv=None, stdin=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdin`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. With more than one item in `argv`, write `usage: python -m gradebook.cli [file.csv]` and a newline to stderr and return 2. With one item, it is the path of a CSV file: read it as UTF-8 (open it with `newline=\\\"\\\"`). With none, read all of `stdin`. Then write `report.render(report.parse_rows(text))` followed by a newline to stdout and return 0. If reading the file raises `OSError` or parsing raises `ValueError`, write `error: ` and a newline to stderr, write nothing to stdout, and return 2.\\n- End the file with `if __name__ == \\\"__main__\\\": raise SystemExit(main())`.\",\n \"files\": [\n \"gradebook/report.py\",\n \"gradebook/cli.py\"\n ]\n }\n ],\n \"gates\": [\n \"python3 -m unittest discover -s tests -q\",\n \"python3 -c \\\"import gradebook.stats, gradebook.report, gradebook.cli\\\"\"\n ],\n \"models\": {\n \"scout\": \"haiku\",\n \"builder\": \"sonnet\",\n \"integrator\": \"sonnet\",\n \"reviewer\": \"sonnet\",\n \"checker\": \"haiku\",\n \"judge\": \"opus\"\n }\n}", + "max_turns": 12, + "timeout_seconds": 1800, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Glob", + "Grep", + "Bash", + "Skill", + "Workflow" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: gradebook/stats.py, gradebook/report.py and gradebook/cli.py exist and agree on the shared stats interface, and the gates pass: the tests pass and all three modules import. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "workflow-ran", + "tool": "Workflow", + "min": 1 + }, + { + "type": "regex", + "name": "scorecard-relayed", + "target": "last_message", + "pattern": "verdict: (VERIFIED|UNVERIFIED|REFUSED|PLAN ONLY)" + } + ], + "expected_outcome": "gradebook/stats.py, gradebook/report.py and gradebook/cli.py exist and agree on the shared stats interface, and the gates pass: the tests pass and all three modules import." +} diff --git a/plugins/dojo-flow/evals/hf-gradebook-routed/scaffold.sh b/plugins/dojo-flow/evals/hf-gradebook-routed/scaffold.sh new file mode 100644 index 0000000..bcf22d9 --- /dev/null +++ b/plugins/dojo-flow/evals/hf-gradebook-routed/scaffold.sh @@ -0,0 +1,96 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# gradebook + +Turns a class's scores (CSV text) into a ranked table with letter grades. + +Layout: + +- `gradebook/stats.py` numbers: mean, median, letter grades, summary +- `gradebook/report.py` CSV parsing and the text table +- `gradebook/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'gradebook' +cat > 'gradebook/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""gradebook: scores in, ranked report out.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for gradebook.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_gradebook.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from gradebook import cli, report, stats + + +class Stats(unittest.TestCase): + def test_mean(self): + self.assertEqual(stats.mean([1, 2, 3, 4]), 2.5) + + def test_median_odd_and_even(self): + self.assertEqual(stats.median([3, 1, 2]), 2) + self.assertEqual(stats.median([4, 1, 3, 2]), 2.5) + + def test_letter_grades(self): + self.assertEqual(stats.letter_grade(95), "A") + self.assertEqual(stats.letter_grade(85), "B") + self.assertEqual(stats.letter_grade(75), "C") + self.assertEqual(stats.letter_grade(65), "D") + self.assertEqual(stats.letter_grade(10), "F") + + def test_summarize(self): + summary = stats.summarize([50, 100]) + self.assertEqual(summary["count"], 2) + self.assertEqual(summary["mean"], 75) + self.assertEqual(summary["min"], 50) + self.assertEqual(summary["max"], 100) + + +class Report(unittest.TestCase): + def test_parse_rows(self): + rows = report.parse_rows("name,score\nAda,95\nBob,82.5\n") + self.assertEqual(rows, [("Ada", 95.0), ("Bob", 82.5)]) + + def test_render_has_header_rows_and_summary(self): + text = report.render([("Bob", 82.5), ("Ada", 95.0)]) + lines = text.split("\n") + self.assertEqual(lines[0], "Name | Score | Grade") + self.assertEqual(lines[1], "Ada | 95.0 | A") + self.assertEqual(lines[2], "Bob | 82.5 | B") + self.assertTrue(lines[3].startswith("Class: 2 students")) + + def test_render_nothing(self): + self.assertEqual(report.render([]), "No scores") + + +class Cli(unittest.TestCase): + def test_reads_stdin(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main([], io.StringIO("name,score\nAda,95\n"), out, err) + self.assertEqual(code, 0) + self.assertIn("Ada | 95.0 | A", out.getvalue()) + self.assertEqual(err.getvalue(), "") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/plugins/dojo-flow/evals/hf-semver-ranges-allopus/case.yaml b/plugins/dojo-flow/evals/hf-semver-ranges-allopus/case.yaml new file mode 100644 index 0000000..c718602 --- /dev/null +++ b/plugins/dojo-flow/evals/hf-semver-ranges-allopus/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hf-semver-ranges-allopus", + "description": "Two parallel tracks build version comparison (full Semantic Versioning precedence rules, many edge cases) and range checking on top of it. (every role on opus)", + "tags": [ + "held-out", + "routing", + "careful", + "allopus" + ], + "runs": 5, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow tool with name \"dojo-flow:build\"; if that name is not known, call it with scriptPath set to the plugin's workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not change the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to finish, then reply with its scorecard exactly as returned.\n\nargs (add repo yourself):\n{\n \"goal\": \"Build a small Python package `pkgcheck` (standard library only) that compares package versions and checks them against version ranges, with a command line front end. The directory `pkgcheck/` and its `__init__.py` already exist, and `tests/test_visible.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `semver` writes `pkgcheck/semver.py`; `constraints` writes `pkgcheck/constraints.py` and `pkgcheck/cli.py`. The shared interface is `semver.compare(a, b)` and `semver.parse_core(version)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.\",\n \"tracks\": [\n {\n \"name\": \"semver\",\n \"prompt\": \"Write the file `pkgcheck/semver.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/constraints.py` and `pkgcheck/cli.py` at the same time and will import your two public functions exactly as described here, so keep their names, arguments and return types.\\n\\nVersion strings look like `MAJOR.MINOR.PATCH`, optionally followed by a prerelease part introduced by `-` and/or a build part introduced by `+`: `1.4.2`, `2.0.0-rc.1`, `1.0.0+build.7`, `1.0.0-beta.2+exp.sha.5114f85`.\\n\\nHow to read a version string:\\n1. Split off the build part first: it starts at the first `+` and runs to the end. It is ignored completely, whatever it contains (even nothing, so `1.2.3+` is valid). A `-` inside the build part is not a prerelease marker: `1.0.0+build-5` has no prerelease.\\n2. In what remains, the core is the text before the first `-` (or all of it if there is no `-`). It must be exactly three dot-separated, non-empty strings made only of the ASCII digits 0-9 (`1.2.3` is fine; `1.2`, `1.2.3.4`, `v1.2.3`, `1.x.3`, `1..3` and the empty string are malformed).\\n3. If there is a `-`, everything after the first `-` (up to the build part) is the prerelease. It may itself contain `-` characters: `1.0.0-x-y` has the prerelease `x-y`. Split it on `.` into identifiers. No identifier may be empty, so `1.2.3-`, `1.2.3-a..b`, `1.2.3-.a` and `1.2.3-a.` are malformed.\\n4. A malformed string (including a value that is not a str) raises `ValueError`.\\n\\nProvide these two public functions:\\n- `parse_core(version)` -> `tuple[int, int, int]`: the `(major, minor, patch)` integers. It checks the whole string by the rules above and raises `ValueError` if it is malformed. Example: `parse_core(\\\"1.2.3-rc.1+b.5\\\") == (1, 2, 3)`.\\n- `compare(a, b)` -> int: returns `-1`, `0` or `1` when `a` has lower, equal or higher precedence than `b`. It raises `ValueError` if either string is malformed. Precedence works like this:\\n - Compare the cores as numbers, part by part from the left (`1.10.0` is higher than `1.9.0`, `2.0.0` is lower than `10.0.0`).\\n - If the cores are equal, a version without a prerelease is higher than one with a prerelease (`1.0.0` is higher than `1.0.0-rc.1`).\\n - If both have prereleases, compare the identifiers pairwise from left to right and stop at the first pair that differs. Two identifiers made only of digits compare as integers (`2` is lower than `10`). An identifier made only of digits is lower than any other identifier (`1` is lower than `a`, and `2` is lower than `1a`). Two identifiers that are not both all digits compare as plain strings by character code (`alpha` is lower than `beta`, `Beta` is lower than `alpha`, and `rc2` is higher than `rc10`).\\n - If every compared pair is equal, the prerelease with more identifiers is higher (`alpha` is lower than `alpha.1`).\\n - Build parts never matter.\\n\\nFor orientation, this list is in increasing order: `1.0.0-alpha`, `1.0.0-alpha.1`, `1.0.0-alpha.beta`, `1.0.0-beta`, `1.0.0-beta.2`, `1.0.0-beta.11`, `1.0.0-rc.1`, `1.0.0`. Equal versions compare as 0, and `compare(b, a)` is always the negative of `compare(a, b)`.\",\n \"files\": [\n \"pkgcheck/semver.py\"\n ]\n },\n {\n \"name\": \"constraints\",\n \"prompt\": \"Write the files `pkgcheck/constraints.py` and `pkgcheck/cli.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/semver.py` at the same time; do not write that file. Import it with `from pkgcheck import semver` and use only these two functions from it, which will behave as follows:\\n- `semver.compare(a, b)` -> int: `-1`, `0` or `1` as version string `a` has lower, equal or higher precedence than `b` (Semantic Versioning precedence; the build part after `+` is ignored; a prerelease version is lower than the same core without one). Raises `ValueError` if either string is malformed.\\n- `semver.parse_core(version)` -> `tuple[int, int, int]`: `(major, minor, patch)` as ints, for example `parse_core(\\\"1.2.3-rc.1+b.5\\\") == (1, 2, 3)`. Raises `ValueError` if the string is malformed.\\nDo not re-implement version comparison yourself.\\n\\n`pkgcheck/constraints.py` provides:\\n- `satisfies(version, spec)` -> bool (a real `bool`). `spec` is one or more clauses separated by commas, and the version satisfies the spec only if it satisfies every clause. Whitespace around a clause, and between its operator and its version, is ignored. A clause is an operator followed by a version string. The operators are `==`, `!=`, `>=`, `<=`, `>`, `<`, `^` and `~` (match the two-character ones before the one-character ones), and a clause with no operator means `==`. The version in a clause may carry a prerelease and/or build part.\\n Meaning, with `c = semver.compare(version, clause_version)`: `==` is `c == 0`, `!=` is `c != 0`, `>=` is `c >= 0`, `<=` is `c <= 0`, `>` is `c > 0`, `<` is `c < 0`.\\n `~X.Y.Z` (tilde) means: at least that version, and below `X.(Y+1).0`.\\n `^X.Y.Z` (caret) means: at least that version, and below an upper bound taken from the first non-zero part: if X > 0 the bound is `(X+1).0.0`; otherwise if Y > 0 it is `0.(Y+1).0`; otherwise it is `0.0.(Z+1)`. So `^1.2.3` is below `2.0.0`, `^0.2.3` is below `0.3.0` and `^0.0.3` is below `0.0.4`. Take X, Y, Z from `semver.parse_core` of the clause version (any prerelease or build part on it still counts for the \\\"at least\\\" test), and make both tests with `semver.compare`.\\n There is no special rule for prereleases: a version satisfies a bound exactly when `compare` says so, so `2.0.0-rc.1` does satisfy `^1.2.3` because it is below `2.0.0`. That is intended.\\n Raise `ValueError` for: an empty spec, an empty clause (`\\\">=1.0.0,\\\"`, `\\\">=1.0.0,,<2.0.0\\\"`), a clause with an unknown operator or no version (`\\\"=>1.0.0\\\"`, `\\\"*\\\"`, `\\\"~\\\"`), a malformed clause version (`\\\">=1.0\\\"`), or a malformed `version` argument. Every clause must be checked for errors before any clause is evaluated, so an invalid clause raises even when an earlier clause would already have failed.\\n- `newest(versions, spec)` -> str or None: `versions` is any iterable of version strings. Return the satisfying version with the highest precedence, as the original string, or `None` if none satisfies `spec`. When two satisfying versions have equal precedence (for example `1.0.0+a` and `1.0.0+b`) return the one that comes first. A malformed entry in `versions` raises `ValueError`.\\n\\n`pkgcheck/cli.py` provides:\\n- `main(argv=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. `argv[0]` is a spec and the rest are versions. With fewer than two items write `usage: python -m pkgcheck.cli SPEC VERSION [VERSION ...]` and a newline to stderr and return 2. Otherwise call `constraints.newest(argv[1:], argv[0])`: if it raises `ValueError`, write `error: ` and a newline to stderr and return 2; if it returns `None`, write `none` and a newline to stdout and return 1; otherwise write the version and a newline to stdout and return 0.\\n- End the file with `if __name__ == \\\"__main__\\\": raise SystemExit(main())`.\",\n \"files\": [\n \"pkgcheck/constraints.py\",\n \"pkgcheck/cli.py\"\n ]\n }\n ],\n \"gates\": [\n \"python3 -m unittest discover -s tests -q\",\n \"python3 -c \\\"from pkgcheck import semver, constraints, cli\\\"\"\n ],\n \"models\": {\n \"scout\": \"opus\",\n \"builder\": \"opus\",\n \"integrator\": \"opus\",\n \"reviewer\": \"opus\",\n \"checker\": \"opus\",\n \"judge\": \"opus\"\n }\n}", + "max_turns": 12, + "timeout_seconds": 1800, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Glob", + "Grep", + "Bash", + "Skill", + "Workflow" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: pkgcheck/semver.py implements the full precedence rules (numeric core, numeric versus alphanumeric prerelease identifiers, length tie-break, build metadata ignored, strict validation) and pkgcheck/constraints.py and pkgcheck/cli.py use it as specified; the gates pass. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "workflow-ran", + "tool": "Workflow", + "min": 1 + }, + { + "type": "regex", + "name": "scorecard-relayed", + "target": "last_message", + "pattern": "verdict: (VERIFIED|UNVERIFIED|REFUSED|PLAN ONLY)" + } + ], + "expected_outcome": "pkgcheck/semver.py implements the full precedence rules (numeric core, numeric versus alphanumeric prerelease identifiers, length tie-break, build metadata ignored, strict validation) and pkgcheck/constraints.py and pkgcheck/cli.py use it as specified; the gates pass." +} diff --git a/plugins/dojo-flow/evals/hf-semver-ranges-allopus/scaffold.sh b/plugins/dojo-flow/evals/hf-semver-ranges-allopus/scaffold.sh new file mode 100644 index 0000000..fd48e5a --- /dev/null +++ b/plugins/dojo-flow/evals/hf-semver-ranges-allopus/scaffold.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# pkgcheck + +Version precedence and version-range checks for package versions such as +`1.4.2`, `2.0.0-rc.1` or `1.0.0+build.7`. + +Layout: + +- `pkgcheck/semver.py` comparing two versions +- `pkgcheck/constraints.py` `satisfies(version, spec)` and `newest(versions, spec)` +- `pkgcheck/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'pkgcheck' +cat > 'pkgcheck/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""pkgcheck: compare versions and check them against ranges.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for pkgcheck.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_visible.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from pkgcheck import cli, constraints, semver + + +class Compare(unittest.TestCase): + def test_plain_ordering(self): + self.assertEqual(semver.compare("1.2.3", "1.2.4"), -1) + self.assertEqual(semver.compare("2.0.0", "1.9.9"), 1) + self.assertEqual(semver.compare("1.2.3", "1.2.3"), 0) + + def test_numbers_are_not_compared_as_text(self): + self.assertEqual(semver.compare("1.10.0", "1.9.0"), 1) + + def test_prerelease_is_lower_than_release(self): + self.assertEqual(semver.compare("1.0.0-alpha", "1.0.0"), -1) + self.assertEqual(semver.compare("1.0.0", "1.0.0-rc.1"), 1) + + def test_core(self): + self.assertEqual(semver.parse_core("3.4.5"), (3, 4, 5)) + + def test_malformed(self): + with self.assertRaises(ValueError): + semver.compare("1.2", "1.2.3") + + +class Satisfies(unittest.TestCase): + def test_comparison_operators(self): + self.assertTrue(constraints.satisfies("1.2.3", ">=1.2.0")) + self.assertFalse(constraints.satisfies("1.2.3", "<1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "==1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "!=1.2.4")) + + def test_caret_and_tilde(self): + self.assertTrue(constraints.satisfies("1.9.0", "^1.2.3")) + self.assertFalse(constraints.satisfies("2.0.0", "^1.2.3")) + self.assertTrue(constraints.satisfies("1.2.9", "~1.2.3")) + self.assertFalse(constraints.satisfies("1.3.0", "~1.2.3")) + + def test_several_clauses(self): + self.assertTrue(constraints.satisfies("1.4.0", ">=1.2.0, <1.5.0")) + self.assertFalse(constraints.satisfies("1.5.0", ">=1.2.0, <1.5.0")) + + def test_newest(self): + versions = ["1.0.0", "1.10.0", "1.9.0", "2.0.0"] + self.assertEqual(constraints.newest(versions, "^1.0.0"), "1.10.0") + self.assertIsNone(constraints.newest(versions, "<1.0.0")) + + +class Cli(unittest.TestCase): + def test_prints_the_newest_match(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main(["^1.0.0", "1.0.0", "1.1.0", "2.0.0"], out, err) + self.assertEqual(code, 0) + self.assertEqual(out.getvalue(), "1.1.0\n") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/plugins/dojo-flow/evals/hf-semver-ranges-routed/case.yaml b/plugins/dojo-flow/evals/hf-semver-ranges-routed/case.yaml new file mode 100644 index 0000000..395752b --- /dev/null +++ b/plugins/dojo-flow/evals/hf-semver-ranges-routed/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hf-semver-ranges-routed", + "description": "Two parallel tracks build version comparison (full Semantic Versioning precedence rules, many edge cases) and range checking on top of it. (routed: workflow defaults)", + "tags": [ + "held-out", + "routing", + "careful", + "routed" + ], + "runs": 5, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Run the dojo-flow build workflow on this repository with exactly the arguments below. Call the Workflow tool with name \"dojo-flow:build\"; if that name is not known, call it with scriptPath set to the plugin's workflows/build.js. Set repo to the absolute path of the current directory (run pwd to get it). Do not change the goal, tracks, gates or models, and do not edit any file yourself. Wait for the workflow to finish, then reply with its scorecard exactly as returned.\n\nargs (add repo yourself):\n{\n \"goal\": \"Build a small Python package `pkgcheck` (standard library only) that compares package versions and checks them against version ranges, with a command line front end. The directory `pkgcheck/` and its `__init__.py` already exist, and `tests/test_visible.py` already exists and must not be edited. Two tracks run at the same time and cannot talk to each other: `semver` writes `pkgcheck/semver.py`; `constraints` writes `pkgcheck/constraints.py` and `pkgcheck/cli.py`. The shared interface is `semver.compare(a, b)` and `semver.parse_core(version)`, described identically in both track prompts. At the end the test suite in `tests/` must pass.\",\n \"tracks\": [\n {\n \"name\": \"semver\",\n \"prompt\": \"Write the file `pkgcheck/semver.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only this one file and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/constraints.py` and `pkgcheck/cli.py` at the same time and will import your two public functions exactly as described here, so keep their names, arguments and return types.\\n\\nVersion strings look like `MAJOR.MINOR.PATCH`, optionally followed by a prerelease part introduced by `-` and/or a build part introduced by `+`: `1.4.2`, `2.0.0-rc.1`, `1.0.0+build.7`, `1.0.0-beta.2+exp.sha.5114f85`.\\n\\nHow to read a version string:\\n1. Split off the build part first: it starts at the first `+` and runs to the end. It is ignored completely, whatever it contains (even nothing, so `1.2.3+` is valid). A `-` inside the build part is not a prerelease marker: `1.0.0+build-5` has no prerelease.\\n2. In what remains, the core is the text before the first `-` (or all of it if there is no `-`). It must be exactly three dot-separated, non-empty strings made only of the ASCII digits 0-9 (`1.2.3` is fine; `1.2`, `1.2.3.4`, `v1.2.3`, `1.x.3`, `1..3` and the empty string are malformed).\\n3. If there is a `-`, everything after the first `-` (up to the build part) is the prerelease. It may itself contain `-` characters: `1.0.0-x-y` has the prerelease `x-y`. Split it on `.` into identifiers. No identifier may be empty, so `1.2.3-`, `1.2.3-a..b`, `1.2.3-.a` and `1.2.3-a.` are malformed.\\n4. A malformed string (including a value that is not a str) raises `ValueError`.\\n\\nProvide these two public functions:\\n- `parse_core(version)` -> `tuple[int, int, int]`: the `(major, minor, patch)` integers. It checks the whole string by the rules above and raises `ValueError` if it is malformed. Example: `parse_core(\\\"1.2.3-rc.1+b.5\\\") == (1, 2, 3)`.\\n- `compare(a, b)` -> int: returns `-1`, `0` or `1` when `a` has lower, equal or higher precedence than `b`. It raises `ValueError` if either string is malformed. Precedence works like this:\\n - Compare the cores as numbers, part by part from the left (`1.10.0` is higher than `1.9.0`, `2.0.0` is lower than `10.0.0`).\\n - If the cores are equal, a version without a prerelease is higher than one with a prerelease (`1.0.0` is higher than `1.0.0-rc.1`).\\n - If both have prereleases, compare the identifiers pairwise from left to right and stop at the first pair that differs. Two identifiers made only of digits compare as integers (`2` is lower than `10`). An identifier made only of digits is lower than any other identifier (`1` is lower than `a`, and `2` is lower than `1a`). Two identifiers that are not both all digits compare as plain strings by character code (`alpha` is lower than `beta`, `Beta` is lower than `alpha`, and `rc2` is higher than `rc10`).\\n - If every compared pair is equal, the prerelease with more identifiers is higher (`alpha` is lower than `alpha.1`).\\n - Build parts never matter.\\n\\nFor orientation, this list is in increasing order: `1.0.0-alpha`, `1.0.0-alpha.1`, `1.0.0-alpha.beta`, `1.0.0-beta`, `1.0.0-beta.2`, `1.0.0-beta.11`, `1.0.0-rc.1`, `1.0.0`. Equal versions compare as 0, and `compare(b, a)` is always the negative of `compare(a, b)`.\",\n \"files\": [\n \"pkgcheck/semver.py\"\n ]\n },\n {\n \"name\": \"constraints\",\n \"prompt\": \"Write the files `pkgcheck/constraints.py` and `pkgcheck/cli.py` (Python 3, standard library only). The package `pkgcheck` already exists with an `__init__.py`. Write only these two files and do not edit anything else, including `tests/`. Another builder is writing `pkgcheck/semver.py` at the same time; do not write that file. Import it with `from pkgcheck import semver` and use only these two functions from it, which will behave as follows:\\n- `semver.compare(a, b)` -> int: `-1`, `0` or `1` as version string `a` has lower, equal or higher precedence than `b` (Semantic Versioning precedence; the build part after `+` is ignored; a prerelease version is lower than the same core without one). Raises `ValueError` if either string is malformed.\\n- `semver.parse_core(version)` -> `tuple[int, int, int]`: `(major, minor, patch)` as ints, for example `parse_core(\\\"1.2.3-rc.1+b.5\\\") == (1, 2, 3)`. Raises `ValueError` if the string is malformed.\\nDo not re-implement version comparison yourself.\\n\\n`pkgcheck/constraints.py` provides:\\n- `satisfies(version, spec)` -> bool (a real `bool`). `spec` is one or more clauses separated by commas, and the version satisfies the spec only if it satisfies every clause. Whitespace around a clause, and between its operator and its version, is ignored. A clause is an operator followed by a version string. The operators are `==`, `!=`, `>=`, `<=`, `>`, `<`, `^` and `~` (match the two-character ones before the one-character ones), and a clause with no operator means `==`. The version in a clause may carry a prerelease and/or build part.\\n Meaning, with `c = semver.compare(version, clause_version)`: `==` is `c == 0`, `!=` is `c != 0`, `>=` is `c >= 0`, `<=` is `c <= 0`, `>` is `c > 0`, `<` is `c < 0`.\\n `~X.Y.Z` (tilde) means: at least that version, and below `X.(Y+1).0`.\\n `^X.Y.Z` (caret) means: at least that version, and below an upper bound taken from the first non-zero part: if X > 0 the bound is `(X+1).0.0`; otherwise if Y > 0 it is `0.(Y+1).0`; otherwise it is `0.0.(Z+1)`. So `^1.2.3` is below `2.0.0`, `^0.2.3` is below `0.3.0` and `^0.0.3` is below `0.0.4`. Take X, Y, Z from `semver.parse_core` of the clause version (any prerelease or build part on it still counts for the \\\"at least\\\" test), and make both tests with `semver.compare`.\\n There is no special rule for prereleases: a version satisfies a bound exactly when `compare` says so, so `2.0.0-rc.1` does satisfy `^1.2.3` because it is below `2.0.0`. That is intended.\\n Raise `ValueError` for: an empty spec, an empty clause (`\\\">=1.0.0,\\\"`, `\\\">=1.0.0,,<2.0.0\\\"`), a clause with an unknown operator or no version (`\\\"=>1.0.0\\\"`, `\\\"*\\\"`, `\\\"~\\\"`), a malformed clause version (`\\\">=1.0\\\"`), or a malformed `version` argument. Every clause must be checked for errors before any clause is evaluated, so an invalid clause raises even when an earlier clause would already have failed.\\n- `newest(versions, spec)` -> str or None: `versions` is any iterable of version strings. Return the satisfying version with the highest precedence, as the original string, or `None` if none satisfies `spec`. When two satisfying versions have equal precedence (for example `1.0.0+a` and `1.0.0+b`) return the one that comes first. A malformed entry in `versions` raises `ValueError`.\\n\\n`pkgcheck/cli.py` provides:\\n- `main(argv=None, stdout=None, stderr=None)` -> int. When an argument is None use `sys.argv[1:]`, `sys.stdout` and `sys.stderr`, looked up when `main` runs and not at import time. `argv[0]` is a spec and the rest are versions. With fewer than two items write `usage: python -m pkgcheck.cli SPEC VERSION [VERSION ...]` and a newline to stderr and return 2. Otherwise call `constraints.newest(argv[1:], argv[0])`: if it raises `ValueError`, write `error: ` and a newline to stderr and return 2; if it returns `None`, write `none` and a newline to stdout and return 1; otherwise write the version and a newline to stdout and return 0.\\n- End the file with `if __name__ == \\\"__main__\\\": raise SystemExit(main())`.\",\n \"files\": [\n \"pkgcheck/constraints.py\",\n \"pkgcheck/cli.py\"\n ]\n }\n ],\n \"gates\": [\n \"python3 -m unittest discover -s tests -q\",\n \"python3 -c \\\"from pkgcheck import semver, constraints, cli\\\"\"\n ],\n \"models\": {\n \"scout\": \"haiku\",\n \"builder\": \"sonnet\",\n \"integrator\": \"sonnet\",\n \"reviewer\": \"sonnet\",\n \"checker\": \"haiku\",\n \"judge\": \"opus\"\n }\n}", + "max_turns": 12, + "timeout_seconds": 1800, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Glob", + "Grep", + "Bash", + "Skill", + "Workflow" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: pkgcheck/semver.py implements the full precedence rules (numeric core, numeric versus alphanumeric prerelease identifiers, length tie-break, build metadata ignored, strict validation) and pkgcheck/constraints.py and pkgcheck/cli.py use it as specified; the gates pass. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "workflow-ran", + "tool": "Workflow", + "min": 1 + }, + { + "type": "regex", + "name": "scorecard-relayed", + "target": "last_message", + "pattern": "verdict: (VERIFIED|UNVERIFIED|REFUSED|PLAN ONLY)" + } + ], + "expected_outcome": "pkgcheck/semver.py implements the full precedence rules (numeric core, numeric versus alphanumeric prerelease identifiers, length tie-break, build metadata ignored, strict validation) and pkgcheck/constraints.py and pkgcheck/cli.py use it as specified; the gates pass." +} diff --git a/plugins/dojo-flow/evals/hf-semver-ranges-routed/scaffold.sh b/plugins/dojo-flow/evals/hf-semver-ranges-routed/scaffold.sh new file mode 100644 index 0000000..fd48e5a --- /dev/null +++ b/plugins/dojo-flow/evals/hf-semver-ranges-routed/scaffold.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > '.gitignore' <<'__SCAFFOLD_FILE_END__' +__pycache__/ +*.pyc +__SCAFFOLD_FILE_END__ + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# pkgcheck + +Version precedence and version-range checks for package versions such as +`1.4.2`, `2.0.0-rc.1` or `1.0.0+build.7`. + +Layout: + +- `pkgcheck/semver.py` comparing two versions +- `pkgcheck/constraints.py` `satisfies(version, spec)` and `newest(versions, spec)` +- `pkgcheck/cli.py` command line front end +- `tests/` the checks that must pass +__SCAFFOLD_FILE_END__ + +mkdir -p 'pkgcheck' +cat > 'pkgcheck/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""pkgcheck: compare versions and check them against ranges.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for pkgcheck.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_visible.py' <<'__SCAFFOLD_FILE_END__' +import io +import unittest + +from pkgcheck import cli, constraints, semver + + +class Compare(unittest.TestCase): + def test_plain_ordering(self): + self.assertEqual(semver.compare("1.2.3", "1.2.4"), -1) + self.assertEqual(semver.compare("2.0.0", "1.9.9"), 1) + self.assertEqual(semver.compare("1.2.3", "1.2.3"), 0) + + def test_numbers_are_not_compared_as_text(self): + self.assertEqual(semver.compare("1.10.0", "1.9.0"), 1) + + def test_prerelease_is_lower_than_release(self): + self.assertEqual(semver.compare("1.0.0-alpha", "1.0.0"), -1) + self.assertEqual(semver.compare("1.0.0", "1.0.0-rc.1"), 1) + + def test_core(self): + self.assertEqual(semver.parse_core("3.4.5"), (3, 4, 5)) + + def test_malformed(self): + with self.assertRaises(ValueError): + semver.compare("1.2", "1.2.3") + + +class Satisfies(unittest.TestCase): + def test_comparison_operators(self): + self.assertTrue(constraints.satisfies("1.2.3", ">=1.2.0")) + self.assertFalse(constraints.satisfies("1.2.3", "<1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "==1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "1.2.3")) + self.assertTrue(constraints.satisfies("1.2.3", "!=1.2.4")) + + def test_caret_and_tilde(self): + self.assertTrue(constraints.satisfies("1.9.0", "^1.2.3")) + self.assertFalse(constraints.satisfies("2.0.0", "^1.2.3")) + self.assertTrue(constraints.satisfies("1.2.9", "~1.2.3")) + self.assertFalse(constraints.satisfies("1.3.0", "~1.2.3")) + + def test_several_clauses(self): + self.assertTrue(constraints.satisfies("1.4.0", ">=1.2.0, <1.5.0")) + self.assertFalse(constraints.satisfies("1.5.0", ">=1.2.0, <1.5.0")) + + def test_newest(self): + versions = ["1.0.0", "1.10.0", "1.9.0", "2.0.0"] + self.assertEqual(constraints.newest(versions, "^1.0.0"), "1.10.0") + self.assertIsNone(constraints.newest(versions, "<1.0.0")) + + +class Cli(unittest.TestCase): + def test_prints_the_newest_match(self): + out, err = io.StringIO(), io.StringIO() + code = cli.main(["^1.0.0", "1.0.0", "1.1.0", "2.0.0"], out, err) + self.assertEqual(code, 0) + self.assertEqual(out.getvalue(), "1.1.0\n") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +git init -q +git add -A +git -c user.name=scaffold -c user.email=scaffold@example.invalid commit -qm scaffold diff --git a/plugins/dojo-router/README.md b/plugins/dojo-router/README.md index 85ad99c..897d7a4 100644 --- a/plugins/dojo-router/README.md +++ b/plugins/dojo-router/README.md @@ -148,7 +148,17 @@ Options reach the classic hooks as `CLAUDE_PLUGIN_OPTION_` and the mod thro execute `hooks.json` command hooks. The two halves are tested separately: a mod test shows that the rewritten input reaches the layer beneath, and a unit test shows the classic hook stays silent on that rewritten input. -- Nothing here measures what routing changes. There are no numbers in this plugin, and none are claimed. +- One pre-registered eval has measured what routing changes (below). No other number is claimed. + +## What has been measured + +In a pre-registered eval +([evals/routing-2026-10-23](https://github.com/DojoGenesis/plugins/blob/main/evals/routing-2026-10-23/PREREG-2026-10-23-routing.md)), +an Opus session asked to delegate lookups spent 0.62 as much with dojo-router installed as +without (95 percent interval 0.57 to 0.67; 60 runs, five tasks, no shell, Claude Code 2.1.305, +dojo-router 0.1.1), with every hidden test passing in both arms. The saving came from the session +following the router skill, not from the hooks. Harder tasks, sessions with a shell and sessions +that don't delegate were not measured. ## Tests @@ -157,6 +167,7 @@ cd plugins/dojo-router && python3 -m unittest discover -s tests -v CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1 claude plugin test plugins/dojo-router ``` -`evals/unpinned-dispatch-pinned/case.yaml` is a scaffold for `claude plugin eval`; it has not been run. +`evals/unpinned-dispatch-pinned/case.yaml` is a scaffold for `claude plugin eval`; it has not been run. The +`evals/hr-*` cases belong to the eval above and were run once. Tested with Claude Code 2.1.286. diff --git a/plugins/dojo-router/evals/hr-limiter-starve/case.yaml b/plugins/dojo-router/evals/hr-limiter-starve/case.yaml new file mode 100644 index 0000000..fbc7a22 --- /dev/null +++ b/plugins/dojo-router/evals/hr-limiter-starve/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hr-limiter-starve", + "description": "Subtle token-bucket bug: whole-token refill math throws away fractional progress whenever the bucket is looked at more often than once per token interval; naive fixes bank idle time.", + "tags": [ + "held-out", + "routing", + "careful" + ], + "runs": 6, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Our rate limiter has a nasty problem. With a policy of 2 requests per 4 seconds, a client that gets a 429 and keeps retrying every quarter of a second is never let back in, even after waiting far longer than the two seconds it should take for a slot to free up. A client that goes quiet for a few seconds and then tries once gets through fine, and people polling the remaining-quota status seem to cause the same trouble. The tests in tests/ all pass, so they don't cover it. Ask a subagent to review the token bucket in `ratelimit/bucket.py` carefully and work out what actually causes this (not a guess), then fix it so those clients get a slot back on time, without changing how the limiter behaves for clients that never hit the problem.", + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Bash", + "Glob", + "Grep", + "Agent", + "Skill" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: The bucket keeps fractional refill progress across frequent calls so retrying or polling clients regain a token on schedule, does not bank time spent full, and leaves all existing bucket and limiter behaviour unchanged. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "dispatched", + "tool": "Agent", + "min": 1 + }, + { + "type": "regex", + "name": "hook-warned", + "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", + "arm": "with-only" + } + ], + "expected_outcome": "The bucket keeps fractional refill progress across frequent calls so retrying or polling clients regain a token on schedule, does not bank time spent full, and leaves all existing bucket and limiter behaviour unchanged." +} diff --git a/plugins/dojo-router/evals/hr-limiter-starve/scaffold.sh b/plugins/dojo-router/evals/hr-limiter-starve/scaffold.sh new file mode 100644 index 0000000..51a97ba --- /dev/null +++ b/plugins/dojo-router/evals/hr-limiter-starve/scaffold.sh @@ -0,0 +1,252 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# ratelimit + +A small per-client rate limiter. + +- `ratelimit/policy.py` `Policy(limit, window)` +- `ratelimit/bucket.py` `TokenBucket` +- `ratelimit/limiter.py` `Limiter`: one bucket per client +- `ratelimit/middleware.py` `guard(limiter, client_id, handler)` +- `ratelimit/clock.py` `SystemClock` and a `ManualClock` for tests +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""ratelimit: per-client token bucket limiter.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/bucket.py' <<'__SCAFFOLD_FILE_END__' +"""Token bucket. + +The bucket holds up to `capacity` tokens and gains `rate` tokens per second. +A request takes one token, or is refused when the bucket is empty. +""" + + +class TokenBucket: + def __init__(self, capacity, rate, clock): + self.capacity = capacity + self.rate = rate + self.clock = clock + self.tokens = capacity # whole tokens currently in the bucket + self.last = clock.now() # when we last looked at the clock + + def _refill(self): + now = self.clock.now() + elapsed = now - self.last + self.last = now + self.tokens = min(self.capacity, self.tokens + int(elapsed * self.rate)) + + def available(self): + """Whole tokens that could be taken right now.""" + self._refill() + return self.tokens + + def allow(self, amount=1): + """Take `amount` tokens if they are there. Returns True when taken.""" + self._refill() + if self.tokens >= amount: + self.tokens -= amount + return True + return False +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/clock.py' <<'__SCAFFOLD_FILE_END__' +"""Clocks. The limiter only ever asks a clock for the current time in seconds.""" + +import time + + +class SystemClock: + def now(self): + return time.monotonic() + + +class ManualClock: + """A clock that only moves when told to (used by the tests).""" + + def __init__(self, start=0.0): + self._now = float(start) + + def now(self): + return self._now + + def advance(self, seconds): + self._now += seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/limiter.py' <<'__SCAFFOLD_FILE_END__' +"""One bucket per client, all following the same policy.""" + +from ratelimit.bucket import TokenBucket + + +class Limiter: + def __init__(self, policy, clock): + self.policy = policy + self.clock = clock + self._buckets = {} + + def _bucket(self, client_id): + bucket = self._buckets.get(client_id) + if bucket is None: + bucket = TokenBucket(self.policy.limit, self.policy.rate, self.clock) + self._buckets[client_id] = bucket + return bucket + + def allow(self, client_id): + return self._bucket(client_id).allow() + + def status(self, client_id): + """What a quota dashboard shows for this client.""" + return { + "remaining": self._bucket(client_id).available(), + "limit": self.policy.limit, + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/middleware.py' <<'__SCAFFOLD_FILE_END__' +"""Glue between the limiter and a request handler.""" + + +def guard(limiter, client_id, handler): + """Run `handler()` unless the client is over its limit. + + Returns (status_code, body). + """ + if not limiter.allow(client_id): + return 429, "slow down" + return 200, handler() +__SCAFFOLD_FILE_END__ + +mkdir -p 'ratelimit' +cat > 'ratelimit/policy.py' <<'__SCAFFOLD_FILE_END__' +"""How many requests a client may make in a time window.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Policy: + limit: int # requests allowed per window (also the burst size) + window: float # seconds + + @property + def rate(self): + """Tokens gained per second.""" + return self.limit / self.window +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for ratelimit.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_bucket.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from ratelimit.bucket import TokenBucket +from ratelimit.clock import ManualClock + + +def make(capacity=2, rate=0.5): + clock = ManualClock() + return clock, TokenBucket(capacity, rate, clock) + + +class BucketBasics(unittest.TestCase): + def test_starts_full(self): + _, bucket = make() + self.assertEqual(bucket.available(), 2) + + def test_burst_then_refused(self): + _, bucket = make() + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_long_wait_refills_to_capacity(self): + clock, bucket = make() + bucket.allow() + bucket.allow() + clock.advance(100) + self.assertEqual(bucket.available(), 2) + + def test_never_holds_more_than_capacity(self): + clock, bucket = make() + clock.advance(1000) + self.assertEqual(bucket.available(), 2) + self.assertTrue(bucket.allow()) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_one_token_after_two_seconds(self): + clock, bucket = make() + bucket.allow() + bucket.allow() + clock.advance(2.0) + self.assertTrue(bucket.allow()) + self.assertFalse(bucket.allow()) + + def test_asking_for_several_tokens(self): + clock, bucket = make(capacity=5, rate=1.0) + self.assertTrue(bucket.allow(3)) + self.assertFalse(bucket.allow(3)) + clock.advance(1.0) + self.assertTrue(bucket.allow(3)) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_limiter.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from ratelimit.clock import ManualClock +from ratelimit.limiter import Limiter +from ratelimit.middleware import guard +from ratelimit.policy import Policy + + +class LimiterBasics(unittest.TestCase): + def setUp(self): + self.clock = ManualClock() + self.limiter = Limiter(Policy(limit=2, window=4.0), self.clock) + + def test_clients_are_independent(self): + self.assertTrue(self.limiter.allow("a")) + self.assertTrue(self.limiter.allow("a")) + self.assertFalse(self.limiter.allow("a")) + self.assertTrue(self.limiter.allow("b")) + + def test_status_shows_remaining(self): + self.limiter.allow("a") + status = self.limiter.status("a") + self.assertEqual(status, {"remaining": 1, "limit": 2}) + + def test_guard_returns_429_when_over(self): + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (200, "hi")) + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (200, "hi")) + self.assertEqual(guard(self.limiter, "a", lambda: "hi"), (429, "slow down")) + + def test_waiting_quietly_lets_a_client_back_in(self): + self.limiter.allow("a") + self.limiter.allow("a") + self.clock.advance(2.0) + self.assertTrue(self.limiter.allow("a")) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/plugins/dojo-router/evals/hr-note-cache/case.yaml b/plugins/dojo-router/evals/hr-note-cache/case.yaml new file mode 100644 index 0000000..b8aedf1 --- /dev/null +++ b/plugins/dojo-router/evals/hr-note-cache/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hr-note-cache", + "description": "Call-graph lookup: find every function that ends up writing to the store (some indirectly, some in other modules) and add the missing cache invalidation.", + "tags": [ + "held-out", + "routing", + "standard" + ], + "runs": 6, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "In the notes package (`notesvc`), reads of a note go through a cache, and some of the code that changes notes doesn't invalidate it, so people see stale data after an edit. Use subagents to find every function in the package that writes to the store (anything that ends up calling `store.save` or `store.delete`, directly or through another function) and work out which of them already drop that note's cache entry. Then fix the ones that don't: invalidate just that note's entry, and keep the cache working for everything else.", + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Bash", + "Glob", + "Grep", + "Agent", + "Skill" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: edit_body, archive/unarchive, delete_note, tags.add_tag and importer.import_notes (and restore, through delete_note) all invalidate only the affected note's cache entry so reads are fresh, while other cached notes and normal read caching are untouched. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "dispatched", + "tool": "Agent", + "min": 1 + }, + { + "type": "regex", + "name": "hook-warned", + "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", + "arm": "with-only" + } + ], + "expected_outcome": "edit_body, archive/unarchive, delete_note, tags.add_tag and importer.import_notes (and restore, through delete_note) all invalidate only the affected note's cache entry so reads are fresh, while other cached notes and normal read caching are untouched." +} diff --git a/plugins/dojo-router/evals/hr-note-cache/scaffold.sh b/plugins/dojo-router/evals/hr-note-cache/scaffold.sh new file mode 100644 index 0000000..f3f0ab5 --- /dev/null +++ b/plugins/dojo-router/evals/hr-note-cache/scaffold.sh @@ -0,0 +1,329 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +mkdir -p 'notesvc' +cat > 'notesvc/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""notesvc: a small note service with an in-memory store and a read cache.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/backup.py' <<'__SCAFFOLD_FILE_END__' +"""Snapshot and restore the whole note set.""" + + +def snapshot(service): + return service.store.all() + + +def restore(service, notes): + """Make the store hold exactly `notes`.""" + wanted = {n.id for n in notes} + for existing in service.store.all(): + if existing.id not in wanted: + service.delete_note(existing.id) + for note in notes: + service._write(note) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/cache.py' <<'__SCAFFOLD_FILE_END__' +"""A read cache keyed by note id.""" + +import copy + + +class ReadCache: + def __init__(self): + self._items = {} + self.hits = 0 + self.misses = 0 + + def get(self, key): + if key in self._items: + self.hits += 1 + return copy.deepcopy(self._items[key]) + self.misses += 1 + return None + + def put(self, key, value): + self._items[key] = copy.deepcopy(value) + + def invalidate(self, key): + self._items.pop(key, None) + + def clear(self): + self._items.clear() + + def __contains__(self, key): + return key in self._items +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/importer.py' <<'__SCAFFOLD_FILE_END__' +"""Bulk import from plain dictionaries.""" + +from notesvc.note import Note + + +def import_notes(service, rows): + """Create or replace one note per row. Returns how many rows were imported. + + Each row looks like {"id": "n7", "title": "...", "body": "...", "tags": [...]}. + """ + count = 0 + for row in rows: + note = Note( + id=row["id"], + title=row["title"], + body=row.get("body", ""), + tags=list(row.get("tags", [])), + ) + service.store.save(note) + count += 1 + return count +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/note.py' <<'__SCAFFOLD_FILE_END__' +"""The Note record.""" + +from dataclasses import dataclass, field + + +@dataclass +class Note: + id: str + title: str + body: str = "" + tags: list = field(default_factory=list) + archived: bool = False +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/report.py' <<'__SCAFFOLD_FILE_END__' +"""Read-only summary numbers.""" + + +def summary(service): + notes = service.store.all() + return { + "total": len(notes), + "archived": sum(1 for n in notes if n.archived), + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/search.py' <<'__SCAFFOLD_FILE_END__' +"""Read-only search helpers.""" + + +def find_by_text(service, text): + text = text.lower() + return sorted( + n.id + for n in service.store.all() + if text in n.title.lower() or text in n.body.lower() + ) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/service.py' <<'__SCAFFOLD_FILE_END__' +"""The note service: reads go through the cache, writes go to the store.""" + +from notesvc.note import Note + + +class NoteService: + def __init__(self, store, cache): + self.store = store + self.cache = cache + self._next = 1 + + # ---- reads ------------------------------------------------------- + + def get_note(self, note_id): + cached = self.cache.get(note_id) + if cached is not None: + return cached + note = self.store.get(note_id) + if note is not None: + self.cache.put(note_id, note) + return note + + def _load(self, note_id): + """Read straight from the store (used before changing a note).""" + note = self.store.get(note_id) + if note is None: + raise KeyError(note_id) + return note + + # ---- writes ------------------------------------------------------ + + def _write(self, note): + self.store.save(note) + self.cache.invalidate(note.id) + + def create_note(self, title, body="", tags=()): + while self.store.exists(f"n{self._next}"): + self._next += 1 + note = Note(id=f"n{self._next}", title=title, body=body, tags=list(tags)) + self._next += 1 + self._write(note) + return note.id + + def rename_note(self, note_id, title): + note = self._load(note_id) + note.title = title + self._write(note) + + def edit_body(self, note_id, body): + note = self._load(note_id) + note.body = body + self.store.save(note) + + def archive_note(self, note_id): + self._set_flag(note_id, "archived", True) + + def unarchive_note(self, note_id): + self._set_flag(note_id, "archived", False) + + def _set_flag(self, note_id, name, value): + note = self._load(note_id) + setattr(note, name, value) + self.store.save(note) + + def delete_note(self, note_id): + self.store.delete(note_id) +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/store.py' <<'__SCAFFOLD_FILE_END__' +"""In-memory storage. Keeps its own copies so callers cannot change stored notes.""" + +import copy + + +class Store: + def __init__(self): + self._notes = {} + self.reads = 0 # number of get() calls, handy for checking cache hits + + def save(self, note): + self._notes[note.id] = copy.deepcopy(note) + + def get(self, note_id): + self.reads += 1 + note = self._notes.get(note_id) + return copy.deepcopy(note) if note is not None else None + + def exists(self, note_id): + return note_id in self._notes + + def delete(self, note_id): + self._notes.pop(note_id, None) + + def all(self): + return [copy.deepcopy(n) for n in self._notes.values()] +__SCAFFOLD_FILE_END__ + +mkdir -p 'notesvc' +cat > 'notesvc/tags.py' <<'__SCAFFOLD_FILE_END__' +"""Tag helpers.""" + + +def add_tag(service, note_id, tag): + note = service._load(note_id) + if tag not in note.tags: + note.tags.append(tag) + service.store.save(note) + + +def notes_with_tag(service, tag): + return sorted(n.id for n in service.store.all() if tag in n.tags) +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for notesvc.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_service.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from notesvc import backup, importer, report, search, tags +from notesvc.cache import ReadCache +from notesvc.service import NoteService +from notesvc.store import Store + + +def make(): + store = Store() + cache = ReadCache() + return store, cache, NoteService(store, cache) + + +class Basics(unittest.TestCase): + def test_create_and_get(self): + _, _, svc = make() + note_id = svc.create_note("Groceries", "milk", tags=["home"]) + note = svc.get_note(note_id) + self.assertEqual((note.title, note.body, note.tags), ("Groceries", "milk", ["home"])) + + def test_missing_note_is_none(self): + _, _, svc = make() + self.assertIsNone(svc.get_note("nope")) + + def test_second_read_comes_from_the_cache(self): + store, _, svc = make() + note_id = svc.create_note("A") + before = store.reads + svc.get_note(note_id) + svc.get_note(note_id) + self.assertEqual(store.reads - before, 1) + + def test_rename_is_visible_after_a_read(self): + _, _, svc = make() + note_id = svc.create_note("Old") + svc.get_note(note_id) + svc.rename_note(note_id, "New") + self.assertEqual(svc.get_note(note_id).title, "New") + + def test_edit_of_a_missing_note_raises(self): + _, _, svc = make() + with self.assertRaises(KeyError): + svc.edit_body("nope", "x") + + +class Helpers(unittest.TestCase): + def test_search_and_report(self): + _, _, svc = make() + a = svc.create_note("Red apples", "fruit") + svc.create_note("Blue", "sky") + svc.archive_note(a) + self.assertEqual(search.find_by_text(svc, "apple"), [a]) + self.assertEqual(report.summary(svc), {"total": 2, "archived": 1}) + + def test_tags_and_import(self): + _, _, svc = make() + a = svc.create_note("A") + tags.add_tag(svc, a, "x") + self.assertEqual(tags.notes_with_tag(svc, "x"), [a]) + self.assertEqual(importer.import_notes(svc, [{"id": "n9", "title": "Imported"}]), 1) + self.assertEqual(svc.get_note("n9").title, "Imported") + + def test_snapshot_and_restore_round_trip(self): + _, _, svc = make() + a = svc.create_note("A", "one") + snap = backup.snapshot(svc) + extra = svc.create_note("Extra") + backup.restore(svc, snap) + self.assertIsNotNone(svc.get_note(a)) + self.assertEqual(report.summary(svc)["total"], 1) + self.assertNotEqual(extra, a) + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/plugins/dojo-router/evals/hr-page-size/case.yaml b/plugins/dojo-router/evals/hr-page-size/case.yaml new file mode 100644 index 0000000..60ef3f2 --- /dev/null +++ b/plugins/dojo-router/evals/hr-page-size/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hr-page-size", + "description": "Where a value is defined and used: change the default page size and cap, notice the hard-coded 20 in the renderer, and leave the unrelated printed-sheet 20 alone.", + "tags": [ + "held-out", + "routing", + "standard" + ], + "runs": 6, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "The product listing returns 20 products per page by default and lets callers ask for up to 50 with `page_size`. I want the default to be 50, and callers should be able to ask for up to 100 (anything bigger is cut down to 100; zero, negative or non-numeric values fall back to the default). Use subagents first to find where the page size is defined and every place that relies on it or assumes the old number, and to pick out anything that just happens to use the number 20 for something else, because that should stay as it is. Then make the change.", + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Bash", + "Glob", + "Grep", + "Agent", + "Skill" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: The default listing page size is 50 with a cap of 100, the 'Showing a-b of n' text follows the actual page size, and the printed-sheet export still uses 20 per sheet. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "dispatched", + "tool": "Agent", + "min": 1 + }, + { + "type": "regex", + "name": "hook-warned", + "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", + "arm": "with-only" + } + ], + "expected_outcome": "The default listing page size is 50 with a cap of 100, the 'Showing a-b of n' text follows the actual page size, and the printed-sheet export still uses 20 per sheet." +} diff --git a/plugins/dojo-router/evals/hr-page-size/scaffold.sh b/plugins/dojo-router/evals/hr-page-size/scaffold.sh new file mode 100644 index 0000000..692fc58 --- /dev/null +++ b/plugins/dojo-router/evals/hr-page-size/scaffold.sh @@ -0,0 +1,153 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# shop + +Small product catalogue. + +- `shop/config.py` settings +- `shop/catalog.py` the products +- `shop/paging.py` page arithmetic +- `shop/api.py` `list_products(query)` returns items plus paging info +- `shop/render.py` text rendering of a listing +- `shop/web.py` `page(query_string)` ties api and render together +- `shop/export_pdf.py` printed sheets +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""shop: a tiny product catalogue with paged listings.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/api.py' <<'__SCAFFOLD_FILE_END__' +"""The listing endpoint, as a plain function.""" + +from shop import catalog, paging + + +def list_products(query=None): + """`query` holds the raw (string) URL parameters: category, page, page_size.""" + query = query or {} + items = catalog.all_products() + category = query.get("category") + if category: + items = [p for p in items if p["category"] == category] + result = paging.paginate(items, query.get("page"), query.get("page_size")) + return { + "items": result["items"], + "meta": {key: result[key] for key in ("page", "page_size", "total", "pages")}, + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/catalog.py' <<'__SCAFFOLD_FILE_END__' +"""The product data (generated, 130 products).""" + +_PRODUCTS = [ + {"id": i, "name": f"Product {i:03d}", "category": "tools" if i % 2 else "garden"} + for i in range(1, 131) +] + + +def all_products(): + return [dict(p) for p in _PRODUCTS] +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/config.py' <<'__SCAFFOLD_FILE_END__' +"""Settings for the shop code.""" + +# Product listings (API and web pages) +DEFAULT_PAGE_SIZE = 20 +MAX_PAGE_SIZE = 50 + +# Rows that fit on one printed catalogue sheet +PDF_PAGE_SIZE = 20 +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/export_pdf.py' <<'__SCAFFOLD_FILE_END__' +"""Split the catalogue into printed sheets.""" + +from shop import config + + +def paginate_for_pdf(items): + size = config.PDF_PAGE_SIZE + return [items[i:i + size] for i in range(0, len(items), size)] +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/paging.py' <<'__SCAFFOLD_FILE_END__' +"""Page arithmetic shared by the listing code.""" + +from shop import config + + +def clamp_page_size(value): + """Turn a requested page size (any type) into a usable one.""" + try: + size = int(value) + except (TypeError, ValueError): + return config.DEFAULT_PAGE_SIZE + if size < 1: + return config.DEFAULT_PAGE_SIZE + return min(size, config.MAX_PAGE_SIZE) + + +def clamp_page(value): + try: + page = int(value) + except (TypeError, ValueError): + return 1 + return max(page, 1) + + +def paginate(items, page=None, page_size=None): + size = clamp_page_size(page_size) + page = clamp_page(page) + total = len(items) + start = (page - 1) * size + return { + "items": items[start:start + size], + "page": page, + "page_size": size, + "total": total, + "pages": -(-total // size), + } +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/render.py' <<'__SCAFFOLD_FILE_END__' +"""Plain-text rendering of a listing result.""" + + +def render_listing(result): + meta = result["meta"] + total = meta["total"] + if not result["items"]: + header = f"Showing 0-0 of {total}" + else: + start = (meta["page"] - 1) * 20 + 1 + end = min(meta["page"] * 20, total) + header = f"Showing {start}-{end} of {total}" + lines = [header] + [f"- {p['name']}" for p in result["items"]] + return "\n".join(lines) +__SCAFFOLD_FILE_END__ + +mkdir -p 'shop' +cat > 'shop/web.py' <<'__SCAFFOLD_FILE_END__' +"""The web page: query string in, text out.""" + +from urllib.parse import parse_qsl + +from shop import api, render + + +def page(query_string=""): + query = dict(parse_qsl(query_string)) + return render.render_listing(api.list_products(query)) +__SCAFFOLD_FILE_END__ diff --git a/plugins/dojo-router/evals/hr-reminder-every/case.yaml b/plugins/dojo-router/evals/hr-reminder-every/case.yaml new file mode 100644 index 0000000..2beaba4 --- /dev/null +++ b/plugins/dojo-router/evals/hr-reminder-every/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hr-reminder-every", + "description": "Which module handles each reminder kind: the recurring handler has its own broken unit table and rejects compound intervals; visible tests fail until fixed.", + "tags": [ + "held-out", + "routing", + "standard" + ], + "runs": 6, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "Reminders written like `stretch every 2h` go off every two minutes instead of every two hours. Please use subagents to find which module handles each kind of reminder line (`in`, `at` and `every`) and where each one turns the time text into seconds, then fix the `every` case. There are tests in tests/; run them when you're done.", + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Bash", + "Glob", + "Grep", + "Agent", + "Skill" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: Every-reminders convert hours and days correctly and also accept compound intervals such as 1h30m (for example by reusing the shared duration parser), zero or malformed intervals still raise ValueError, in/at reminders are unchanged, and the tests in tests/ pass. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "dispatched", + "tool": "Agent", + "min": 1 + }, + { + "type": "regex", + "name": "hook-warned", + "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", + "arm": "with-only" + } + ], + "expected_outcome": "Every-reminders convert hours and days correctly and also accept compound intervals such as 1h30m (for example by reusing the shared duration parser), zero or malformed intervals still raise ValueError, in/at reminders are unchanged, and the tests in tests/ pass." +} diff --git a/plugins/dojo-router/evals/hr-reminder-every/scaffold.sh b/plugins/dojo-router/evals/hr-reminder-every/scaffold.sh new file mode 100644 index 0000000..d273d7d --- /dev/null +++ b/plugins/dojo-router/evals/hr-reminder-every/scaffold.sh @@ -0,0 +1,207 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +mkdir -p 'remind' +cat > 'remind/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""remind: parse one-line reminders such as 'stretch every 2h'.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/absolute.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' at HH:MM' (24 hour clock).""" + +DAY = 24 * 3600 + + +def parse(rest, now=0): + """`rest` is 'HH:MM'. `now` is seconds since midnight. + + Returns the number of seconds until the next occurrence of that time. + """ + hours, sep, minutes = rest.partition(":") + if not sep or not hours.isdigit() or not minutes.isdigit(): + raise ValueError(f"bad time of day: {rest!r}") + hh, mm = int(hours), int(minutes) + if hh > 23 or mm > 59: + raise ValueError(f"bad time of day: {rest!r}") + target = hh * 3600 + mm * 60 + return (target - now) % DAY +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/cli.py' <<'__SCAFFOLD_FILE_END__' +"""Tiny command line front end: prints one line per reminder.""" + +from remind.parser import parse_line + + +def describe(reminder): + if reminder.kind == "every": + return f"{reminder.text}: every {reminder.seconds}s" + return f"{reminder.text}: fires in {reminder.seconds}s" + + +def main(lines, now=0): + for line in lines: + print(describe(parse_line(line, now))) + return 0 +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/durations.py' <<'__SCAFFOLD_FILE_END__' +"""Shared duration parsing: '90s', '2h', '1h30m' -> whole seconds.""" + +import re + +UNIT_SECONDS = {"s": 1, "m": 60, "h": 3600, "d": 86400} +_CHUNK = re.compile(r"(\d+)([smhd])") + + +def parse_duration(text): + text = text.strip().lower() + if not text: + raise ValueError("empty duration") + total = 0 + pos = 0 + for match in _CHUNK.finditer(text): + if match.start() != pos: + raise ValueError(f"bad duration: {text!r}") + total += int(match.group(1)) * UNIT_SECONDS[match.group(2)] + pos = match.end() + if pos != len(text): + raise ValueError(f"bad duration: {text!r}") + return total +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/parser.py' <<'__SCAFFOLD_FILE_END__' +"""Split a reminder line and hand the time part to the right module.""" + +import re + +from remind import absolute, recurring, relative +from remind.reminder import Reminder + +_LINE = re.compile(r"^(?P.+?)\s+(?Pin|at|every)\s+(?P\S+)$") + + +def parse_line(line, now=0): + """Parse ' in|at|every ' into a Reminder. + + `now` (seconds since midnight) is only used for 'at' reminders. + """ + match = _LINE.match(line.strip()) + if not match: + raise ValueError(f"cannot parse reminder: {line!r}") + kind = match.group("kind") + rest = match.group("rest") + if kind == "in": + seconds = relative.parse(rest) + elif kind == "at": + seconds = absolute.parse(rest, now) + else: + seconds = recurring.parse(rest) + return Reminder(match.group("text"), kind, seconds) +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/recurring.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' every '.""" + +import re + +_UNITS = {"s": 1, "m": 60, "h": 60, "d": 24 * 60} +_SINGLE = re.compile(r"(\d+)([smhd])") + + +def _to_seconds(text): + match = _SINGLE.fullmatch(text.strip().lower()) + if not match: + raise ValueError(f"bad interval: {text!r}") + return int(match.group(1)) * _UNITS[match.group(2)] + + +def parse(rest): + """`rest` is the text after the word 'every'. Returns the interval in seconds.""" + seconds = _to_seconds(rest) + if seconds <= 0: + raise ValueError("interval must be positive") + return seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/relative.py' <<'__SCAFFOLD_FILE_END__' +"""Handles lines of the form ' in '.""" + +from remind.durations import parse_duration + + +def parse(rest): + """`rest` is the text after the word 'in'. Returns the delay in seconds.""" + seconds = parse_duration(rest) + if seconds <= 0: + raise ValueError("delay must be positive") + return seconds +__SCAFFOLD_FILE_END__ + +mkdir -p 'remind' +cat > 'remind/reminder.py' <<'__SCAFFOLD_FILE_END__' +"""The parsed form of a reminder line.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Reminder: + text: str + kind: str # "in", "at" or "every" + seconds: int # delay (in/at) or interval (every) +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""Tests for remind.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'tests' +cat > 'tests/test_parser.py' <<'__SCAFFOLD_FILE_END__' +import unittest + +from remind.parser import parse_line + + +class RelativeAndAbsolute(unittest.TestCase): + def test_in_minutes(self): + self.assertEqual(parse_line("call mom in 10m").seconds, 600) + + def test_in_compound(self): + self.assertEqual(parse_line("standup in 1h30m").seconds, 5400) + + def test_at_later_today(self): + self.assertEqual(parse_line("lunch at 12:00", now=0).seconds, 12 * 3600) + + def test_at_wraps_to_tomorrow(self): + self.assertEqual(parse_line("lunch at 12:00", now=13 * 3600).seconds, 23 * 3600) + + +class Every(unittest.TestCase): + def test_every_seconds(self): + reminder = parse_line("ping every 45s") + self.assertEqual(reminder.kind, "every") + self.assertEqual(reminder.seconds, 45) + + def test_every_hours(self): + self.assertEqual(parse_line("stretch every 2h").seconds, 7200) + + def test_every_compound(self): + self.assertEqual(parse_line("water plants every 1h30m").seconds, 5400) + + def test_every_zero_is_rejected(self): + with self.assertRaises(ValueError): + parse_line("nothing every 0s") + + +if __name__ == "__main__": + unittest.main() +__SCAFFOLD_FILE_END__ diff --git a/plugins/dojo-router/evals/hr-retry-limit/case.yaml b/plugins/dojo-router/evals/hr-retry-limit/case.yaml new file mode 100644 index 0000000..c6e429f --- /dev/null +++ b/plugins/dojo-router/evals/hr-retry-limit/case.yaml @@ -0,0 +1,58 @@ +# Held-out case for the pre-registered routing eval (evals/routing-2026-10-23/PREREG-2026-10-23-routing.md). +# The task was written by an independent model author from held-out-brief.md, without access to this +# repository; its hidden tests live in evals/routing-2026-10-23/cases/, never in the sandbox. The graders +# are the evaluator's, fixed by the pre-registration. JSON is valid YAML. Runs spend money: never run +# without a cap and a yes. +{ + "schema_version": "1.0", + "name": "hr-retry-limit", + "description": "Setting trace: a retry count of 0 is dropped in three different files of a small fetch client.", + "tags": [ + "held-out", + "routing", + "standard" + ], + "runs": 6, + "context": { + "scaffold_script": "scaffold.sh", + "add_dirs": [] + }, + "execution": { + "prompt": "I can't turn retries off in our little fetch client (the fetchkit package). `--retries 0` on the command line, or `max_retries` set to 0 in code, still ends up making four attempts when the server keeps failing; I want 0 to mean one attempt and then give up. Please use subagents to trace where the retry count is read and set across the package, starting from the CLI flag and from the Client, and report every place that touches it. Then fix it so 0 really means no retries, without changing what happens when no retry count is given.", + "max_turns": 16, + "timeout_seconds": 900, + "model": "opus", + "allowed_tools": [ + "Read", + "Edit", + "Write", + "Bash", + "Glob", + "Grep", + "Agent", + "Skill" + ] + }, + "graders": [ + { + "type": "llm", + "name": "quality-judge", + "focus": "files", + "criteria": "You are grading the final state of a small Python project after a coding task. Read the project's files. A correct finish: A retry count of 0 (from the --retries flag, from settings overrides, from the FETCHKIT_MAX_RETRIES environment variable, or passed to the client directly) results in exactly one attempt, while the default of three retries and other explicit counts behave as before. Decide whether the code now does what that sentence describes, judging behaviour, not style or comments. PASS if it does. FAIL if it does not, or only partly does." + }, + { + "type": "tool_used", + "name": "dispatched", + "tool": "Agent", + "min": 1 + }, + { + "type": "regex", + "name": "hook-warned", + "target": "trace", + "pattern": "dojo-router: this .* dispatch names no model", + "arm": "with-only" + } + ], + "expected_outcome": "A retry count of 0 (from the --retries flag, from settings overrides, from the FETCHKIT_MAX_RETRIES environment variable, or passed to the client directly) results in exactly one attempt, while the default of three retries and other explicit counts behave as before." +} diff --git a/plugins/dojo-router/evals/hr-retry-limit/scaffold.sh b/plugins/dojo-router/evals/hr-retry-limit/scaffold.sh new file mode 100644 index 0000000..db8c44a --- /dev/null +++ b/plugins/dojo-router/evals/hr-retry-limit/scaffold.sh @@ -0,0 +1,181 @@ +#!/bin/sh +# Writes the project files into the current directory. +set -e + +cat > 'README.md' <<'__SCAFFOLD_FILE_END__' +# fetchkit + +A tiny fetch client with retries. The transport is a stand-in that fails a set +number of times, so everything can be exercised without a network. + +- `fetchkit/defaults.py` built-in defaults +- `fetchkit/settings.py` merges defaults, environment (`FETCHKIT_*`) and overrides +- `fetchkit/client.py` `Client.fetch(url)` with retry and backoff +- `fetchkit/transport.py` `FlakyTransport` +- `fetchkit/cli.py` `run(argv, transport)` +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/__init__.py' <<'__SCAFFOLD_FILE_END__' +"""fetchkit: a tiny fetch client with retries, driven by a fake transport.""" +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/cli.py' <<'__SCAFFOLD_FILE_END__' +"""Command line entry point: `fetchkit URL [--retries N] [--timeout S]`.""" + +import argparse + +from fetchkit.client import Client +from fetchkit.errors import RetriesExhausted +from fetchkit.settings import load_settings + + +def parse_args(argv): + parser = argparse.ArgumentParser(prog="fetchkit") + parser.add_argument("url") + parser.add_argument( + "--retries", type=int, default=None, help="extra attempts after the first one" + ) + parser.add_argument("--timeout", type=float, default=None) + return parser.parse_args(argv) + + +def run(argv, transport, env=None, sleep=lambda seconds: None): + """Run one fetch. Returns the process exit code.""" + args = parse_args(argv) + overrides = {} + if args.retries: + overrides["max_retries"] = args.retries + if args.timeout is not None: + overrides["timeout"] = args.timeout + settings = load_settings(overrides, env) + client = Client(transport, settings, sleep=sleep) + try: + print(client.fetch(args.url)) + except RetriesExhausted as exc: + print(f"error: {exc}") + return 1 + return 0 +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/client.py' <<'__SCAFFOLD_FILE_END__' +"""The client: send a request, retry on transport errors.""" + +import time + +from fetchkit.defaults import DEFAULTS +from fetchkit.errors import RetriesExhausted, TransportError +from fetchkit.settings import load_settings + + +class Client: + def __init__(self, transport, settings=None, sleep=time.sleep): + self.transport = transport + self.settings = settings if settings is not None else load_settings() + self.sleep = sleep + # number of extra attempts allowed after the first one + self.max_retries = self.settings.get("max_retries") or DEFAULTS["max_retries"] + + def fetch(self, url): + attempts = 0 + while True: + attempts += 1 + try: + return self.transport.send(url, timeout=self.settings["timeout"]) + except TransportError as exc: + if attempts > self.max_retries: + raise RetriesExhausted(url, attempts) from exc + self.sleep(self.settings["backoff"] * attempts) +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/defaults.py' <<'__SCAFFOLD_FILE_END__' +"""Built-in defaults for the fetch client.""" + +DEFAULTS = { + "timeout": 5.0, + "max_retries": 3, + "backoff": 0.5, +} +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/errors.py' <<'__SCAFFOLD_FILE_END__' +"""Exceptions raised by fetchkit.""" + + +class TransportError(Exception): + """One attempt to reach the server failed.""" + + +class RetriesExhausted(Exception): + """Every allowed attempt failed.""" + + def __init__(self, url, attempts): + super().__init__(f"gave up on {url} after {attempts} attempt(s)") + self.url = url + self.attempts = attempts +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/settings.py' <<'__SCAFFOLD_FILE_END__' +"""Build the effective settings from defaults, environment and explicit overrides.""" + +from fetchkit.defaults import DEFAULTS + +ENV_PREFIX = "FETCHKIT_" + + +def _from_env(env): + found = {} + if not env: + return found + raw = env.get(ENV_PREFIX + "MAX_RETRIES") + if raw not in (None, ""): + found["max_retries"] = int(raw) + raw = env.get(ENV_PREFIX + "TIMEOUT") + if raw not in (None, ""): + found["timeout"] = float(raw) + raw = env.get(ENV_PREFIX + "BACKOFF") + if raw not in (None, ""): + found["backoff"] = float(raw) + return found + + +def load_settings(overrides=None, env=None): + """Return the effective settings. + + Explicit overrides win over the environment, which wins over the defaults. + """ + merged = dict(DEFAULTS) + merged.update(_from_env(env)) + if overrides: + for key in ("timeout", "backoff"): + if overrides.get(key) is not None: + merged[key] = overrides[key] + merged["max_retries"] = overrides.get("max_retries") or merged["max_retries"] + return merged +__SCAFFOLD_FILE_END__ + +mkdir -p 'fetchkit' +cat > 'fetchkit/transport.py' <<'__SCAFFOLD_FILE_END__' +"""A fake transport used by the tests and the demo CLI.""" + +from fetchkit.errors import TransportError + + +class FlakyTransport: + """Fails the first `fail_times` calls, then succeeds.""" + + def __init__(self, fail_times=0): + self.fail_times = fail_times + self.calls = 0 + + def send(self, url, timeout=None): + self.calls += 1 + if self.calls <= self.fail_times: + raise TransportError(f"simulated failure {self.calls}") + return "ok:" + url +__SCAFFOLD_FILE_END__