feat(runtime): structuralRollout strategy + rollout-policy improve surface (measured +8..+21pp)#503
Merged
Merged
Conversation
…f-k + self-repair on docstring examples) Selection + repair grounded only on visible docstring >>> examples (doctest, nonce-sentinel), hidden check() grading physically after all harness decisions. Paired bootstrap + exact sign test. Calibrated: 164/164 hidden self-check, 75/164 coverage, 2/75 false-fail (dataset bugs). First result: Llama-3-8B 44.3%->57.3% pass@1 (+13.0pp, CI [+7.8,+18.4], p=1.5e-5, n=164). humaneval.ts: HUMANEVAL_GZ local-cache support (GitHub raw 429s under reps).
…ructural-lever rig Model writes single-line asserts from the prompt alone, BEFORE any candidate exists; asserts run individually in the jail and add to the doctest signal (split counts kept for audit). Extends honest-oracle coverage from 46% of HumanEval toward 100%. Opt-in via TESTGEN=<count>; default off keeps prior runs reproducible.
…all-k-samples-fail bucket
…EN), hidden=remaining asserts Same architecture as hev-structural (nonce judges, phase separation, incremental persistence, bootstrap+sign-test). Calibration: 427/427 hidden self-check, 0/427 visible false-fail, 0 dropped. Official-first scoring: MBPP one-sentence specs make model-guessed asserts ~70% wrong on passing code — shown assert ranks first, guesses break ties (pilot: selection -6.7pp -> +16.7pp).
… only net-new seam; extend strategy family, verifier-environment, field routing
…select + guarded repair (measured +8..+21pp)
…ull result (hev-eval, hev-improve)
…lRollout dials under the held-out gate
…tool-loop conversation runBrainLoop returned on a no-tool-call turn WITHOUT pushing the final assistant message, so ToolLoopResult.messages violated its own contract (seed + every assistant/tool turn): a shot that answers without tools came back with a transcript missing its own answer. Depth continuation then criticized a solution absent from the model's history, and structuralRollout's fenced-code candidate extraction (defaultExtractCandidate's designed fallback) read an empty conversation on non-tool-calling models (Llama-3-8B-Instruct-Lite never tool-calls under tool_choice:auto — measured on Together). Proven by the runtime smoke (bench/src/smoke-structural-rollout.mts): candidates now flow through the strategy on a real non-tool-calling model.
…ory-less consults consultAnalyst rendered EVERY raw consult in analyst framing (instruction as system, 'TASK: ...TRAJECTORY:' as the user turn). With no trajectory — the pre-task case, i.e. structuralRollout's authored-check generation — that user turn is just the coding task, which a weak model answers directly, ignoring the system instruction: measured on Llama-3-8B-Instruct-Lite, authored-assert yield 6/18 reps (0 asserts on the rest) vs 18/18 with the instruction and task fused into one user message (the proven hev-structural rig shape). Trajectory-bearing consults keep the analyst framing unchanged. Runtime smoke after the fix: authored checks on 20/20 HumanEval tasks (was 14/20).
…Eval, hidden grading script-side The ship gate for the ported strategy: runAgentic + structuralRollout(default policy: k=5, repairs<=2, testgen=6) over createVerifierEnvironment with an INERT check, so no hidden-test signal reaches selection/repair; visible checks are the strategy's own model-authored asserts run by the shipped sandboxCheckRunner over a docker --network=none exec channel (the script's thin adapter). The task's own check() suite grades every locked candidate script-side (runChecker) after each rollout finishes. Receipts are hard-verified against the recorded outcomes per candidate. Llama-3-8B-Instruct-Lite (Together), first 20 tasks: blind mean-of-k 63.0% -> selected 80.0% -> final 85.0% (+22pp), authored checks 20/20, 4 tasks rescued (selection or guarded repair passes hidden where sample 0 fails), 0 crashes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
structuralRolloutstrategy — a new member of the Strategy family: sample k candidates, self-check each against visible checks, select the best, then guarded repair for candidates that fail. The design doc maps the integration:CheckSourceis the only net-new seam; everything else extends the existing strategy family, verifier-environment, and field routing.improve()can tune the structuralRollout dials (k, repair budget, testgen on/off) under the held-out promotion gate.Evidence
docs/results/structural-lever-humaneval.md.Design doc
docs/design/structural-rollout-integration.md🤖 Generated with Claude Code