Skip to content

fix(trainer): re-measure the force-written slow-update skill (#404) - #422

Merged
Yifan Yang (Yif-Yang) merged 1 commit into
mainfrom
maintenance/slow-update-remeasure
Nov 17, 2026
Merged

Yifan Yang (Yif-Yang) merged 1 commit into
mainfrom
maintenance/slow-update-remeasure

Conversation

@Yif-Yang

Copy link
Copy Markdown
Contributor

Fixes the stale-score bug reported in #404.

Problem. In the default force-write slow-update mode (optimizer.slow_update_gate_with_selection: false), the epoch-end guidance is injected into current_skill without validation, but:

  • current_score keeps the pre-injection value, so later step gates compare candidates against a score the injected skill never achieved;
  • the selection cache stores the new skill as (current_score, 0.0), which is a gate score where a (hard, soft) pair belongs. A byte-identical later candidate would then reuse those fake numbers, which is wrong under soft/mixed gate metrics.

Fix.

  • Re-measure after injection. The injected skill is measured once on the selection split (or a real cache entry is reused). The real (hard, soft) is cached, and current_score is set with the same select_gate_score(...) call the baseline uses. slow_result.json records selection_hard, selection_soft and current_score.
  • Resume. Resuming exactly at that boundary restores the score from slow_result.json, with no rollout.
  • Unchanged. best_skill, best_score, best_step and best_origin are untouched (as since ffe5810), so the final-skill validation and promotion work as before. Gated mode is unchanged.
  • Cost. One selection rollout per epoch boundary in force-write mode (none when the exact skill text is already cached), the same as gated mode.

Docs. training-loop.md and the config table now state that the paper's numbers and the released ckpt/ skills used slow_update_gate_with_selection: true.

Tests. tests/test_trainer_slow_update_score.py drives the real train() loop with a mock adapter and no model calls:

  • force-write with and without use_gate;
  • the cache contents;
  • resume with no extra rollout;
  • gated mode (reject and accept-new-best).

On current main, the 5 force-write, cache and resume tests fail, and the 2 gated tests pass, which confirms gated mode is unchanged. With the fix, the full suite, the strict docs build and ruff all pass.

The code was written with Claude Code and reviewed and verified by the maintainer.

In force-write slow-update mode (optimizer.slow_update_gate_with_selection:
false, the default), the epoch-end guidance was injected into current_skill
without validation, but current_score kept the pre-injection value and the
selection cache stored the new skill as (current_score, 0.0): a gate score
where a (hard, soft) pair belongs. Later step gates compared candidates
against a score the injected skill never achieved, and a byte-identical
later candidate reused the fake numbers.

Measure the injected skill once on the selection split (or reuse a real
cache entry), store its real (hard, soft), and set current_score with the
same select_gate_score call as the baseline. slow_result.json records
selection_hard, selection_soft and current_score. best_skill, best_score,
best_step and best_origin are untouched, so final validation and promotion
work as before. Gated mode is unchanged.

On resume at exactly that epoch boundary, current_score is restored from
slow_result.json without a rollout, covering an interruption between that
write and runtime_state.json, and the history-only resume path.

Cost: one selection rollout per epoch boundary in force-write mode, none
when the exact skill text is already cached.

Fixes #404

Co-authored-by: Claude <noreply@anthropic.com>
@Yif-Yang
Yifan Yang (Yif-Yang) merged commit 22c1ed8 into main Nov 17, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant