You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Slow update: paper describes a dev-gated update, but the default since 4a1b984 force-writes it. Why the change? #404
Hi,
I have a question about how the epoch-end slow update is accepted. The paper and the current default code seem to disagree, and I'd like to understand why.
What the paper says
§3.6, Appendix C.3, and Algorithm 1 (line 33) describe the slow-update candidate as going through the same validation (selection-set) gate as step-level edits. It is kept only if it passes.
What the code does
In v0.1.0 (244e346), the slow update is gated. The candidate is scored on the selection set and passed to evaluate_gate() against current_score / best_score.
4a1b984 ("refactor: rename teacher/student …") includes "Fix slow update gate (force write into skill)". After this commit, the guidance is written into the skill unconditionally.
00602df re-adds gating as an opt-in switch, optimizer.slow_update_gate_with_selection. It defaults to false (force-injected), described as "unchanged behavior".
ffe5810 then limits the forced write to the current skill and keeps best unpolluted.
So with default settings, the slow update is never validated. Also, current_score is not re-measured after the injection: the selection cache keeps the pre-injection score for the new skill hash. In effect, the next step's gate compares against a stale score.
Questions
What was the reason for switching from gated to force-write in 4a1b984? For example, was the gate rejecting slow updates too often because the selection set is small and noisy?
Which variant produced the numbers reported in the paper: gated, as in v0.1.0, or force-written?
Which setting do you recommend for reproducing the paper: slow_update_gate_with_selection: true or false?
If force-write is intended, should current_score be re-evaluated after the injection? As it stands, step-level gates after an epoch boundary compare against a score that the current skill never achieved.
Hi Yanan (@yanan1116), thanks for the careful reading. Your description of the history is accurate. Here is what we can say, with sources:
1. Which variant produced the paper numbers? The gated one. The released skills in ckpt/ were produced with the gated slow-update semantics of §3.6 (optimizer.slow_update_gate_with_selection: true). ckpt/README.md states this and notes that re-training with the current default can produce a different best_skill.md. Our technical blog also calls this setting the "Paper-Style Fully Gated Baseline" (hard validation gate for step edits, validation-selected slow updates) and says explicitly that it is not the current main default.
2. Which setting reproduces the paper?optimizer.slow_update_gate_with_selection: true, together with the default hard gate (evaluation.use_gate: true).
3. Why does main default to force-write? It is a post-submission change, made for exploration rather than for paper reproduction. The slow update is epoch-level, longitudinal guidance. On a small, noisy selection set, a single gated comparison often rejected it, even though its purpose (countering cross-epoch forgetting) only shows up over later steps. In the blog's ablation sweep, policies that keep slow updates always on are competitive with or better than the fully gated baseline for stronger targets, but not universally: for gpt-5.4-nano on SearchQA, the fully gated baseline remains a recommended setting. Those are exploratory, descriptive aggregates rather than repeated-seed results, and the blog says so. ffe5810 later kept best unpolluted (the force-written content goes into current only), and the run ends with a validation pass on the final skill, which is promoted to best only if it beats the incumbent. Selection and test reporting therefore never rely on an unvalidated slow update.
4. The stale current_score after injection. You're right, and it's a real issue in force-write mode. After the injection, the trainer records the new skill hash in the selection cache with the pre-injection score (sel_cache[slow_candidate_hash] = (current_score, 0.0) in skillopt/engine/trainer.py). Step-level gates after an epoch boundary therefore compare against a score the current skill never achieved, and the cached soft score of 0.0 is also wrong for soft/mixed gate metrics. This is now fixed in #422. In force-write mode, the injected skill is measured once on the selection set (or a real cache entry is reused), the real (hard, soft) is cached, and current_score is set from it, so later step gates compare against a score the current skill actually achieved. best is still never touched by the injection. Gated mode is unchanged: the new tests for gated mode pass on the old code too. The cost is one selection rollout per epoch boundary, the same as gated mode.
The same PR updates docs/guide/training-loop.md and docs/reference/config.md to say that the paper's numbers and the released ckpt/ skills used slow_update_gate_with_selection: true.
Activity
Yif-Yang commentedon Nov 17, 2026
Hi Yanan (@yanan1116), thanks for the careful reading. Your description of the history is accurate. Here is what we can say, with sources:
1. Which variant produced the paper numbers? The gated one. The released skills in
ckpt/were produced with the gated slow-update semantics of §3.6 (optimizer.slow_update_gate_with_selection: true).ckpt/README.mdstates this and notes that re-training with the current default can produce a differentbest_skill.md. Our technical blog also calls this setting the "Paper-Style Fully Gated Baseline" (hard validation gate for step edits, validation-selected slow updates) and says explicitly that it is not the currentmaindefault.2. Which setting reproduces the paper?
optimizer.slow_update_gate_with_selection: true, together with the default hard gate (evaluation.use_gate: true).3. Why does
maindefault to force-write? It is a post-submission change, made for exploration rather than for paper reproduction. The slow update is epoch-level, longitudinal guidance. On a small, noisy selection set, a single gated comparison often rejected it, even though its purpose (countering cross-epoch forgetting) only shows up over later steps. In the blog's ablation sweep, policies that keep slow updates always on are competitive with or better than the fully gated baseline for stronger targets, but not universally: forgpt-5.4-nanoon SearchQA, the fully gated baseline remains a recommended setting. Those are exploratory, descriptive aggregates rather than repeated-seed results, and the blog says so.ffe5810later keptbestunpolluted (the force-written content goes intocurrentonly), and the run ends with a validation pass on the final skill, which is promoted tobestonly if it beats the incumbent. Selection and test reporting therefore never rely on an unvalidated slow update.4. The stale
current_scoreafter injection. You're right, and it's a real issue in force-write mode. After the injection, the trainer records the new skill hash in the selection cache with the pre-injection score (sel_cache[slow_candidate_hash] = (current_score, 0.0)inskillopt/engine/trainer.py). Step-level gates after an epoch boundary therefore compare against a score the current skill never achieved, and the cached soft score of0.0is also wrong forsoft/mixedgate metrics. This is now fixed in #422. In force-write mode, the injected skill is measured once on the selection set (or a real cache entry is reused), the real(hard, soft)is cached, andcurrent_scoreis set from it, so later step gates compare against a score the current skill actually achieved.bestis still never touched by the injection. Gated mode is unchanged: the new tests for gated mode pass on the old code too. The cost is one selection rollout per epoch boundary, the same as gated mode.The same PR updates
docs/guide/training-loop.mdanddocs/reference/config.mdto say that the paper's numbers and the releasedckpt/skills usedslow_update_gate_with_selection: true.fix(trainer): re-measure the force-written slow-update skill (microso…