Skip to content

Slow update: paper describes a dev-gated update, but the default since 4a1b984 force-writes it. Why the change? #404

Closed

Description

@yanan1116

Hi,
I have a question about how the epoch-end slow update is accepted. The paper and the current default code seem to disagree, and I'd like to understand why.

What the paper says

§3.6, Appendix C.3, and Algorithm 1 (line 33) describe the slow-update candidate as going through the same validation (selection-set) gate as step-level edits. It is kept only if it passes.

What the code does

  • In v0.1.0 (244e346), the slow update is gated. The candidate is scored on the selection set and passed to evaluate_gate() against current_score / best_score.
  • 4a1b984 ("refactor: rename teacher/student …") includes "Fix slow update gate (force write into skill)". After this commit, the guidance is written into the skill unconditionally.
  • 00602df re-adds gating as an opt-in switch, optimizer.slow_update_gate_with_selection. It defaults to false (force-injected), described as "unchanged behavior".
  • ffe5810 then limits the forced write to the current skill and keeps best unpolluted.

So with default settings, the slow update is never validated. Also, current_score is not re-measured after the injection: the selection cache keeps the pre-injection score for the new skill hash. In effect, the next step's gate compares against a stale score.

Questions

  1. What was the reason for switching from gated to force-write in 4a1b984? For example, was the gate rejecting slow updates too often because the selection set is small and noisy?
  2. Which variant produced the numbers reported in the paper: gated, as in v0.1.0, or force-written?
  3. Which setting do you recommend for reproducing the paper: slow_update_gate_with_selection: true or false?
  4. If force-write is intended, should current_score be re-evaluated after the injection? As it stands, step-level gates after an epoch boundary compare against a score that the current skill never achieved.

Thanks!

Activity

  1. Yif-Yang commented on Nov 17, 2026

    @Yif-Yang
    Contributor

    Hi Yanan (@yanan1116), thanks for the careful reading. Your description of the history is accurate. Here is what we can say, with sources:

    1. Which variant produced the paper numbers? The gated one. The released skills in ckpt/ were produced with the gated slow-update semantics of §3.6 (optimizer.slow_update_gate_with_selection: true). ckpt/README.md states this and notes that re-training with the current default can produce a different best_skill.md. Our technical blog also calls this setting the "Paper-Style Fully Gated Baseline" (hard validation gate for step edits, validation-selected slow updates) and says explicitly that it is not the current main default.

    2. Which setting reproduces the paper? optimizer.slow_update_gate_with_selection: true, together with the default hard gate (evaluation.use_gate: true).

    3. Why does main default to force-write? It is a post-submission change, made for exploration rather than for paper reproduction. The slow update is epoch-level, longitudinal guidance. On a small, noisy selection set, a single gated comparison often rejected it, even though its purpose (countering cross-epoch forgetting) only shows up over later steps. In the blog's ablation sweep, policies that keep slow updates always on are competitive with or better than the fully gated baseline for stronger targets, but not universally: for gpt-5.4-nano on SearchQA, the fully gated baseline remains a recommended setting. Those are exploratory, descriptive aggregates rather than repeated-seed results, and the blog says so. ffe5810 later kept best unpolluted (the force-written content goes into current only), and the run ends with a validation pass on the final skill, which is promoted to best only if it beats the incumbent. Selection and test reporting therefore never rely on an unvalidated slow update.

    4. The stale current_score after injection. You're right, and it's a real issue in force-write mode. After the injection, the trainer records the new skill hash in the selection cache with the pre-injection score (sel_cache[slow_candidate_hash] = (current_score, 0.0) in skillopt/engine/trainer.py). Step-level gates after an epoch boundary therefore compare against a score the current skill never achieved, and the cached soft score of 0.0 is also wrong for soft/mixed gate metrics. This is now fixed in #422. In force-write mode, the injected skill is measured once on the selection set (or a real cache entry is reused), the real (hard, soft) is cached, and current_score is set from it, so later step gates compare against a score the current skill actually achieved. best is still never touched by the injection. Gated mode is unchanged: the new tests for gated mode pass on the old code too. The cost is one selection rollout per epoch boundary, the same as gated mode.

    The same PR updates docs/guide/training-loop.md and docs/reference/config.md to say that the paper's numbers and the released ckpt/ skills used slow_update_gate_with_selection: true.

  2. added a commit that references this issue on Nov 17, 2026
    22c1ed8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions