Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,15 @@ All notable changes to SkillOpt are documented here. This project adheres to
- A flat `--cfg-options` key for an `env` setting (for example `workers=4`)
now overrides the YAML value with a structured config instead of being
silently replaced by it (#409).
- In the default force-write slow-update mode
(`optimizer.slow_update_gate_with_selection: false`), the skill that receives
the epoch-end guidance is now measured once on the selection split, and later
step gates compare candidates against that score. Before, the current score
stayed at the pre-injection value, and the selection cache stored that gate
score with a soft score of `0.0` for the new skill, so a byte-identical later
candidate reused both numbers. This adds one selection rollout per epoch
boundary in force-write mode (none when the exact skill text is already
cached). The validation-best skill and gated mode are unchanged (#404).
- Close four ways invalid evidence could reach the SkillOpt-Sleep gate: recall
no longer brings a content twin of tonight's held-out task into training,
a measured tool route no longer credits a self-reported `TOOL_CALL:` marker,
Expand Down
11 changes: 11 additions & 0 deletions docs/guide/training-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,17 @@ improved/regressed/persistent-fail/stable-success, then generates high-level
the selection split or applied unconditionally. Its purpose is to counter
cross-epoch forgetting.

The paper's numbers and the released `ckpt/` skills used the gated mode
(`slow_update_gate_with_selection: true`): the guidance is a candidate that is
scored on the selection split and accepted or rejected like a step edit. The
current default (`false`) writes the guidance into the current skill without a
gate, then measures that skill once on the selection split, so later step gates
compare candidates against a score it actually achieved. That costs one
selection rollout per epoch boundary (none when the exact skill text was
already scored). The injection itself never replaces the validation-best
skill; only the final-skill validation at the end of training (run when
`evaluation.eval_test` is on) can promote the final skill.

### Meta Skill

A **meta-skill memory** accumulates high-level strategy notes across the training
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ forwarded consistently to both SDK and CLI execution paths.
| `optimizer.skill_update_mode` | str | `patch` | `patch`, `rewrite_from_suggestions`, or `full_rewrite_minibatch` |
| `optimizer.use_slow_update` | bool | `true` | Epoch-boundary longitudinal update |
| `optimizer.slow_update_samples` | int | `20` | Longitudinal evaluation samples |
| `optimizer.slow_update_gate_with_selection` | bool | `false` | Gate slow-update guidance on the selection split |
| `optimizer.slow_update_gate_with_selection` | bool | `false` | Gate slow-update guidance on the selection split (paper setting: `true`); `false` writes it unconditionally and re-measures the current skill with one selection rollout per epoch boundary |
| `optimizer.longitudinal_pair_policy` | str | `mixed` | `mixed`, `changed`, or `unchanged` |
| `optimizer.use_meta_skill` | bool | `true` | Cross-epoch optimizer memory |
| `optimizer.use_skill_aware_reflection` | bool | `false` | Enable skill-defect vs execution-lapse routing |
Expand Down
67 changes: 63 additions & 4 deletions skillopt/engine/trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -1809,6 +1809,18 @@ def _persist_runtime_state(last_completed_step: int) -> None:
current_skill = replace_slow_update_field(
current_skill, slow_saved["slow_update_content"],
)
# Resuming exactly at this boundary: take the
# injected skill's measured score from
# slow_result.json (no rollout). runtime_state.json
# normally holds the same value, but it is written
# after slow_result.json (an interruption in between
# leaves it stale), and a history-only resume only
# has the pre-injection score.
if (
global_step == resume_from - 1
and slow_saved.get("current_score") is not None
):
current_score = float(slow_saved["current_score"])
elif epoch == 1:
# Epoch 1: inject empty placeholder
os.makedirs(slow_dir, exist_ok=True)
Expand Down Expand Up @@ -2049,10 +2061,56 @@ def _persist_runtime_state(last_completed_step: int) -> None:
current_skill = replace_slow_update_field(
current_skill, slow_content,
)
# Update caches so downstream steps use the
# slow-update-injected skill for hashing.
# The injection is not gated, but later step gates
# compare against current_score, so measure the
# injected skill once on the selection set (or
# reuse a real cache entry) instead of keeping the
# pre-injection score. best_* stay untouched; the
# final-skill validation after training handles
# promotion.
slow_candidate_hash = skill_hash(current_skill)
sel_cache[slow_candidate_hash] = (current_score, 0.0)
if slow_candidate_hash in sel_cache:
slow_sel_hard, slow_sel_soft = sel_cache[
slow_candidate_hash
]
print(
f" [slow update] selection cache hit: "
f"hard={slow_sel_hard:.4f}"
)
else:
sel_env, sel_n = _build_eval_env(
split="valid_seen",
env_num=cfg["sel_env_num"],
seed=seed,
)
print(
f" [slow update] re-measuring injected "
f"skill, selection items={sel_n}"
)
slow_eval_dir = os.path.join(
slow_dir, "selection_eval_injected",
)
slow_eval_results = adapter.rollout(
sel_env, current_skill, slow_eval_dir,
)
slow_sel_hard, slow_sel_soft = compute_score(
slow_eval_results
)
sel_cache[slow_candidate_hash] = (
slow_sel_hard, slow_sel_soft,
)
prev_current = current_score
current_score = select_gate_score(
slow_sel_hard, slow_sel_soft,
gate_metric, gate_mixed_weight,
skill_content=current_skill,
use_semantic_density=use_semantic_density,
semantic_density_weight=semantic_density_weight,
leading_words=leading_words,
)
slow_result["selection_hard"] = slow_sel_hard
slow_result["selection_soft"] = slow_sel_soft
slow_result["current_score"] = current_score

slow_result["action"] = "force_accept"
current_origin = f"slow_update_epoch_{epoch:02d}"
Expand All @@ -2061,7 +2119,8 @@ def _persist_runtime_state(last_completed_step: int) -> None:
f" [slow update] force-injected into "
f"current only "
f"({len(slow_content)} chars), "
f"{slow_time}s"
f"{slow_time}s; current score "
f"{prev_current:.4f} -> {current_score:.4f}"
)
else:
slow_result = slow_result or {}
Expand Down
Loading
Loading