Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@ All notable changes to SkillOpt are documented here. This project adheres to
decisions and accepted edits are unchanged even when the token budget binds.
A single-edit candidate needs no extra replay; an N-edit candidate costs up
to N validation replays, which are included in `tokens used` and checked
against `max_tokens_per_night` one replay at a time. Follow-up to #174.
against `max_tokens_per_night` one replay at a time. An ablation replay that
fails leaves its rows `unmeasured` and is logged as an
`edit_ablation_failed` event instead of failing the night, and each ablation
event names its fan-out group (`skill`), `night` and `trial_index`.
Follow-up to #174.
- **OfficeQA Parallel Search MCP** as an opt-in `env.search_mode=parallel`
provider for `openai_chat`/`qwen_chat`. The default stays `offline` and no
config selects Parallel automatically. Selecting it sends the generated
Expand Down
15 changes: 15 additions & 0 deletions docs/sleep/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,21 @@ the key has no effect. Rows are included in `report.md`, `report.json`,
`diagnostics.json`, the CLI's `--json` output, and the evidence log; in
multi-skill fan-out, each group's rows are in the evidence log.

An ablation replay that fails does not fail the night, because every decision
is already final when it runs. Its row is `unmeasured` with the error in its
note, the ablation replays still to come in that consolidation do not start
and say so in their notes, and a `gate` / `edit_ablation_failed` event is
written to the evidence log. The night is staged as it would be with the key
off. Errors that pause or stop the whole cycle (pending handoff calls, the
Cursor backend's fail-closed errors, staging errors) still do.

Ablation events are written after every decision, so in multi-skill fan-out
they sit together at the end of the evidence log instead of next to their
group's other events. Each `edit_ablation` and `edit_ablation_failed` event
therefore carries `skill` (the fan-out group's skill name, empty for the
night's aggregate run), `night`, and `trial_index` (the position of its trial
among that run's gate trials).

Rule-based heading checks are intentionally additive. `section_present` keeps
its legacy strict behavior for existing task sets: an ATX heading cannot append
text after the requested name. Use `section_contains` when a numbered,
Expand Down
132 changes: 94 additions & 38 deletions skillopt_sleep/consolidate.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@
from dataclasses import dataclass, field
from typing import Callable, List, Optional, Tuple

from skillopt_sleep.backend import Backend
from skillopt_sleep.backend import Backend, CursorBackendError

# Self-contained validation gate (vendored from SkillOpt; zero dependency on the
# research package, so this open-source tool stays decoupled from the paper code).
from skillopt_sleep.gate import evaluate_gate, select_gate_score
from skillopt_sleep.handoff_backend import PendingCalls
from skillopt_sleep.memory import apply_edits_detailed
from skillopt_sleep.replay import aggregate_scores, replay_batch
from skillopt_sleep.staging import StagingError
from skillopt_sleep.types import (
EditRecord,
ReplayResult,
Expand Down Expand Up @@ -280,6 +282,7 @@ def consolidate(
night: int = 1,
budget_exhausted: Optional[Callable[[], bool]] = None,
edit_ablation_queue: Optional[List[Callable[[], bool]]] = None,
skill_name: str = "",
) -> ConsolidationResult:
"""Run one consolidation epoch: reflect -> bounded edit -> gate.

Expand Down Expand Up @@ -313,6 +316,20 @@ def consolidate(
``edit_ablation_queue``: this epoch's ablation is appended to it instead
of being run here, and the caller runs it with
:func:`run_edit_ablations` once every consolidation has decided.

An ablation that raises does not cost the epoch its result, on either
path: the decision is already made. The row being measured is
``unmeasured`` with a note, a ``gate`` / ``edit_ablation_failed`` event is
logged, and no further ablation replay of this epoch starts. Only
``PendingCalls``, ``CursorBackendError`` and ``StagingError`` propagate,
as they do from a group in ``consolidate_groups``.

``skill_name`` labels this epoch's ablation events. They are logged after
every decision, so in fan-out they sit at the end of the evidence log,
away from their group's other events. Each one carries ``skill``,
``night`` and ``trial_index`` (the trial's position in ``gate_trials``),
so it can be matched to its trial without comparing edit content. Leave
``skill_name`` empty for a run that is not a per-skill group.
"""
from skillopt_sleep import evidence as evlog
ev = evlog.get(backend)
Expand Down Expand Up @@ -387,7 +404,9 @@ def _edits_payload(edits: List[EditRecord]) -> List[dict]:
# memory step, the final validation replay or a later epoch still checks.
pending_ablations: List[dict] = []

def _ablate_edits(plan: dict, budget_spent: bool) -> Tuple[List[dict], bool]:
def _ablate_edits(
plan: dict, budget_spent: bool, failure: str
) -> Tuple[List[dict], bool, str]:
"""Leave-one-out replay of one accepted trial.

For each applied edit, rebuild the candidate from the pre-trial
Expand All @@ -399,10 +418,24 @@ def _ablate_edits(plan: dict, budget_spent: bool) -> Tuple[List[dict], bool]:
The budget is checked before every replay. Returns the rows and
whether the budget is spent, so a later trial does not start a replay
the budget no longer covers.

The epoch has already decided, so an error here must not cost it its
result. A row whose replay or scoring raises is ``unmeasured``, the
error is logged, and no further replay starts: ``failure`` carries the
error to the rows and trials that follow and is returned with the
rows. ``PendingCalls``, ``CursorBackendError`` and ``StagingError``
still propagate, as the cycle-level control flow they are.
"""
which = plan["which"]
doc, new_doc, applied = plan["doc"], plan["new_doc"], plan["applied"]
with_pairs, with_score = plan["with_pairs"], plan["with_score"]
# Where this trial sits in the epoch's gate_trials. Ablation events are
# logged after every decision (in fan-out, after every group), so they
# name their trial instead of relying on their place in the log.
trial_index = next(
(i for i, trial in enumerate(gate_trials) if trial is plan["trial"]),
None,
)
rows: List[dict] = []
for index, edit in enumerate(applied):
remaining = [e for j, e in enumerate(applied) if j != index]
Expand All @@ -419,43 +452,60 @@ def _ablate_edits(plan: dict, budget_spent: bool) -> Tuple[List[dict], bool]:
]
replayed = False
note = ""
without_pairs: Optional[List[Tuple[TaskRecord, ReplayResult]]]
if not still_applied:
without_pairs = plan["prior_pairs"]
note = "scored by the replay that preceded this trial"
elif without_doc == new_doc:
without_pairs = with_pairs
note = "document is the same without this edit"
else:
if not budget_spent and budget_exhausted is not None:
budget_spent = bool(budget_exhausted())
if budget_spent and ev is not None:
ev.log("budget", "skip", step="edit ablation",
night=night)
if budget_spent:
without_score: Optional[float] = None
task_rows: List[dict] = []
try:
without_pairs: Optional[List[Tuple[TaskRecord, ReplayResult]]]
if not still_applied:
without_pairs = plan["prior_pairs"]
note = "scored by the replay that preceded this trial"
elif without_doc == new_doc:
without_pairs = with_pairs
note = "document is the same without this edit"
elif failure:
without_pairs = None
note = "not replayed: token budget spent"
note = "not replayed: ablation stopped after a failure"
else:
evlog.set_phase(backend, f"gate_ablation:{which}")
without_pairs = replay_batch(
backend,
val_tasks,
without_doc if which == "skill" else plan["skill"],
without_doc if which == "memory" else plan["memory"],
if not budget_spent and budget_exhausted is not None:
budget_spent = bool(budget_exhausted())
if budget_spent and ev is not None:
ev.log("budget", "skip", step="edit ablation",
night=night)
if budget_spent:
without_pairs = None
note = "not replayed: token budget spent"
else:
evlog.set_phase(backend, f"gate_ablation:{which}")
without_pairs = replay_batch(
backend,
val_tasks,
without_doc if which == "skill" else plan["skill"],
without_doc if which == "memory" else plan["memory"],
)
replayed = True
if without_pairs is not None:
task_rows = _task_deltas(
val_tasks, without_pairs, with_pairs,
gate_metric, gate_mixed_weight,
)
replayed = True
if without_pairs is None:
without_score = None
task_rows: List[dict] = []
else:
h, s = aggregate_scores(without_pairs)
without_score = select_gate_score(
h, s, gate_metric, gate_mixed_weight
)
task_rows = _task_deltas(
val_tasks, without_pairs, with_pairs,
gate_metric, gate_mixed_weight,
)
# Assigned last: a row whose scoring raised has no score,
# which is what reports it as unmeasured below.
h, s = aggregate_scores(without_pairs)
without_score = select_gate_score(
h, s, gate_metric, gate_mixed_weight
)
except (PendingCalls, CursorBackendError, StagingError):
# Cycle-level control flow (a paused handoff night, Cursor's
# fail-closed path), not evidence about one edit.
raise
except Exception as exc:
failure = f"{type(exc).__name__}: {exc}"[:300]
note = f"ablation failed: {failure}"
if ev is not None:
ev.log("gate", "edit_ablation_failed", target=which,
skill=skill_name, night=night,
trial_index=trial_index, edit_index=index,
error=failure)
observed = _ablation_observed_change(
with_score, without_score, task_rows
)
Expand Down Expand Up @@ -488,19 +538,25 @@ def _ablate_edits(plan: dict, budget_spent: bool) -> Tuple[List[dict], bool]:
})
if ev is not None:
ev.log("gate", "edit_ablation", target=which,
skill=skill_name, night=night, trial_index=trial_index,
n_edits=len(applied),
n_replays=sum(1 for row in rows if row["replayed"]),
rows=rows)
return rows, budget_spent
return rows, budget_spent, failure

def _run_pending_ablations() -> bool:
"""Ablate every accepted trial of this epoch, in trial order.

Returns True when a replay was skipped because the budget was spent.
A failed ablation is not a budget skip and is not raised: it is
reported on its rows and in the evidence log.
"""
budget_spent = False
failure = ""
for plan in pending_ablations:
rows, budget_spent = _ablate_edits(plan, budget_spent)
rows, budget_spent, failure = _ablate_edits(
plan, budget_spent, failure
)
plan["trial"]["edit_ablation"] = rows
return budget_spent

Expand Down
3 changes: 3 additions & 0 deletions skillopt_sleep/cycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -1202,6 +1202,9 @@ def budget_exhausted() -> bool:
# Every decision of the night is final here. The replays below are
# ordinary model calls: their tokens are part of "tokens used" and count
# toward max_tokens_per_night, which is checked before each of them.
# A replay that fails is contained by the queued run itself: its rows are
# unmeasured and the night is staged as decided. Only cycle-level errors
# reach this handler.
if ablation_queue:
_progress(cfg, "edit ablation start")
try:
Expand Down
6 changes: 4 additions & 2 deletions skillopt_sleep/dream.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ def dream_consolidate(
night: int = 1,
budget_exhausted: Optional[Callable[[], bool]] = None,
edit_ablation_queue: Optional[List[Callable[[], bool]]] = None,
skill_name: str = "",
) -> ConsolidationResult:
"""Recall similar past experience + dream synthetic variants, then run one
gated consolidation epoch over the enlarged training pool.
Expand All @@ -156,8 +157,8 @@ def dream_consolidate(
augmentation only enlarge the TRAIN split, so the val slice the gate scores
on is never polluted. With ``recall_k=0`` and ``dream_rollouts=1`` (the
defaults) this is exactly the previous single-shot ``consolidate``.
``budget_exhausted`` and ``edit_ablation_queue`` are passed through to
``consolidate``.
``budget_exhausted``, ``edit_ablation_queue`` and ``skill_name`` are passed
through to ``consolidate``.
"""
train = [t for t in tasks if t.split == "train"]
enlarged = list(tasks)
Expand Down Expand Up @@ -189,4 +190,5 @@ def dream_consolidate(
evolve_memory=evolve_memory, night=night,
budget_exhausted=budget_exhausted,
edit_ablation_queue=edit_ablation_queue,
skill_name=skill_name,
)
11 changes: 9 additions & 2 deletions skillopt_sleep/multi_skill.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,10 @@ def consolidate_groups(
last group has decided, so its replays cannot use budget a later group
still needs. A caller with further budget-dependent work passes its own
``edit_ablation_queue`` and runs it afterwards; otherwise the ablations run
here, after the loop. Either way a skipped ablation never changes a
group's status or reason.
here, after the loop. Either way a skipped or failed ablation never changes
a group's status or reason. Each group's name is passed to
``consolidate_fn`` as ``skill_name``, so its ablation events can be told
apart from the other groups'.
"""
# This wrapper's contract is stricter than consolidate(): shared memory is
# always read-only. Override a caller-supplied value instead of passing a
Expand Down Expand Up @@ -134,6 +136,11 @@ def consolidate_groups(
continue
try:
group_kwargs = dict(consolidate_kwargs)
if consolidate_kwargs.get("gate_edit_ablation"):
# Ablation events are logged once every group has decided, at
# the end of the evidence log; the name ties each one back to
# its group.
group_kwargs["skill_name"] = name
if group_kwargs_fn is not None:
group_kwargs.update(group_kwargs_fn(group))
# A per-group factory cannot weaken the shared-memory invariant.
Expand Down
Loading