Skip to content

fix(optimizer): keep slow-update and meta-skill content as text - #420

Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix/slow-update-list-content
Open

Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix/slow-update-list-content

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

Summary

Fixes #416.

When the optimizer returned slow-update guidance as a JSON list of strings, the list's Python repr, for example ['Check units first.', 'Prefer exact spans.'], was written into the skill's protected slow-update region. The optimizer meta-skill memory had the same problem. A list of strings is now joined one item per line; any other non-text value, or blank content, counts as no update.

User impact: the slow-update region is part of the skill the target model reads, so it got Python syntax instead of guidance. Whitespace-only content was also accepted as an empty update, which wiped the existing guidance, and a null reasoning field was logged as None.

Proof: the regression tests fail on upstream af8f9ef and pass with this change.

Root cause

run_slow_update() in skillopt/optimizer/slow_update.py and run_meta_skill() in skillopt/optimizer/meta_skill.py accepted any truthy content field and passed it through str(). The prompts ask for a string, but a list of bullet strings is a common model deviation, and nothing normalized it.

Changes

  • New content_text(value) in slow_update.py: a string is stripped; a list of strings is joined one item per line; anything else (None, a dict, a number, a mixed list) gives "".
  • run_slow_update() and run_meta_skill() use it for the content and the reasoning, and return None when the content is empty, as they already do for a parse failure.
  • New tests/test_slow_update_content.py, calling the real functions with a fake optimizer response and checking the guidance written into the skill.
  • CHANGELOG entry under Unreleased / Fixed.

Reproduction

import json
from unittest import mock
from skillopt.optimizer import slow_update

payload = {"reasoning": "r", "slow_update_content": ["Check units first.", "Prefer exact spans."]}
with mock.patch.object(slow_update, "chat_optimizer", return_value=(json.dumps(payload), {})):
    result = slow_update.run_slow_update("# Skill", [], [], [], comparison_pairs=[], system_prompt="slow")
print(slow_update.replace_slow_update_field("# Skill", result["slow_update_content"]))

Before:

# Skill

<!-- SLOW_UPDATE_START -->
['Check units first.', 'Prefer exact spans.']
<!-- SLOW_UPDATE_END -->

After:

# Skill

<!-- SLOW_UPDATE_START -->
Check units first.
Prefer exact spans.
<!-- SLOW_UPDATE_END -->

Testing

  • python -m pytest tests/test_slow_update_content.py -q: on upstream af8f9ef, 8 failed and 1 passed; with the fix, 9 passed.
  • python -m pytest tests/test_slow_update_content.py tests/test_slow_update_robustness.py tests/test_hierarchical_merge.py tests/test_skill_aware_reflection.py tests/test_trainer_resume.py -q: 26 passed.
  • python -m pytest -q: 2296 passed, 12 skipped, 4 xfailed.
  • python -m ruff check and python -m ruff format --check on the changed files: clean.
  • python -m mkdocs build --strict: passed.

Risk and compatibility

String content is handled exactly as before. Only list, non-text and blank content change: lists of strings become readable text, and the rest now count as no update instead of writing a repr or an empty region. No backend, config key, file format or CLI change.

run_slow_update and run_meta_skill accepted any truthy content field and
passed it through str(). When the optimizer returned the guidance as a
JSON list of strings, the list's Python repr was written into the
skill's slow-update region or the optimizer memory, for example
"['Check units first.', 'Prefer exact spans.']". A dict or number was
written the same way, and whitespace-only content was accepted as an
empty update. A null reasoning field became the text "None".

Add content_text(): a string is stripped, a list of strings is joined one
item per line, and anything else is unusable. Both parsers use it for the
content and the reasoning, and return None when the content is empty, as
they already do for a parse failure.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slow-update guidance returned as a list is written into the skill as a Python list repr

1 participant