Skip to content

test(optimizer): add unit tests for the update-mode helpers - #418

Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:test/update-modes
Open

Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:test/update-modes

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

Summary

Adds tests/test_update_modes.py, direct unit tests for skillopt/optimizer/update_modes.py. These helpers decide which payload key the optimizer reads and writes for each update mode (edits, revise_suggestions or skill_candidates), apply the edit budget, and render the item descriptions the optimizer sees when it ranks edits. They had no tests of their own, so a regression would only show up as the optimizer silently reading the wrong key or getting the wrong number of edits.

No runtime code changes.

What the tests cover

  • Mode aliases: patch, edits, rewrite, suggestions, full_rewrite and the other documented names, case and whitespace insensitive, plus the fallback to patch mode for unknown names.
  • Payload keys and labels for all three modes, and that the mode predicates are mutually exclusive.
  • Malformed payloads: None, a non-dict container, or a non-list payload all read as no items.
  • Edit budget: truncate_payload with a budget below, equal to and above the item count, a zero budget, and a negative budget (no limit); it only touches the active mode's key.
  • Optimizer-facing text: describe_item and short_item_summary output for each mode, including a support count of 0 and the stripped new_skill preview.

Testing

  • python -m pytest tests/test_update_modes.py -q: 32 passed.
  • python -m pytest -q: 2319 passed, 12 skipped, 4 xfailed (32 more than main at af8f9ef).
  • python -m ruff check tests/test_update_modes.py and python -m ruff format --check tests/test_update_modes.py: clean.
  • python -m mkdocs build --strict: passed.

To check that the tests catch real regressions, I made each of these changes to update_modes.py in turn, ran the tests, and reverted. Every one makes at least one test fail:

Change to update_modes.py Failing tests
Remove the suggestions alias 2
Raise on unknown modes instead of falling back to patch 1
Stop lowercasing the mode name 1
Use edits as the full-rewrite payload key 2
Accept a non-list payload 1
Truncate on a negative budget 1
Off-by-one in truncation 3
Hide a support count of 0 1
Stop stripping the new_skill preview 1
Keep a non-list change_summary 1

Risk and compatibility

Test-only change plus a CHANGELOG line under Unreleased / Tests. No dependencies added; the tests are pure Python and run without network access.

skillopt/optimizer/update_modes.py decides which payload key the
optimizer reads and writes for each update mode (edits, revise
suggestions or full-skill candidates), applies the edit budget, and
renders the item descriptions shown to the optimizer when it ranks
edits. It had no direct tests.

Cover mode aliases and the fallback to patch mode for unknown names,
payload keys and labels, malformed payload handling, edit-budget
truncation including the zero and negative cases, and the per-mode
describe_item and short_item_summary output.

No runtime code changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant