Skip to content

feat(compact): drop the forced summarize tool_choice after tool results - #2330

Draft
mikasenghaas wants to merge 1 commit into
mainfrom
feat/compact-no-forced-summarize
Draft

feat(compact): drop the forced summarize tool_choice after tool results#2330
mikasenghaas wants to merge 1 commit into
mainfrom
feat/compact-no-forced-summarize

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Aug 12, 2026

Copy link
Copy Markdown
Member

Summary

  • Drop the forced tool_choice={"function": {"name": "summarize"}} on the compact harness's post-tool-result model call — summarize stays the only advertised tool there and the system prompt still requires saving notes, but the model now has to choose the call itself
  • A non-summarize call after a tool result still ends the rollout with no answer, so protocol violations become an observable (and trainable) negative signal instead of being mechanically prevented
  • Simplify chat() accordingly (the tool_choice parameter had no other caller)

Verification

16-task wikispeedia eval (eval split, Qwen/Qwen3-4B on a local vLLM), this branch vs the forced protocol:

  • Unforced (this PR): 0/16 solved. 15/16 rollouts end at their first tool result by emitting another task tool call (wiki_click_link ×11 + one more later, wiki_go_back ×3) instead of summarize — even with only summarize advertised, the server-side parser extracts the call and the protocol ends the rollout. Exactly one rollout followed the click→summarize loop unforced (7 turns) and then answered wrong. Trace dashboard: https://app.primeintellect.ai/dashboard/evaluations/wk9qajkt4afdnsx95enhgqoo
  • Forced (main), same task split, concurrent 500-task run: ~38% solve rate over the first 61 rollouts.

This reproduces the GLM-5.2 finding that motivated the forcing (advertising summarize alone is not enough). As-is, the unforced protocol yields no successful rollouts for Qwen3-4B — fine as a hard negative training signal, unusable for evals. If we want this behavior switchable, a force_summarize: bool = True knob on CompactingHarnessConfig is probably the better shape than removing the forcing outright.

Breaking

  • Compact-harness rollouts where the model replies to a tool result with anything other than a summarize call (or a plain-text finish) now end without an answer instead of being steered into summarize. Weaker instruction-followers will score lower under this harness.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant