Skip to content

FIX: warn instead of raising on reasoning-model token truncation#2166

Open
romanlutz wants to merge 2 commits into
microsoft:mainfrom
romanlutz:romanlutz-legendary-invention
Open

FIX: warn instead of raising on reasoning-model token truncation#2166
romanlutz wants to merge 2 commits into
microsoft:mainfrom
romanlutz:romanlutz-legendary-invention

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Reasoning models spend completion tokens on hidden reasoning before emitting any visible output. When max_completion_tokens is set too low, the API can return finish_reason="length" with empty visible content. OpenAIChatTarget._validate_response treated that empty content as a hard failure and raised EmptyResponseException (surfaced with a misleading "Status Code: 204"), which also triggered the full retry loop even though retrying with the same token budget cannot succeed.

Since a low token budget can be a deliberate configuration choice, this changes the behavior to warn rather than raise:

  • On any finish_reason="length", log a warning explaining that reasoning models consume tokens on hidden reasoning and that max_completion_tokens may need to be increased.
  • When the truncation left no visible content, return a graceful empty response (error="empty") so the run continues instead of raising and retrying.
  • Responses that are empty for other reasons still raise EmptyResponseException as before.

The final stored result for the empty case is unchanged from today (an error="empty" piece). The difference is no exception, no retry storm, and a clear warning pointing at the token limit.

Tests and Documentation

  • Updated test_validate_response_success_length to assert the truncation warning is emitted when content is present.
  • Added test_validate_response_length_empty_returns_empty_response asserting that a truncated-and-empty response returns a Message with error="empty" and raises nothing.
  • Full tests/unit/prompt_target/target/test_openai_chat_target.py suite passes (101 tests).

No documentation changes needed. JupyText not run (no doc or notebook changes).

When a reasoning model hits max_completion_tokens, the API returns finish_reason='length' with empty visible content. Previously this raised EmptyResponseException (misleadingly reported as 'Status Code: 204') and triggered the 10x retry storm.

_validate_response now logs a warning explaining that reasoning models consume tokens on hidden reasoning, and returns a graceful empty response (error='empty') instead of raising, so runs continue. Non-length empty responses still raise as before.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants