Skip to content

OpenAI Chat Completions: tool rounds drop the model's reasoning #326

Open

Description

@gtokman

OpenAILanguageModel's Chat Completions tool loop drops the model's reasoning between tool rounds. Thinking models return it alongside their tool calls, in reasoning_content (DeepSeek's API) or reasoning and reasoning_details (OpenRouter, Vercel AI Gateway). The next request in the same response rebuilds the assistant message from the content and tool calls only, so the reasoning that planned those calls is lost.

Why it matters

Reproduction

Run a session with tools against deepseek/deepseek-v4.1-flash-fast or zai/glm-5.3-fast on the Vercel AI Gateway (https://ai-gateway.vercel.sh/v1), and record the request bodies. Each response's choices[0].message has reasoning and reasoning_details next to tool_calls. In the next request, the matching assistant message has only content and tool_calls. Across one 8-round response, 0 of the 8 assistant messages carried the reasoning back.

PR #290 added reasoning transcript entries and replay for Anthropic. Chat Completions doesn't read or send these fields yet (checked on main at 80d50b0).

A possible fix

In the tool loop, decode the three fields from the response message and send them back unchanged on that round's assistant message. Keeping them within the one response matches DeepSeek's rule, so the transcript doesn't need to change. I did this for the nonstreaming loop in a fork, with a test: gtokman/AnyLanguageModel@daf184b. The commit is on 0.16.0. The streaming loop would need the same change, gathering the reasoning deltas.

A fuller version would store the reasoning as Transcript.Reasoning entries, as #290 does for Anthropic, so it could also be replayed after a transcript is restored. I'm happy to open a PR for either approach.

Activity

  1. mattt commented on Oct 14, 2026

    @mattt
    Collaborator

    Thanks, @gtokman. #325 now builds the tool call message from the transcript, both for the round in flight and for the replay on the next turn. If the reasoning goes only into the in-flight message, the two would differ again, and we'd lose the prompt caching fix. So I think the fuller version fits better: store the reasoning as Transcript.Reasoning entries, as #290 does for Anthropic, and have toolCallsMessage send them back, in both the streaming and nonstreaming loops.

    Could you open that on top of #325? I'd be happy to review it. One question first: does DeepSeek accept reasoning_content from earlier turns, or does it need to be dropped after the turn ends?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions