Repository navigation
fix: JSON mode with stream: true sends its reply; nested objects stay whole - #235
Merged
Merged
Conversation
…ts whole
handleChatStreaming buffers JSON-mode output so a hallucinated prefix
object can be dropped, and only flushed the buffer from inside a chunk,
once 32 chunks had arrived or a second "{" showed up. Nothing flushed it
when the stream ended. The generator holds a "{"-led reply as a possible
bare tool call and hands it over as one chunk, so the thresholds never
tripped and a JSON-mode stream carried no content at all.
The flush is now one function, called at the threshold as before and at
the end of the stream, in the .info case and in the post-loop safety
net. The prefix strip drops only a complete object that precedes a
second one; the old rule cut at any second "{" and turned
{"user": {"name": "Bob"}} into {"name": "Bob"}}.
JSONModeStreamTests drives the handler with a synthetic stream, and
contract Test 13 checks that a JSON-mode stream assembles to the
non-streaming reply.
Closes SharpAI#233.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #233.
What
A JSON-mode stream now sends its reply.
handleChatStreamingbuffers JSON-mode output so a hallucinated prefix object can be dropped, and the buffer was only flushed from inside a chunk, once 32 chunks had arrived or a second{showed up; nothing flushed it when the stream ended. The generator holds a{-led reply as a possible bare tool call and hands it over as one chunk, so the thresholds never tripped and the client got an emptydeltaand[DONE].Two changes in that block:
flushJSONBuffer, called at the threshold as before and now also at the end of the stream: incase .infobefore the finish chunk, and in the post-loop safety net for a stream that ends without one.jsonModeFlushText: it drops a complete object that precedes a second one and otherwise sends the text whole. The old rule cut at the second{wherever it was, so{"user": {"name": "Bob"}}went out as{"name": "Bob"}}.The thresholds, the stop-sequence handling and the non-streaming path are untouched. What the buffer holds is still not scanned for stop sequences before it goes out, same as before this change.
Why
Structured output is where streaming and JSON mode meet, and
client.chat.completions.create(stream=True, response_format={"type": "json_object"})returned nothing to iterate over. Repro and line references in #233.Verified
JSONModeStreamTests(new) driveshandleChatStreamingwith a syntheticGenerationstream, no model needed: a one-chunk reply reaches the client at.infoand when the stream ends without.info; a nested object delivered per token is forwarded whole; a closed prefix object is still dropped; outside JSON mode nothing changes. The prefix rule has its own cases. On main, three of the five handler cases fail.stream: trueassembles to the same content as the non-streaming reply. On main the stream is empty, so it fails there; green here.swift test --filter SwiftLMTests: 326 tests green (M1 Max, Xcode 27 / Swift 6.4).tests/test-contract.shwith the CI model (mlx-community/Qwen2.5-0.5B-Instruct-4bit): 21 passed, 0 failed, 2 skipped.