Skip to content

fix(openai_api_server): use exclude_none for completion stream finish chunks#3872

Open
Chessing234 wants to merge 1 commit into
lm-sys:mainfrom
Chessing234:fix-openai-api-completion-stream-finish-exclude-none
Open

fix(openai_api_server): use exclude_none for completion stream finish chunks#3872
Chessing234 wants to merge 1 commit into
lm-sys:mainfrom
Chessing234:fix-openai-api-completion-stream-finish-exclude-none

Conversation

@Chessing234
Copy link
Copy Markdown

Bug

`generate_completion_stream_generator` (`fastchat/serve/openai_api_server.py`) emits the final SSE chunks for a completion stream with `model_dump_json(exclude_unset=True)`, leaving `None` fields (e.g. `logprobs`) in the payload that the preceding comment explicitly says should be stripped.

Root cause

The function ends with:

```python

There is not "content" field in the last delta message, so exclude_none to exclude field "content".

for finish_chunk in finish_stream_events:
yield f"data: {finish_chunk.model_dump_json(exclude_unset=True)}\n\n"
```

The comment and `exclude_*` keyword disagree. The parallel chat-completion path `chat_completion_stream_generator` carries the identical comment immediately above and correctly uses `exclude_none=True`:

```python

There is not "content" field in the last delta message, so exclude_none to exclude field "content".

for finish_chunk in finish_stream_events:
yield f"data: {finish_chunk.model_dump_json(exclude_none=True)}\n\n"
```

This is a copy-paste asymmetry: the completion-stream version kept `exclude_unset` while the chat-stream version was corrected to `exclude_none`.

Fix

Switch the completion-stream finish loop to `exclude_none=True` so it matches its comment and its chat-stream counterpart. One-line change; per-chunk emission inside the loop body (which legitimately uses `exclude_unset`) is untouched.

… chunks

generate_completion_stream_generator's finish loop carried over the
same '...so exclude_none to exclude field "content"' comment as the
chat variant but called model_dump_json(exclude_unset=True). The chat
counterpart (chat_completion_stream_generator) correctly uses
exclude_none=True. Switch the completion path to exclude_none so the
final-chunk None fields (e.g. logprobs) are stripped as the comment
and the parallel chat-stream path intend.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant