Skip to content

fix(openai): keep n>1 stream choices separate in recorded output - #1864

Open
sailikhithk wants to merge 2 commits into
langfuse:mainfrom
sailikhithk:fix/openai-stream-multi-choice-merging
Open

fix(openai): keep n>1 stream choices separate in recorded output#1864
sailikhithk wants to merge 2 commits into
langfuse:mainfrom
sailikhithk:fix/openai-stream-multi-choice-merging

Conversation

@sailikhithk

@sailikhithk sailikhithk commented Sep 5, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes langfuse/langfuse#16998

_extract_streamed_openai_response used a single accumulator for every choice in a chat completion stream. An n>1 stream interleaves choices within a single chunk, so content and tool-call arguments from separate choices were merged into one corrupted output (the issue's repro records "AB10" instead of "A0" and "B1").

The extractor now accumulates per choice index and returns one response per choice when the stream carries multiple choices. Single-choice streams keep the existing output shape, so the common n=1 case is unchanged. A stream that only ever yields output for one nonzero choice index (partial consumption) now records that choice instead of None.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Refactor
  • Documentation update
  • Tooling, CI, or repo maintenance

Verification

uv run --frozen pytest tests/unit/test_openai.py                 # 37 passed
uv run --frozen pytest -n auto --dist worksteal tests/unit       # 672 passed, 2 skipped (18 pre-existing test_prompt.py fixture errors reproduce on clean main without provider credentials)
uv run --frozen ruff check .                                     # All checks passed!
uv run --frozen ruff format --check .                            # clean; only pre-existing tests/unit/test_media.py drift on main
uv run --frozen mypy langfuse --no-error-summary                 # exit 0

Checklist

  • I self-reviewed the diff using code_review.md.
  • I added or updated tests for behavior changes.
  • I updated docs, examples, or .env.template if needed. (no docs impact: internal extractor fix, public output shape for n=1 unchanged)
  • I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
  • I did not commit secrets or credentials.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@CLAassistant

CLAassistant commented Sep 5, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread langfuse/openai.py Outdated
_extract_streamed_openai_response used a single accumulator for every
choice in a chat completion stream, so an n>1 stream with interleaved
choices merged content and tool-call arguments from separate choices
into one corrupted output.

Accumulate per choice index and return one response per choice when
the stream carries multiple choices; single-choice streams keep the
existing output shape.
An n>1 stream that only ever yields output for one nonzero choice
index, for example after partial consumption, was read back from
index 0 and recorded as None. Fall back to the single collected
choice regardless of its index.
@sailikhithk
sailikhithk force-pushed the fix/openai-stream-multi-choice-merging branch from eddd8e8 to aeb8e6b Compare September 5, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(sdk-python): OpenAI streams merge multiple choices

2 participants