Skip to content

fix: guard against IndexError on empty LLM choices/content list#3874

Open
qizwiz wants to merge 2 commits into
lm-sys:mainfrom
qizwiz:fix/guard-empty-llm-choices
Open

fix: guard against IndexError on empty LLM choices/content list#3874
qizwiz wants to merge 2 commits into
lm-sys:mainfrom
qizwiz:fix/guard-empty-llm-choices

Conversation

@qizwiz
Copy link
Copy Markdown

@qizwiz qizwiz commented May 17, 2026

Problem

Three files access choices[0] or content[0] without first checking whether the API returned any results. An empty list can occur when:

  • the LLM applies a content-policy filter and returns choices=[]
  • a token limit is hit before the first choice arrives
  • a rate-limit fallback returns a non-standard response body
  • the Anthropic API returns content=[] on a refusal

This raises a bare IndexError that propagates as an unhandled exception.

Changes

File Line Fix
fastchat/serve/api_provider.py 351 if not res.choices: return before accessing choices[0]
fastchat/serve/monitor/classify/label.py 67 if not completion.choices: break before accessing choices[0]
fastchat/serve/monitor/classify/label.py 117 if not response.content: break before accessing Anthropic content[0]
fastchat/serve/monitor/criteria_labeling.py 92 if not completion.choices: break before accessing choices[0]

All three files already use retry loops — break on empty response exits the loop and returns the API_ERROR_OUTPUT sentinel, which is the same behaviour as any other API error.

No new files, no behaviour change on successful API responses.


Found by pact static analysis — llm_response_unguarded mode.

qizwiz added 2 commits May 17, 2026 13:51
Three call sites access choices[0] or content[0] without checking
whether the API returned any results — content-policy filters, token
limits, and rate-limit fallbacks can all return an empty list.

- api_provider.py: early return when res.choices is empty
- classify/label.py: skip iteration when completion.choices is empty
- classify/label.py: skip Anthropic iteration when response.content is empty
- criteria_labeling.py: skip iteration when completion.choices is empty
…t-filter)

Gemini 2.5 Flash returns HTTP 200 with choices[0].message=None when content
is filtered (finish_reason: PROHIBITED_CONTENT), causing AttributeError even
when choices is non-empty. Extends all three guards from `if not choices:`
to `if not choices or choices[0].message is None:`.
@qizwiz
Copy link
Copy Markdown
Author

qizwiz commented May 17, 2026

Update: extended guards to cover a second crash vector

Testing against live providers revealed that Gemini 2.5 Flash returns HTTP 200 with choices[0].message = None (not an empty choices list) when content is filtered — finish_reason: PROHIBITED_CONTENT. The previous guards if not completion.choices: and if not res.choices: only prevent IndexError; they do not prevent AttributeError: 'NoneType' object has no attribute 'content' when message itself is None.

Pushed a follow-up commit (cab948d) extending all three guards:

  • fastchat/serve/monitor/classify/label.py
  • fastchat/serve/monitor/criteria_labeling.py
  • fastchat/serve/api_provider.py

New guard pattern: if not completion.choices or completion.choices[0].message is None:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant