Skip to content

fix(sglang): size batch truncation from the longest input in the batch, not the first - #1355

Open
shoemoney wants to merge 1 commit into
huggingface:mainfrom
shoemoney:fix-sglang-context-size-longest-input
Open

fix(sglang): size batch truncation from the longest input in the batch, not the first#1355
shoemoney wants to merge 1 commit into
huggingface:mainfrom
shoemoney:fix-sglang-context-size-longest-input

Conversation

@shoemoney

Copy link
Copy Markdown

SGLangModel._greedy_until decides whether to truncate a whole batch from context_size = len(inputs[0]) (src/lighteval/models/sglang/sglang_model.py:271 on main). But GenerativeTaskDataset sorts each split by character length of the query (src/lighteval/data.py:250, -(len(query) + gen_length)), so inputs[0] is the longest prompt in characters, not necessarily in tokens. A batch whose first prompt is short in tokens (long ASCII text) skips truncation entirely, and a later prompt that is shorter in characters but longer in tokens (for example CJK text) is sent to the sglang engine above max_length, corrupting that sample's result.

This is the same defect that issue #1204 confirmed for the vllm backend, whose fix PR #1205 makes its model-side change in src/lighteval/models/vllm/vllm_model.py. Lines 271-292 of the sglang model mirror vllm_model.py:372-397 line for line except for the max_length None guard discussed below, so this PR applies the identical one-line fix to the sglang twin:

context_size = max((len(input_ids) for input_ids in inputs), default=0)

The added unit test builds a two-doc batch where the character-longest prompt tokenizes to 10 tokens and the character-shorter one to 60, with max_length=50 and generation_size=10. On main it fails with 70 not less than or equal to 50 (the 60-token input reaches _generate untruncated); with the fix every input is left-truncated to 40 tokens and it passes. ruff format --check and ruff check are clean.

Deliberately not changed here: unlike the vllm copy, the sglang path has no self.max_length is None guard before the comparison. _create_auto_model defaults _max_length to 8192 when unset, so the None case appears unreachable for sglang, and adding the guard would widen this PR beyond the confirmed defect.

GenerativeTaskDataset sorts each split by character length of the query
(src/lighteval/data.py:250), so inputs[0] is the longest prompt in
characters but not necessarily in tokens. _greedy_until sized its
truncation check from len(inputs[0]) alone, so a batch whose first
prompt is short in tokens skips truncation entirely and any
longer-in-tokens prompt later in the batch is sent to the sglang engine
above max_length.

Same defect shape as the vllm copy: confirmed in issue huggingface#1204 and fixed
by PR huggingface#1205, which touches only src/lighteval/models/vllm/vllm_model.py
and leaves this identical sglang twin unfixed. This mirrors that fix:
context_size = max((len(input_ids) for input_ids in inputs), default=0).

Adds a unit test with a batch where the character-longest prompt is not
the token-longest; it fails on main (a 60-token input goes through
untruncated against max_length=50) and passes with the fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant