Skip to content

fix(backends): preserve an explicit seed of 0 in sglang and vllm - #11786

Open
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:explicit-zero-seed
Open

fix(backends): preserve an explicit seed of 0 in sglang and vllm#11786
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:explicit-zero-seed

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Description

#11772 exempted Temperature from the zero-filter in both Python backend adapters, because proto3 has no field presence and an explicit 0 is indistinguishable from "unset". Seed has exactly the same property and is still filtered:

if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
    continue

A caller pinning "seed": 0 for a reproducible run therefore gets a random seed instead — no error, no log line. That is the one case where the failure is invisible precisely because the request looked deliberate.

Both adapters now share a named tuple of fields whose zero is meaningful, so the next one is added in one place rather than as a second special case.

Notes for Reviewers

Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero is not a value a caller means — sglang disables top_k with -1, not 0, so forwarding a 0 there would turn a default into an invalid argument.

Verified on an sglang backend in production (Qwen3.5-MoE FP8, GB10/arm64). With the temperature fix from #11772 alone, two identical requests at temperature: 0 are byte-identical; pinning seed: 0 still has no effect until this change.

For what it is worth as motivation: the silent substitution that #11772 fixed was not neutral in practice. In a code-audit benchmark here (one fixture with a planted bug, 5 runs per temperature, everything else at the model card's top_p 0.95 / top_k 20), the substituted default landed in the worse band:

temperature bug found false positives
0.0 – 0.6 35/35 0
0.7 – 1.0 16/20 1

Anyone benchmarking at temperature: 0 was measuring something other than what they asked for, with no way to tell. A dropped seed: 0 has the same shape.

Tests extended in both backend/python/sglang/test.py and backend/python/vllm/test.py; they assert that the seed survives and that top_k/top_p keep falling through to the engine default.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

mudler#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:

    if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
        continue

A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.

Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.

Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.

Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mudler Good to merge. The change preserves seed 0 in both Python sampling adapters without forwarding zero-valued parameters whose zero means unset or invalid. The upstream Go request builder already randomizes an omitted seed and preserves an explicitly configured 0, so this does not make unseeded requests deterministic. The contributor commit passes DCO; the changed Python files compile cleanly and the diff passes git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants