Skip to content

fix(realtime): validate and forward vLLM HF overrides - #3810

Merged
LauraGPT merged 1 commit into
mainfrom
fix/realtime-hf-overrides-20261101
Oct 31, 2026
Merged

LauraGPT merged 1 commit into
mainfrom
fix/realtime-hf-overrides-20261101

Conversation

@LauraGPT

Copy link
Copy Markdown
Collaborator

Summary

Integrates @gotorion's #3768 on current main, with credit and the original fork/head untouched. The realtime CLI accepts --hf-overrides JSON objects and forwards them to vLLM. The saved October compatibility correction keeps callers without the optional Namespace attribute working. Non-object JSON is rejected before model loading. The same regression tests now run in both existing NumPy CI matrices.

Validation

  • Seven new non-object rejection cases fail before validation; 185 focused realtime/PCM tests pass afterward, with zero skips. This includes 90 realtime service, 13 new JSON validation and 82 PCM cases.
  • Native workflow collection previously selected no realtime tests. It now collects 358 cases, including all 90 service and 13 new validation cases, with PR and main-push filters covered. The collection probe initially expected 100 service tests by mistake; actual successful JUnit established 90, and the probe was corrected without changing product tests or acceptance criteria.
  • Actual L20Z/native vLLM CUDA generation passes in FP32 and BF16, each using two fixed token prompts and four output tokens. Locally seeded random Qwen2 weights are saved and hashed identically for both modes (SHA256 7e47b3b731f30a7a4be1c21280f3eda57ab9767ec98894c71605869b9d554608). An intentionally unsupported architecture in the saved config is overridden by the parsed CLI object.
  • The first GPU fixture failed because its head dimension 8 was below FlexAttention's minimum 16. That failure is retained. The corrected fixture uses head dimension 32 and exact saved random weights, not downloaded model weights; product source did not change between GPU attempts.
  • Runtime: Python 3.12.3, Torch 2.8.0a0+5228986c39.nv25.06/CUDA 12.9, Transformers 5.3.0. The preinstalled custom vLLM reports import version 0.11.1 while distribution metadata says 0.13.0rc2.dev2227+gf6e85aa14.d20260630; both and its source hash are recorded. Non-fatal optional-kernel/import and NCCL exit warnings are retained; worker processes exited and the GPU was idle afterward.
  • Scope: CPU loader-boundary contracts plus real CLI-to-native-vLLM config/CUDA execution, not pretrained ASR accuracy, full AutoModel audio end-to-end inference, every custom architecture, the complete repository suite, or Nano leaderboard work.
  • New-head CI/artifacts and a normal protected merge are required. No administrator bypass, protection change, release or PyPI upload. feat: support hf-overrides for realtime server #3768 is only to be closed as superseded after integration, not described as an original-commit merge.

Integrate #3768 with the preserved compatibility correction, object validation and durable CI.

Co-authored-by: Li Junhui <lijunhui.me@foxmail.com>

Signed-off-by: LauraGPT <zhifu.gzf@alibaba-inc.com>
@LauraGPT
LauraGPT merged commit 3808562 into main Oct 31, 2026
2 checks passed
@LauraGPT

Copy link
Copy Markdown
Collaborator Author

Normally merged with a verified signature and DCO. Final main matches the tested source tree. Both new-head and final-main NumPy matrices passed 358 cases each, with all 90 realtime and 13 new validation cases present, zero failures/skips, and actual downloaded artifacts/logs checked against exact source commits.

CPU and synthetic native FP32/BF16 vLLM GPU scope and the retained failed fixture are detailed above. #3768 is closed as superseded with its author credited and fork untouched. This is a source fix, not a new release, pretrained ASR validation or Nano leaderboard work.

@LauraGPT LauraGPT mentioned this pull request Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant