Skip to content

[Bugfix][EPLB] Disabled shared expert overlap when EPLB is enabled #28377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Nov 10, 2025
Merged

[Bugfix][EPLB] Disabled shared expert overlap when EPLB is enabled #28377

merged 5 commits into from
Nov 10, 2025

Conversation

@SageMoore
Copy link
Contributor

@SageMoore SageMoore commented Nov 10, 2025 •
edited by github-actions bot
Loading

Purpose

EPLB is currently broken on main after #28164. This PR works around the issue by disabling shared expert overlap when EPLB is enabled.

Test Plan

lm eval

Test Result

Server command

VLLM_ALL2ALL_BACKEND=deepep_low_latency vllm serve --model="deepseek-ai/DeepSeek-V2-Lite" --max-num-seqs 512 --trust-remote-code --data-parallel-size 2 --enable-expert-parallel --gpu-memory-utilization 0.75 --disable-log-requests --enable-eplb --eplb-config '{"window_size":100,"step_interval":100}'

Before

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |    0|±  |     0|
|     |       |strict-match    |     5|exact_match|↑  |    0|±  |     0|

After

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3633|±  |0.0278|
|     |       |strict-match    |     5|exact_match|↑  |0.3600|±  |0.0278|

gemini-code-assist[bot]
gemini-code-assist bot reviewed Nov 10, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a workaround to fix a correctness issue with Expert Parallelism Load Balancing (EPLB) by disabling shared expert overlap when EPLB is enabled. The change is straightforward and effectively addresses the issue described. I have one suggestion to improve a comment's clarity to ensure it accurately reflects the code's logic, which will enhance future maintainability.

@simon-mo simon-mo added this to the v0.11.1 milestone Nov 10, 2025
yewentao256
yewentao256 approved these changes Nov 10, 2025
View reviewed changes
Copy link
Member

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, thanks for the work!

yewentao256
yewentao256 reviewed Nov 10, 2025
View reviewed changes
@yewentao256 yewentao256 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 10, 2025
SageMoore and others added 5 commits November 10, 2025 18:26
@SageMoore
init
0f2d197 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
@SageMoore
fix comment
cc5d3b7 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
@SageMoore @gemini-code-assist
Update vllm/model_executor/layers/fused_moe/shared_fused_moe.py
cf88eef 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Sage Moore <sagemoore@utexas.edu>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
@SageMoore @yewentao256
Update vllm/model_executor/layers/fused_moe/shared_fused_moe.py
87bef6c 
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Sage Moore <sagemoore@utexas.edu>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
@yewentao256 @SageMoore
Update vllm/model_executor/layers/fused_moe/shared_fused_moe.py
bd2c846 
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
@SageMoore SageMoore force-pushed the sage/eplb-shared-overlap-fix branch from d949b89 to bd2c846 Compare November 10, 2025 18:27
@mgoin mgoin added the bug Something isn't working label Nov 10, 2025
@SageMoore SageMoore changed the title Disabled shared expert overlap when EPLB is enabled [BUGFIX][EPLB] Disabled shared expert overlap when EPLB is enabled Nov 10, 2025
@SageMoore SageMoore changed the title [BUGFIX][EPLB] Disabled shared expert overlap when EPLB is enabled [Bugfix][EPLB] Disabled shared expert overlap when EPLB is enabled Nov 10, 2025
@yewentao256 yewentao256 enabled auto-merge (squash) November 10, 2025 19:53
@yewentao256 yewentao256 merged commit 40d3326 into vllm-project:main Nov 10, 2025
54 checks passed
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Nov 13, 2025
@SageMoore @gemini-code-assist
@yewentao256 @xuebwang-amd
[Bugfix][EPLB] Disabled shared expert overlap when EPLB is enabled (v…
7a700a5 
…llm-project#28377)

Signed-off-by: Sage Moore <sage@neuralmagic.com>
Signed-off-by: Sage Moore <sagemoore@utexas.edu>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yewentao256 yewentao256 yewentao256 approved these changes

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

v0.11.1

Development

Successfully merging this pull request may close these issues.

4 participants

@SageMoore @yewentao256 @mgoin @simon-mo