Skip to content

[AMD] improve dsr1 fp4 disagg #1584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
billishyahao wants to merge 13 commits into
base: main
Choose a base branch
from
+71 −11
Open

[AMD] improve dsr1 fp4 disagg #1584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d93b9c3
[AMD] improve dsr1 fp4 disagg
billishyahao May 29, 2026
1739443
add perf changelog
billishyahao May 29, 2026
1e5e851
Merge remote-tracking branch 'inf/main' into amd/mi355x-dsfp4-may29
billishyahao May 30, 2026
bfb242c
fix
billishyahao May 30, 2026
9a39f19
fix regression
billishyahao May 30, 2026
ae39e6b
sync config
billishyahao May 30, 2026
7a72e6d
fix
billishyahao May 31, 2026
7407a28
Merge branch 'main' into amd/mi355x-dsfp4-may29
billishyahao May 31, 2026
72ea8c3
fix
billishyahao May 31, 2026
12173fa
suppress aiter log
billishyahao May 31, 2026
af822ed
Merge branch 'main' into amd/mi355x-dsfp4-may29
billishyahao May 31, 2026
44c1376
fix
billishyahao May 31, 2026
a0a1ddc
fix
billishyahao May 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 61 additions & 10 deletions .github/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1861,8 +1861,8 @@ dsr1-fp4-mi355x-sglang-disagg:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=0"

dsr1-fp4-mi355x-sglang-disagg-mtp:
image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260519
dsr1-fp4-mi355x-sglang-disagg-1k1k-mtp:
image: lmsysorg/sglang-rocm:v0.5.12.post1-rocm720-mi35x-20260529
model: amd/DeepSeek-R1-0528-MXFP4-v2
model-prefix: dsr1
runner: mi355x-disagg
Expand Down Expand Up @@ -1970,7 +1970,19 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"


dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp:
image: lmsysorg/sglang-rocm:v0.5.12.post1-rocm720-mi35x-20260529
model: amd/DeepSeek-R1-0528-MXFP4-v2
model-prefix: dsr1
runner: mi355x-disagg
precision: fp4
framework: sglang-disagg
multinode: true
disagg: true
scenarios:
fixed-seq-len:
- isl: 8192
osl: 1024
search-space:
Expand Down Expand Up @@ -2015,7 +2027,7 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:

# 1P2D TP8
- spec-decoding: "mtp"
conc-list: [ 64, 128, 256 ]
conc-list: [ 32, 64 ]
prefill:
num-worker: 1
tp: 8
Expand All @@ -2030,11 +2042,11 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
dp-attn: false
additional-settings:
- "DECODE_NODES=2"
- "DECODE_MTP_SIZE=2"
- "DECODE_MTP_SIZE=3"

# 1*DEP8 + 1*DEP8
- spec-decoding: "mtp"
conc-list: [ 128, 512 ]
conc-list: [ 512 ]
prefill:
num-worker: 1
tp: 8
Expand All @@ -2049,11 +2061,11 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
dp-attn: true
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"
- "DECODE_MTP_SIZE=3"

# 1*DEP8 + 1*DEP8
- spec-decoding: "mtp"
conc-list: [ 64, 256 ]
conc-list: [ 256 ]
prefill:
num-worker: 1
tp: 8
Expand All @@ -2068,11 +2080,50 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
dp-attn: true
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"
- "DECODE_MTP_SIZE=3"


# 1*DEP8 + 1*DEP8
- spec-decoding: "mtp"
conc-list: [ 128 ]
prefill:
num-worker: 1
tp: 8
ep: 8
dp-attn: true
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 1
tp: 8
ep: 8
dp-attn: true
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=3"

# 1*DEP8 + 1*DEP8
- spec-decoding: "mtp"
conc-list: [ 64 ]
prefill:
num-worker: 1
tp: 8
ep: 8
dp-attn: true
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 1
tp: 8
ep: 8
dp-attn: true
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=3"
Comment thread
Oseltamivir marked this conversation as resolved.

# 2*DEP8 + 1*DEP8
- spec-decoding: "mtp"
conc-list: [ 1024, 2048, 4096 ]
conc-list: [ 2048, 4096 ]
prefill:
num-worker: 2
tp: 8
Expand All @@ -2088,7 +2139,7 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"


# DSv4-Pro FP4 on MI355X via SGLang. Uses a rocm720 mi35x image built off the
# amd/deepseek_v4 branch in sgl-project/sglang; the SHA is encoded in the
Expand Down
4 changes: 3 additions & 1 deletion benchmarks/multi_node/amd_utils/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,11 @@ else
# =========================================================================

export SGLANG_USE_AITER=1
export AITER_LOG_LEVEL=ERROR

export SGLANG_MORI_DISPATCH_DTYPE=auto
export SGLANG_MORI_FP8_COMB=true
export MORI_COMBINE_DTYPE_PREFILL=fp8_direct_cast
export MORI_COMBINE_DTYPE_DECODE=fp8
export SGLANG_MORI_QP_PER_TRANSFER=4
export SGLANG_MORI_NUM_WORKERS=4
export MORI_IO_SQ_BACKOFF_TIMEOUT_US=50000
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3201,6 +3201,13 @@
- "MoRI conn.py overlay (48e459bd) via job.slurm; launcher qwen3.5_fp4_mi355x_sglang-disagg.sh"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1579

- config-keys:
- dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp
description:
- "Bump the image to May 26"
- "Add conc 128/256 new sweep point"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1584

- config-keys:
- glm5-fp8-gb300-dynamo-sglang
description:
Expand Down