Skip to content

config: make aggregate multinode topology explicit #2479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 1 commit into from
Open

config: make aggregate multinode topology explicit #2479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 12 additions & 52 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7104,37 +7104,27 @@ qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg:
- dram-utilization: 0.80
search-space:
- spec-decoding: "mtp"
num-nodes: 1

@claude claude Bot Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 PR title and description are English-only, missing the Simplified Chinese translation that AGENTS.md mandates for every PR (<English title> / <中文标题> and a ## 中文说明 body section). This is a pre-existing repo convention this PR doesn't follow; please add the Chinese translation before merging.

Extended reasoning...

AGENTS.md line 7 states an explicit, non-negotiable convention: 'PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...).' It goes on to say 'This applies to every PR and every issue.'

PR #2479's title is config: declare aggregate multinode allocation sizes — English only, with no trailing / <中文标题> segment. The description body (Summary + Validation sections) contains no Chinese text at all, and specifically no ## 中文说明 section as the convention requires.

Step-by-step verification:

  1. Read AGENTS.md line 7 — confirms the bilingual title/description rule is mandatory and applies to every PR.
  2. Read the PR title in the metadata: config: declare aggregate multinode allocation sizes — no / <中文> suffix present.
  3. Read the PR description: contains only ## Summary and ## Validation sections, both English, no ## 中文说明 or any Chinese characters.
  4. Conclusion: the PR as submitted violates the documented, mandatory bilingual convention.

Nothing in the diff or the PR's actual content (a YAML config change backfilling num-nodes fields) is affected by this — the change itself is fine. The gap is purely in the PR metadata (title/description), which the author can fix without touching code: rename the title to append / <中文标题> and add a ## 中文说明 section mirroring the Summary.

This does not cause any runtime failure, benchmark regression, or data-correctness issue — it's a process/metadata compliance gap against a documented repo convention, not a code defect. Per the review rubric, a normal-severity finding should represent something that fails or breaks if unaddressed; this doesn't. It also isn't a PR-description-mismatch case (the description isn't inaccurate, it's just monolingual), so it's evaluated directly against the AGENTS.md rule rather than downgraded under that heuristic. Given AGENTS.md explicitly designates this as mandatory for every PR, it still merits being called out, but as a nit the author can resolve in under a minute rather than a blocking issue.

conc-list: [1]
kv-offloading: dram
kv-offload-backend: { name: hicache }
prefill:
num-worker: 1
worker:
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml"
decode:
num-worker: 0
tp: 4
ep: 1
dp-attn: false
- spec-decoding: "mtp"
num-nodes: 1
conc-list: [96]
kv-offloading: dram
kv-offload-backend: { name: hicache }
prefill:
num-worker: 1
worker:
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c96-mtp-hicache-jid2195211.yaml"
decode:
num-worker: 0
tp: 4
ep: 1
dp-attn: false


qwen3.5-fp4-gb300-dynamo-sglang-agentic-disagg:
Expand Down Expand Up @@ -8793,37 +8783,23 @@ dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg:
agentic-coding:
- search-space:
- spec-decoding: mtp
num-nodes: 2
conc-list: [1]
prefill:
num-worker: 1
worker:
tp: 8
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/deepseek-v4/agentic/agg-gb300-tp8-mtp-agentic.yaml"
# The aggregate worker also performs decode; keep the decode worker
# count at zero so result aggregation counts eight GPUs only once.
decode:
num-worker: 0
tp: 8
ep: 1
dp-attn: false
- spec-decoding: mtp
num-nodes: 1
conc-list: [4]
prefill:
num-worker: 1
worker:
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/deepseek-v4/agentic/agg-gb300-tp4-mtp-agentic.yaml"
# The aggregate worker also performs decode; keep the decode worker
# count at zero so result aggregation counts four GPUs only once.
decode:
num-worker: 0
tp: 4
ep: 1
dp-attn: false

dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-disagg:
image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f
Expand Down Expand Up @@ -8883,24 +8859,16 @@ dsv4-fp4-gb300-dynamo-sglang-agentic-agg:
- dram-utilization: 0.80
search-space:
- spec-decoding: mtp
num-nodes: 1
conc-list: [1, 4, 8, 16]
kv-offloading: dram
kv-offload-backend: { name: hicache }
prefill:
num-worker: 1
worker:
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/sglang/deepseek-v4/agentic/agg-gb300-tp4-mtp-kvoffload.yaml"
# Aggregated worker: prefill's GPUs also serve decode, so decode has no
# separate allocation (num-worker: 0). Deployment is recipe-driven
# (agg_workers: 1); this only makes GPU accounting count 4, not 4+4.
decode:
num-worker: 0
tp: 4
ep: 1
dp-attn: false

dsv4-fp4-gb300-dynamo-sglang-agentic-disagg:
image: lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8
Expand Down Expand Up @@ -9235,23 +9203,15 @@ kimik3-fp4-b200-dynamo-vllm-agentic:
agentic-coding:
- search-space:
- spec-decoding: none
num-nodes: 2
conc-list: [1, 2, 4, 8, 16, 32]
prefill:
num-worker: 1
worker:
tp: 8
pp: 2
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml"
# The aggregate worker also performs decode; keep the decode worker
# count at zero so result aggregation counts the 16 GPUs only once.
decode:
num-worker: 0
tp: 8
pp: 2
ep: 1
dp-attn: false

qwen3.5-fp8-gb300-dynamo-sglang-mtp:
image: lmsysorg/sglang:v0.5.14-cu130@sha256:5027e95bf6ec536856b1b52a91d1f35ff5c564ab83e8a94758a169ff09bb8df3
Expand Down
46 changes: 29 additions & 17 deletions utils/matrix_logic/generate_sweep_configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,27 @@ def with_worker_parallelism_defaults(worker: dict) -> dict:
}


def multinode_worker_pair(benchmark: dict, disagg: bool) -> tuple[dict, dict]:
"""Return the legacy prefill/decode matrix pair for a master entry."""
if disagg:
return (
with_worker_parallelism_defaults(benchmark[Fields.PREFILL.value]),
with_worker_parallelism_defaults(benchmark[Fields.DECODE.value]),
)

worker = with_worker_parallelism_defaults(benchmark[Fields.WORKER.value])
prefill = {Fields.NUM_WORKER.value: 1, **worker}
decode = {
Fields.NUM_WORKER.value: 0,
**{
key: value
for key, value in worker.items()
if key != Fields.ADDITIONAL_SETTINGS.value
},
}
return prefill, decode


def worker_gpus_per_node(worker: dict, gpus_per_node: int) -> int:
"""Return GPUs a single worker replica occupies on one node.

Expand Down Expand Up @@ -309,7 +330,10 @@ def agentic_dram_offload_gb(
utilization = Decimal(str(agentic_config[Fields.DRAM_UTILIZATION.value]))
gpus_per_node = runner_gpus_per_node(runner, runner_data)

if Fields.PREFILL.value in benchmark:
if Fields.WORKER.value in benchmark:
gpu_count = worker_gpus_per_node(
benchmark[Fields.WORKER.value], gpus_per_node)
elif Fields.PREFILL.value in benchmark:
gpu_count = worker_gpus_per_node(
benchmark[Fields.PREFILL.value], gpus_per_node)
else:
Expand Down Expand Up @@ -649,10 +673,7 @@ def generate_full_sweep(args, all_config_data, runner_data):
# spec_decoding defaults to "none" if not specified
spec_decoding = bmk.get(Fields.SPEC_DECODING.value, "none")

prefill = with_worker_parallelism_defaults(
bmk[Fields.PREFILL.value])
decode = with_worker_parallelism_defaults(
bmk[Fields.DECODE.value])
prefill, decode = multinode_worker_pair(bmk, disagg)

# Get concurrency values (can be list or range)
conc_list = bmk.get(Fields.CONC_LIST.value)
Expand Down Expand Up @@ -858,10 +879,7 @@ def generate_full_sweep(args, all_config_data, runner_data):

for bmk in bmk_space:
if is_multinode:
prefill = with_worker_parallelism_defaults(
bmk[Fields.PREFILL.value])
decode = with_worker_parallelism_defaults(
bmk[Fields.DECODE.value])
prefill, decode = multinode_worker_pair(bmk, disagg)
spec_decoding = bmk.get(Fields.SPEC_DECODING.value, "none")
kv_offloading = bmk.get(Fields.KV_OFFLOADING.value, "none")
kv_offload_backend = bmk.get(Fields.KV_OFFLOAD_BACKEND.value)
Expand Down Expand Up @@ -1050,10 +1068,7 @@ def generate_test_config_sweep(args, all_config_data, runner_data=None):
if is_multinode:
# Multinode config
spec_decoding = bmk.get(Fields.SPEC_DECODING.value, "none")
prefill = with_worker_parallelism_defaults(
bmk[Fields.PREFILL.value])
decode = with_worker_parallelism_defaults(
bmk[Fields.DECODE.value])
prefill, decode = multinode_worker_pair(bmk, disagg)

# Get concurrency values
if Fields.CONC_LIST.value in bmk:
Expand Down Expand Up @@ -1172,10 +1187,7 @@ def generate_test_config_sweep(args, all_config_data, runner_data=None):

for bmk in bmk_space:
if is_multinode:
prefill = with_worker_parallelism_defaults(
bmk[Fields.PREFILL.value])
decode = with_worker_parallelism_defaults(
bmk[Fields.DECODE.value])
prefill, decode = multinode_worker_pair(bmk, disagg)
spec_decoding = bmk.get(Fields.SPEC_DECODING.value, "none")
kv_offloading = bmk.get(Fields.KV_OFFLOADING.value, "none")
kv_offload_backend = bmk.get(Fields.KV_OFFLOAD_BACKEND.value)
Expand Down
43 changes: 43 additions & 0 deletions utils/matrix_logic/test_generate_sweep_configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
mark_all_eval_entries,
mark_eval_entries,
multinode_node_count,
multinode_worker_pair,
seq_len_itos,
seq_len_stoi,
seq_len_to_str,
Expand Down Expand Up @@ -45,6 +46,40 @@ def test_disaggregated_multinode_node_count_rejects_num_nodes():
with pytest.raises(ValueError, match="num-nodes.*disaggregated"):
add_multinode_node_count(entry, {}, num_nodes=3)


def test_aggregated_worker_expands_to_legacy_matrix_pair():
benchmark = {
"worker": {
"tp": 8,
"pp": 2,
"ep": 1,
"dp-attn": False,
"additional-settings": ["CONFIG_FILE=recipes/aggregate.yaml"],
}
}

prefill, decode = multinode_worker_pair(benchmark, disagg=False)

assert prefill == {
"num-worker": 1,
"tp": 8,
"pp": 2,
"dcp-size": 1,
"pcp-size": 1,
"ep": 1,
"dp-attn": False,
"additional-settings": ["CONFIG_FILE=recipes/aggregate.yaml"],
}
assert decode == {
"num-worker": 0,
"tp": 8,
"pp": 2,
"dcp-size": 1,
"pcp-size": 1,
"ep": 1,
"dp-attn": False,
}

def test_multinode_node_count_uses_role_gpu_footprints(sample_runner_config):
prefill = {"num-worker": 3, "tp": 2, "pp": 1, "pcp-size": 1}
decode = {"num-worker": 2, "tp": 8, "pp": 1, "pcp-size": 1}
Expand Down Expand Up @@ -1193,6 +1228,8 @@ def test_runner_node_filter_multinode(self, sample_runner_config, full_sweep_arg
"framework": "dynamo-trt",
"runner": "h200",
"multinode": True,
"disagg": True,
"kv-p2p-transfer": "nixl",
"scenarios": {
"fixed-seq-len": [

Expand Down Expand Up @@ -1456,6 +1493,8 @@ def test_multinode_conc_range_expansion(self, sample_runner_config, full_sweep_a
"framework": "dynamo-trt",
"runner": "gb200",
"multinode": True,
"disagg": True,
"kv-p2p-transfer": "nixl",
"scenarios": {
"fixed-seq-len": [

Expand Down Expand Up @@ -1574,6 +1613,8 @@ def test_multinode_max_conc_zero_or_negative_skips(self, sample_runner_config, f
"framework": "dynamo-trt",
"runner": "gb200",
"multinode": True,
"disagg": True,
"kv-p2p-transfer": "nixl",
"scenarios": {
"fixed-seq-len": [

Expand Down Expand Up @@ -1622,6 +1663,8 @@ def test_multinode_max_conc_creates_config_when_below_min(self, sample_runner_co
"framework": "dynamo-trt",
"runner": "gb200",
"multinode": True,
"disagg": True,
"kv-p2p-transfer": "nixl",
"scenarios": {
"fixed-seq-len": [

Expand Down
Loading