[Klaud Cold] kimik2.5-fp4-b300-vllm: v0.26.0 image, extend conc to 512, add DEP4 arm / 升级 B300 镜像至 v0.26.0，扩展并发，新增 DEP4 - #2442
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
xinli-sw
changed the title
xinli-sw
added
full-sweep-fail-fast
full-sweep-enabled
and removed
full-sweep-fail-fast
labels
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30685318303 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30685980309 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30686074240 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30686418573 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30686474354 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30686474354 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30698837294 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30698837294 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30729674874 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30730026714 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30730985769 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30733192802 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30733234117 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30751776682 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30755774107 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30759578846 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30759578846 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30761976337 |
c3cace5 to
bfc24ba
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30766804009 |
58cb5cd to
22f4d7f
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30768418255 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30768705005 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30772091801 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30782159555 |
fbbdf4a to
80d9ed8
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30785195162 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30785317798 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30785413941 |
982a984 to
67892db
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30788333443 |
e1b0ff9 to
b13329c
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30793840724 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30793898084 |
| MODEL_BASENAME="${MODEL##*/}" | ||
| if [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then | ||
| if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then | ||
| export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}" | ||
| elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then | ||
| export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}" |
There was a problem hiding this comment.
🔴 The new staged-model bypass in launch_b300-nv.sh keys only on MODEL_PREFIX=="kimik2.5" && PRECISION=="fp4", so it also matches the unrelated kimik2.5-fp4-b300-vllm-agentic and kimik2.5-fp4-b300-vllm-agentic-mtp configs, forcing their already-staged models (Kimi-K2.5-NVFP4, Kimi-K2.6-NVFP4) off the read-only pre-staged mount and onto WRITABLE_MODELS_DIR. For the mtp config, MODEL is the bare string Kimi-K2.6-NVFP4 (no HF org prefix), so if that path isn't already populated the forced hf download will fail outright since it isn't a valid HF repo id. The condition should key off the specific model basename or config name (e.g. kimik2.5-fp4-b300-vllm) instead of the broad prefix+precision pair.
Extended reasoning...
The new branch added at runners/launch_b300-nv.sh (single-node else-branch, ~line 409) is:
if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}"
...It's inserted before the STAGED_MODELS membership check, and it keys purely on MODEL_PREFIX+PRECISION, not on the actual model being benchmarked or the specific config. The PR's intent (per the diff/description) is to force the one target config, kimik2.5-fp4-b300-vllm (model nvidia/Kimi-K2.5-NVFP4), onto the writable dir — presumably to pick up a fresh nightly image/weights rather than a stale pre-staged snapshot. But two other, unrelated configs in configs/nvidia-master.yaml share the same model-prefix: kimik2.5 / precision: fp4 pair and are unintentionally swept into the new branch:
kimik2.5-fp4-b300-vllm-agentic(line 1580):model: nvidia/Kimi-K2.5-NVFP4,runner: cluster:b300-nv,multinode: false.kimik2.5-fp4-b300-vllm-agentic-mtp(line 1600):model: Kimi-K2.6-NVFP4,runner: cluster:b300-nv,multinode: false.
Both route through the same single-node else-branch of launch_b300-nv.sh (the launcher is resolved generically from the runner name, and launch_b300-nv.sh is the only b300 launcher script), so both hit the new condition.
Why this matters — concrete proof:
STAGED_MODELS(defined a few lines above, line ~392 in the file) includes bothKimi-K2.5-NVFP4andKimi-K2.6-NVFP4. Before this PR, both agentic configs'MODEL_BASENAMEmatched that list and resolved to the read-only pre-staged mount:MODEL_PATH=/scratch/models/Kimi-K2.5-NVFP4and/scratch/models/Kimi-K2.6-NVFP4respectively.- After this PR, the new condition fires first for both (since
MODEL_PREFIX=kimik2.5andPRECISION=fp4for both), soMODEL_PATHbecomes/data/models/Kimi-K2.5-NVFP4and/data/models/Kimi-K2.6-NVFP4— the writable dir — regardless of whether anything is staged there. - The downstream bench script (same idempotent-download pattern as
kimik2.5_fp4_b300.sh:if [[ ! -d \"$MODEL_PATH\" || -z \"$(ls -A \"$MODEL_PATH\")\" ]]; then hf download \"$MODEL\" --local-dir \"$MODEL_PATH\"; fi) will trigger a freshhf downloadif/data/models/...isn't already populated for these models. - For
kimik2.5-fp4-b300-vllm-agentic,MODEL=nvidia/Kimi-K2.5-NVFP4is at least a valid HF repo id, so this 'only' costs an unnecessary multi-hundred-GB re-download of a model that was already sitting on the read-only staged mount. - For
kimik2.5-fp4-b300-vllm-agentic-mtp,MODEL=Kimi-K2.6-NVFP4has no HF org/namespace prefix — it is not a resolvable HF repo id at all. If/data/models/Kimi-K2.6-NVFP4doesn't happen to already exist from some unrelated prior run,hf download Kimi-K2.6-NVFP4 --local-dir ...will fail outright, breaking that recipe's sweep run entirely.
Existing code doesn't prevent this because the new condition was added as a plain prefix/precision predicate rather than being scoped to the specific model basename (MODEL_BASENAME == \"Kimi-K2.5-NVFP4\") or the specific config/recipe name, so it can't distinguish the PR's intended target from any other kimik2.5/fp4 recipe that happens to share those two fields.
Fix: scope the new branch to the actual model basename (e.g. [[ $MODEL_BASENAME == \"Kimi-K2.5-NVFP4\" ]]) or to the specific config name being run, so it only affects kimik2.5-fp4-b300-vllm and leaves the unrelated agentic/agentic-mtp recipes on their existing staged-model resolution path.
| PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1) | ||
| GMU=0.90 | ||
| PREFILL_SCHEDULE_ARGS=() | ||
| if [ "${DP_ATTENTION:-false}" = "true" ]; then | ||
| PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP") | ||
| GMU=0.85 | ||
| PREFILL_SCHEDULE_ARGS=(--prefill-schedule-interval 4) | ||
| fi | ||
|
|
||
| EP_ARGS=() | ||
| if [ "${EP_SIZE:-1}" -gt 1 ]; then | ||
| EP_ARGS=(--enable-expert-parallel) | ||
| fi |
There was a problem hiding this comment.
🟡 The top-of-file comment in kimik2.5_fp4_b300.sh (lines 3-5) and the matching comment in configs/nvidia-master.yaml (lines 1526-1528) both say the B300 recipe 'reuses the existing Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available' — but this PR adds exactly that B300-specific tuning (DP_ATTENTION/EP handling, prefill-schedule-interval, fp8 kv-cache, attention-config, linear-backend, new env vars), so the comment is now stale and misleading. Purely a documentation issue — recommend removing or updating both comments.
Extended reasoning...
What the bug is: The unmodified top-of-file comment in benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b300.sh (lines 3-5) states:
does not have a B300-specific recipe, so this script reuses the existing Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available.
The identical claim is duplicated in configs/nvidia-master.yaml at lines 1526-1528, directly above the kimik2.5-fp4-b300-vllm config block. Neither comment was touched by this PR, but this PR is precisely what makes them false.
Why it's now false: Comparing the new kimik2.5_fp4_b300.sh vllm serve invocation against kimik2.5_fp4_b200.sh, the two scripts diverge substantially after this PR:
- B300 adds a
DP_ATTENTIONbranch that switches--tensor-parallel-size/--data-parallel-size, adjusts GPU memory utilization (0.90 -> 0.85), and adds--prefill-schedule-interval 4; B200 has none of this. - B300 adds
EP_ARGS->--enable-expert-parallel; absent in B200. - B300 adds
--kv-cache-dtype fp8,--attention-config '{\"mla_prefill_backend\":\"FLASHINFER\",...}', and--linear-backend flashinfer_cutlass; none of these exist in B200. - B300's
--max-cudagraph-capture-sizeis now $((CONC*2)) vs B200's fixed 2048, and--stream-intervalis 32 vs B200's 20. - B300 sets new env vars
VLLM_USE_V2_MODEL_RUNNER=0,VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS,VLLM_RPC_TIMEOUT; B200 instead setsVLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, which B300 does not.
So the B300 recipe is no longer a copy-as-is of B200 — it is a distinct, B300-tuned recipe, and that tuning is exactly what the comment says doesn't exist yet.
Code path / proof: A reader opening kimik2.5_fp4_b300.sh sees the comment at lines 3-5 claiming 'reuses B200 as-is,' then scrolls a few lines down (post-PR, lines 19-31 and 67-86) and sees DP_ATTENTION branching, EP handling, fp8 kv-cache, attention-config, and linear-backend flags that have no B200 counterpart. The comment directly contradicts the code immediately below it. The same applies to the nvidia-master.yaml comment at lines 1526-1528, sitting right above the kimik2.5-fp4-b300-vllm: block that this PR also modifies (new nightly image, new DEP4 search-space arm).
Why existing code/review doesn't catch this: This is a plain-text comment, not something any linter, test, or CI check inspects — it can only be caught by manual review comparing the comment's claim against the diff.
Impact: Purely a documentation/maintainability issue — nothing breaks functionally. But a future engineer syncing B300 with B200, or auditing why the recipes differ, would be misled into thinking B300 has no tuning beyond a straight copy, when in fact it now has meaningfully different parallelism, quantization, and scheduling logic.
Fix: Remove or rewrite both comments (script header and yaml block comment) to reflect that B300-specific tuning has been added, rather than stating it 'reuses B200 as-is.'
77f4b75 to
51c8c80
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30959756748 |
1 participant
Summary
v0.22.0→vllm/vllm-openai:v0.26.0conc-end: 512tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512kimik2.5_fp4_b200.sh):DP_ATTENTIONhandling added — switches to--data-parallel-size, adds--enable-expert-paralleland--prefill-schedule-interval 4in DEP4 mode中文说明
v0.22.0→vllm/vllm-openai:v0.26.0conc-end: 512tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512kimik2.5_fp4_b200.sh）：新增DP_ATTENTION处理逻辑，DEP4 模式下切换为--data-parallel-size，并启用--enable-expert-parallel和--prefill-schedule-interval 4