[3/4] [Power] feat: enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集 - #2456
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @edwingao28's task in 3m 59s —— View job Review — PR #2456 (official dcgm energy lane, gb200/gb300 1P1D)
LGTM - no blocking issues found. Things I specifically checked and confirmed sound:
One non-blocking observation (no action needed now): the lane check is two independent greps ( Also noting the PR's own plan: the producer pin currently points at the fork SHA ( |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
No bugs were found, but I'm deferring to a human reviewer given the scope and risk profile of this change. This PR is explicitly marked draft-on-purpose — the producer pin currently points at a third-party fork (edwingao28/srt-slurm) rather than the eventual upstream merge SHA — and it modifies the critical gb200/gb300 multinode launcher scripts that gate all benchmark submissions on those clusters, not just the new power lane.
What was reviewed: the recipe-driven power-lane detection logic in both launchers (grep-based, resolved against the workspace recipe mirror before the srt-slurm clone exists), the pinned-clone-and-assert-HEAD contract that writes power-producer-sha.txt, the CI step in benchmark-multinode-tmpl.yml that derives POWER_PRODUCER_SHA from that stamp (input still overrides), and the new contract tests covering both launchers. The bug hunting system additionally checked the gb300 dcgm-exporter image enroot import syntax and ruled it out as correct.
Extended reasoning...
Overview
This PR wires an official DCGM power-telemetry lane into the gb200/gb300 1P1D qwen3.5 recipes. It adds a telemetry block to the two recipe YAMLs, adds recipe-gated (grep-detected) exporter provisioning and a pinned srt-slurm producer clone to both runners/launch_gb200-nv.sh and runners/launch_gb300-nv.sh, threads a power-producer-sha.txt stamp from the launcher through benchmark-multinode-tmpl.yml into POWER_PRODUCER_SHA, and adds static contract tests (utils/test_gb200_power_official_contract.py, utils/test_gb300_power_official_contract.py) asserting the launcher text matches the intended contract. Non-power lanes are intended to be byte-identical to their pre-PR behavior (exporter/pin logic is gated behind a USES_DCGM_POWER flag derived from the recipe).
Security risks
The main risk is supply-chain: the power lane clones and executes setup/build steps from a personal fork (edwingao28/srt-slurm) rather than the org-controlled NVIDIA/srt-slurm. This is mitigated by pinning to an exact 40-hex commit SHA and asserting git rev-parse HEAD matches it before proceeding, and the PR description itself states this is a temporary, draft-only state pending the real upstream merge SHA. Still, until that swap happens, CI on official gb200/gb300 power-lane runs executes code from a non-org-controlled repository. No other injection/auth-bypass/data-exposure concerns stood out — the new grep-based recipe detection only reads local repo files, and the sed/yaml edits are scoped to generated config, not user-controlled input.
Level of scrutiny
This warrants a higher level of scrutiny than a typical recipe or config PR: it touches the shared launcher scripts (runners/launch_gb200-nv.sh, runners/launch_gb300-nv.sh) that every model/precision/framework combination on those clusters depends on, and it is explicitly flagged by the author as draft/not-yet-final (fork pin instead of upstream pin, pending a second smoke test on both platforms). A human familiar with the srt-slurm producer and the gb200/gb300 CI paths should confirm the pin-swap plan and re-validate before this leaves draft.
Other factors
The author has already done meaningful validation (real GB200/GB300 runs with power_valid=1 and stored==recomputed sidecars, 124 local tests passing) and added static contract tests enforcing the launcher/recipe/workflow text stays in sync. The bug-hunting system found no bugs. Given the explicit draft status and the fork-pin caveat called out by the author themselves, this PR is not yet intended to be merged as-is, so deferring for human sign-off rather than approving is the right call.
edwingao28
changed the title
43d76f6 to
2daca89
Compare
…in gb launchers 中文:launcher 按 recipe 判定 power lane,provision exporter 并 pin producer;非 power lane 行为不变。
…de template 中文:CI 从 launcher stamp 读取 producer SHA,workflow input 保留为手动覆盖。
中文:两个官方 1P1D recipe 声明 telemetry;gb300 用 19401 端口避开集群级 exporter。
中文:契约测试覆盖 lane 检测、恰好两个 power recipe、pin 单一来源与 stamp 一致性。
86c65bb to
048e4e4
Compare
1 participant
Merge strategy: this PR merges with a fork pin. The producer pin points at edwingao28/srt-slurm@6fc1bed (the reviewed v2 producer branch; immutable SHA, additionally protected by fork branch
wenyao/pr2-pin-validated-v2). The upstream NVIDIA/srt-slurm PR proceeds independently; once it merges, a one-line follow-up PR swaps URL+SHA to upstream and re-smokes both platforms. Pin-swap re-smoke evidence for 6fc1bed: GB200 30969382477 (full ladder, 8/8 power_valid=1, manifest producer==6fc1bed, offline validator exit 0), GB300 30981806370 (full ladder, 8/8 power_valid=1, manifest producer==6fc1bed). The N=3 ladder tables below were measured under the functionally identical v1 pin (6609d46); the v2 runs reproduce them (gb200 c4 5.180 vs N=3 mean 5.219; gb300 c4 4.586 vs 4.593, c128 1.011 vs 1.010).中文:本 PR 以 fork pin 合入(SHA 内容寻址不可变 + 兜底保护分支),上游 PR 独立推进,合入后一个 one-line 跟进 PR 换成 upstream SHA 并双平台 re-smoke。
Stacked on #2437 (multinode consumer), which stacks on #2323 (single-node).
What this adds
Scope: 1P1D only. No nvidia-master.yaml change. Dispatches for this PR use exact-key test-config, not full-sweep.
Validation so far (fork pin): GB200 run 30618706258 (340.5 W/GPU avg, 193,404 J), GB300 run 30663050396 (349.3 W/GPU avg, 172,800 J), both with power_valid=1 and stored==recomputed sidecars. Local: 124 tests pass.
中文:为 gb200/gb300 官方 1P1D lane 打开 DCGM 能耗采集。recipe 声明 telemetry(gb300 用 19401 端口),launcher 按 recipe 判定是否 provision exporter 与 pin producer(非 power lane 路径不变),CI 从 launcher stamp 读取 POWER_PRODUCER_SHA。当前 pin 指向已验证的 fork SHA,upstream merge 后换 pin 并重跑双平台 c4 smoke,再转正式 review。
Official ladder results (D1 runs on this branch, fork pin; one dispatch per platform runs the recipe's full concurrency sweep): GB200 30951410151, GB300 30951412632. All 16 windows power_valid=1, manifest producer SHA == pin, stored == recomputed sidecars, per-GPU sample gaps ≤ 1.03 s.
中文:上表为两平台正式 ladder(每平台一次 dispatch 覆盖 c1–128 全部 8 窗)。GB300 每输出 token 能耗全程低于 GB200(c4 低 ~10.7%,c128 低 ~6.4%),且 TPOT 同步更优;两平台到 c128 尚未出现能效饱和。
Reproducibility, N=3 per platform (mean over 3 independent dispatches; CV% in parentheses). GB200: 30951410151, 30960242894, 30962794356. GB300: 30951412632, 30958592389, 30958596147. All 48 windows power_valid=1. Worst CV across the 16 (platform, conc) cells: 1.59% on J/out-tok, 1.10% on W/GPU.
中文:每平台 3 次独立 dispatch(重新部署、重新采样)后按窗口聚合。16 个 (平台, 并发) 单元格里 J/out-tok 的 CV 最差 1.59%(gb200 c4),其余全部 ≤0.77%;W/GPU 最差 1.10%。仪器与结果均可复现。两次 gb200 失败 run(30958590082/30958594280)为同集群并发 dispatch 导致的冷启动互踩、非仪器问题,已剔除并以串行重跑替代。