Repository navigation
[PowerX] collect NVL72 CPU and per-host GPU power / [PowerX] 采集 NVL72 CPU 与逐主机 GPU 功耗 - #3296
Open
edwingao28 wants to merge 56 commits into
Open
edwingao28 wants to merge 56 commits into
edwingao28 wants to merge 56 commits into
Conversation
Read srt-slurm's power/cpu/{samples.csv,cpu_manifest.json} (v1 long and v2
wide) next to the GPU DCGM package, integrate each (host, socket) Grace-side
and module series over the bound formal window with the shared trapezoid and
gap rule, and emit cpu_power_valid plus avg_cpu_socket_power_w,
avg_total_cpu_power_w, total_cpu_energy_j and the module keys. The verdict is
independent of power_valid and REQUIRE_POWER; a package without cpu/ produces
byte-identical output (GB200 fixture replay).
中文：在多节点功耗校验器中接入 srt-slurm 的 power/cpu/ 子产物包（v1 长格式与 v2
宽格式），按 (主机, socket) 在同一正式测量窗口内用共享的梯形法和采样间隔规则积分
Grace 侧及模块功耗，输出 cpu_power_valid 与 avg_cpu_socket_power_w、
avg_total_cpu_power_w、total_cpu_energy_j 及模块字段；结论独立于 power_valid 和
REQUIRE_POWER，无 cpu/ 的产物包输出逐字节不变（GB200 fixture 回放验证）。
validate_and_integrate() forced cpu_power_valid=0 with cpu_producer_unverified
whenever the GPU leg's producer pin failed, so a run with valid Grace samples
lost its CPU energy for a GPU-side reason, against the ticket contract
("independent of power_valid") and the spec's independent-leg rule. Drop that
gate and the unused blocking_reasons hook: the CPU leg borrows only the bound
formal window and the host topology as inputs, and no GPU verdict reaches it.
Fold the pin-mismatch and pin-missing cases into the existing invalid-GPU-leg
test. Docs (EN/ZH) drop the contradictory sentence and list the codes the leg
actually emits, including the shared non_finite_power_metric and
aggregate_result_* ones. GB200 fixture replay stays byte-identical.
中文：validate_and_integrate() 原本在 GPU 侧 producer 固定版本校验失败时，用
cpu_producer_unverified 强制 cpu_power_valid=0，导致 Grace 采样有效的运行因 GPU 侧
原因丢失 CPU 能耗，与工单契约（“独立于 power_valid”）和 spec 的独立测量环节规则相悖。
移除该门控及未使用的 blocking_reasons 参数：CPU 侧只借用已绑定的正式窗口和主机拓扑
作为输入，GPU 侧的任何结论都不会传导过来。把固定版本不匹配与缺失两种情形并入现有的
GPU 侧无效测试。中英文文档删除自相矛盾的表述，并列出实际输出的原因码，包括共享的
non_finite_power_metric 与 aggregate_result_*。GB200 fixture 回放输出逐字节不变。
…anes Add telemetry.cpu_power_exporter (port 9405, source acpi) to the 1P1D TP4/TP4 STP recipes of qwen3.5-fp8-gb200-dynamo-sglang and qwen3.5-fp8-gb300-dynamo-sglang, the smallest active non-Kimi-K3 NVL72 power lanes; nothing else in the recipes changes. srt-slurm v2.2.1 runs the exporter on the bare worker host and writes power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package for the multinode validator's CPU leg. Append the perf-changelog entry. DSV4 8k1k lanes are not used because that scenario was deprecated after 2026-09-08. 中文：在 qwen3.5-fp8-gb200-dynamo-sglang 与 qwen3.5-fp8-gb300-dynamo-sglang 的 1P1D TP4/TP4 STP recipe 中加入 telemetry.cpu_power_exporter（port 9405， source acpi），这是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane；recipe 其余内容不变。srt-slurm v2.2.1 在 worker 裸主机上运行 exporter，并在 GPU DCGM 产物包旁写入 power/cpu/samples.csv 与 cpu_manifest.json，供多节点 校验器的 CPU 侧测量环节使用。追加 perf-changelog 条目。未采用 DSV4 8k1k lane，因为该场景已于 2026-09-08 之后弃用。
…e tail Bring in main (96bccfe), including the #3252 and #3254 image bumps of the two Qwen3.5 FP8 NVL72 recipes this branch edits. perf-changelog.yaml is resolved as main's bytes followed by this branch's entry so the file stays append-only. 中文：合并 main（96bccfe12），包含 #3252 与 #3254 对本分支所改两份 Qwen3.5 FP8 NVL72 recipe 的镜像更新。perf-changelog.yaml 按 main 的原始字节加本分支 条目的方式解决冲突，保持仅追加。
Replace the pr-link placeholder in the appended perf-changelog.yaml entry with the opened PR URL; no other bytes change. 中文：将新增 perf-changelog.yaml 条目中的 pr-link 占位符替换为已开启的 PR 链接，其余内容不变。
…e tail Synchronize with main so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
edwingao28
added
the
full-sweep-fail-fast
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35632024733 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35632024733 |
…e tail Synchronize with main before re-pinning srt-slurm so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：在重新固定 srt-slurm 之前同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
…branch The first sweep showed the pinned cpu-power-exporter exiting with "no ACPI power_meter hwmon sensors found" on every GB200 node: on 6.8.0-nvidia-64k the acpi_power_meter hwmon node has no `name` attribute, only `device/name`. Pin utils/srt-slurm to SemiAnalysisAI/srt-slurm#23 head 473ecaa8 (984180e5 plus the gap-budget fix and the discovery fix) so the CPU leg can publish; the wire contract stays v1 long, so no consumer change. 中文：首次 sweep 显示固定版本的 cpu-power-exporter 在所有 GB200 节点上以 "no ACPI power_meter hwmon sensors found" 退出：6.8.0-nvidia-64k 内核下 acpi_power_meter 的 hwmon 节点没有 `name` 属性，只有 `device/name`。将 utils/srt-slurm 固定到 SemiAnalysisAI/srt-slurm#23 的 head 473ecaa8（984180e5 + 采样间隙预算修复 + 传感器发现修复），使 CPU 侧采集可以发布；CSV 契约仍为 v1 长表，消费端无需改动。
…e tail Synchronize with main before re-pinning srt-slurm so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：在重新固定 srt-slurm 之前同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
Enable best-effort CPU telemetry on 13 existing GPU-power recipes and retain validated physical-host GPU power. Keep GPU validation and serving settings intact. 为 13 份已有 GPU 功耗配方准备独立 CPU 遥测，并保留通过验证的逐物理主机 GPU 功耗。保持 GPU 校验与服务设置不变。
Collaborator
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
Open
42 tasks
Preserve per-host power and CPU sensor validation while adopting the current project layout and srt-slurm pin. Keep the published changelog history unchanged. 同步当前主线的项目目录与 srt-slurm 版本，保留逐主机功耗和 CPU 传感器校验；历史 changelog 字节保持不变。
edwingao28
removed
the
full-sweep-fail-fast
edwingao28
changed the title
同步当前 main 并保留 noprof GPU 配置与独立 CPU 采集；在既有配方入口固定已修复 ACPI 发现的 exporter v2.40.2，并验证安装失败传播。
应用上游 #553 的提交检查修复，令启用 CPU 采集的配方在 exporter 缺失、不可执行或已知架构不匹配时于 sbatch 前失败，并明确源码采集链路与二进制版本的区别。
中文：同步主分支并保留 NVL72 功耗采集设置。
中文：同步主分支并保留 NVL72 功耗采集设置。
要求 NVL72 扫描提供完整 CPU 功耗，并补齐 Qwen 采集与时钟主机配置。
edwingao28
changed the title
中文：在 AgentX 结果中保留 CPU 功耗审计信息。
8 of 11 tasks
中文：合并 main 并保留 NVL72 功耗变更记录。
中文：同步 NVL72 CPU 功耗分支与主线。
edwingao28
marked this pull request as ready for review
October 8, 2026 22:54
cquil11
force-pushed
the
feat/powerx-nvl72-cpu-power
branch
4 times, most recently
from
October 14, 2026 04:44
b8db6f0 to
8988d91
Compare
cquil11
changed the title
中文：将 main 的配方重构合并到 NVL72 CPU 功耗采集分支。
… with cpu-power The binder merges telemetry.cpu_power_exporter from the cluster record when a master row sets cpu-power beside power, so recipes and per-point settings no longer carry it; make setup pins the exporter release only for those jobs. 中文：Grace CPU 功耗 exporter 改由 cluster 记录声明，master 行用 cpu-power 选择；recipe 与逐点 additional-settings 不再携带，make setup 仅对这些任务固定 exporter 版本。
srt-slurm v2.51.1 cpu_samples.py:110 writes only CPU_SAMPLES_HEADER and binds no module meter, so the v1 long format, the OEM-label regexes and the module and DCGM headline paths never ran. The headline comes from the CPU<n>:cpuSidePowerUsageW rows alone and the two module keys leave CPU_METRIC_KEYS. 中文：CPU leg 仅读取宽格式 Grace socket 采样；删除从未执行的 v1 长格式、OEM 标签正则以及 module/DCGM headline 路径，并从 CPU_METRIC_KEYS 移除两个 module 指标。
…ing the bound recipe The point's cpu-power switch already decides whether the exporter is bound (PowerDecision.cpu), so the launcher no longer re-reads the recipe or writes POWER_EXPECTED_CPU_SOURCE to GITHUB_ENV, and the processors take --expected-cpu-power or the workflow's CPU_POWER. The source check goes with it: the cluster record requires a source (clusters/slurm.py CpuPowerExporter), the binder always writes it, and the Grace leg accepts only its own sensor. 中文：CPU leg 是否必需改由 cpu-power 开关决定；launcher 不再回读 recipe 或写 GITHUB_ENV，处理器改用 --expected-cpu-power 或 workflow 的 CPU_POWER。cluster 记录必须声明 source 且 binder 总会写入，故删除多余的 source 核对。
…trust field The CPU leg now asks _window_contract_holds directly on the serialized window instead of carrying formal_window_trusted across the audit, MAX_SAMPLE_GAP_SECONDS has one definition in common.py, and patch_power_metrics goes because _patch_agg became its only caller's replacement on this branch. 中文：CPU leg 直接用序列化窗口调用 _window_contract_holds，不再在 audit 上携带 formal_window_trusted；MAX_SAMPLE_GAP_SECONDS 只在 common.py 定义一处；删除在本分支上已无调用方的 patch_power_metrics。
AGENT_OPERATIONS.md carries the Grace sensor table Oren asked for (whole-socket meter, CPU and SysIO rails, no module series) in place of the link to the deleted results-and-ingestion.md, and the #3296 changelog entry describes the final diff. 中文：AGENT_OPERATIONS.md 补充 Oren 要求的 Grace 传感器说明（整 socket 读数、CPU/SysIO rail、无 module 序列），替换指向已删除文档的链接；#3296 的 changelog 条目改为描述最终 diff。
…ndow srt-slurm power/windows.py _check_coverage has emitted sample_loss_exceeded beside the three other coverage codes since v2.39.3 (#537), so a window that only lost GPU samples keeps binding the CPU leg instead of reporting cpu_window_unavailable. 中文：srt-slurm 自 v2.39.3 起在同一覆盖检查中输出 sample_loss_exceeded；仅丢失 GPU 采样的窗口仍可绑定 CPU leg，而不再报告 cpu_window_unavailable。
…or owns validation.py _cpu_power_rides_on_power rejects cpu-power without power on every search-space entry, and CPU_POWER reaches the launcher only from that validated matrix row. 中文：删除启动器中重复的 cpu-power 检查；validation.py 已在所有 search-space 条目上拒绝缺少 power 的 cpu-power，启动器的 CPU_POWER 只来自已校验的矩阵行。
cpu_side.py integrates only CPU<n>:cpuSidePowerUsageW rows, which srt-slurm emits from ACPI; a dcgm source always ended in cpu_socket_count_mismatch. 中文：CPU exporter 来源只接受 ACPI；cpu_side.py 只积分 ACPI 输出的 CPU<n>:cpuSidePowerUsageW 行，dcgm 来源必然以 cpu_socket_count_mismatch 失败。
srt-slurm documents cpu_manifest.json as non-authoritative debugging metadata and only logs a failed teardown write (cpu_session.py:218), so a broken manifest no longer invalidates a leg whose samples cover the window. 中文：CPU leg 不再依据 cpu_manifest.json 判定；srt-slurm 将其定义为非权威调试元数据且写入失败只记日志，样本已覆盖窗口时不再因 manifest 损坏而判为无效。
…es the head default srt-slurm schema.py:1162 already places the benchmark client on head, the dedicated frontend node. 中文：删除 Qwen3.5 8P1D recipe 中重复默认值的 benchmark placement；srt-slurm schema.py:1162 默认即在 head（即 dedicated frontend 节点）运行 benchmark client。
fixed_sequence.py and the AgentX adapter call run() with expected_cpu directly; nothing invokes multinode.py as a command. 中文：删除 multinode CLI 中无人传入的 --expected-cpu-power；fixed_sequence.py 与 AgentX adapter 直接以 expected_cpu 调用 run()，没有任何地方以命令行方式调用 multinode.py。
…idators Both schemas declare cpu_power and power, so the getattr defaults implied fields that could be absent. 中文：cpu-power 校验与同文件其他校验器一致，直接读取字段；两个 schema 都声明 cpu_power 与 power，getattr 默认值暗示字段可能缺失。
The release bump then edits one constant instead of a test literal. 中文：测试改用 CPU_POWER_EXPORTER_RELEASE 常量断言 setup 版本，升级版本时只需修改一处常量。
中文：合并 main 到 NVL72 CPU 功耗采集分支；perf-changelog 仅保留 main 原字节并在末尾重新追加本 PR 条目。
中文：合并 main 到 NVL72 CPU 功耗采集分支；perf-changelog 仅保留 main 原字节并在末尾重新追加本 PR 条目。
v2 files are still accepted, and module power is a reference rail that is never summed into the CPU totals. 中文：从 CPU 采样 v3 中发布 NVML module 功耗。仍兼容 v2 文件；module 功耗仅作为参考 rail，不计入 CPU 总功耗。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Collect same-window GPU and Grace CPU/LPDDR5X power across the selected NVL72 recipes. Reject missing or invalid requested CPU telemetry during required-power validation; keep GPU validity independent. Keep the 8P1D benchmark and collector on the same clock host.
Testing: Focused regressions and saved GB300 diagnostic replay pass. Eight points verified through repeat ingest, API, dashboard and JSON/CSV export.
NVL72 check at
ec74d5d27(Qwen3.5 FP8 Dynamo-SGLang 8k1k, 1P1D TP4, conc 4); replaying both bundles through head323371bebgives identical aggregates:Blocker: Full PR sweep/evals remain pending.
中文
为选定的 NVL72 配方采集同一窗口内的 GPU 与 Grace CPU/LPDDR5X 功耗。必需功耗验收会拒绝缺失或无效的预期 CPU 遥测，GPU 有效性仍独立判断。让 8P1D 基准与采集器使用同一时钟主机。
测试： 针对性回归与上方 GB300 诊断产物回放通过。8 个点完成重复入库、API、页面及 JSON/CSV 导出验收。
NVL72 验收： 在
ec74d5d27上 GB200（38634305913）与 GB300（38634310511）各运行一次，GPU 功耗有效（8/8 GPU）、发布校验通过、CPU 功耗有效（4/4 Grace socket）。每 socket Grace total / CPU rail / SysIO 分别为 99.3 / 47.3 / 6.0 W 与 97.9 / 48.3 / 6.3 W。用 head323371beb回放这两次运行的产物，aggregate 完全一致。module 功耗待 srt-slurm#609 合并并升级 pin。阻塞： 完整 PR sweep/eval 尚未完成。
AI 模型： GPT-6（精确变体无法核实）、claude-fable-5-1、claude-opus-5-5；更早版本无法核实。用于实现、整合、验证与文档。无关联 issue；涉及修复、新功能、配置及文档更新。
AI model disclosure
Related Issue
None.
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.