From 2366dc484a51bc24b3331cda8a25ad562109d025 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Fri, 18 Sep 2026 15:30:53 -0700 Subject: [PATCH 01/35] feat: integrate Grace CPU-side power from the multinode cpu/ package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read srt-slurm's power/cpu/{samples.csv,cpu_manifest.json} (v1 long and v2 wide) next to the GPU DCGM package, integrate each (host, socket) Grace-side and module series over the bound formal window with the shared trapezoid and gap rule, and emit cpu_power_valid plus avg_cpu_socket_power_w, avg_total_cpu_power_w, total_cpu_energy_j and the module keys. The verdict is independent of power_valid and REQUIRE_POWER; a package without cpu/ produces byte-identical output (GB200 fixture replay). 中文：在多节点功耗校验器中接入 srt-slurm 的 power/cpu/ 子产物包（v1 长格式与 v2 宽格式），按 (主机, socket) 在同一正式测量窗口内用共享的梯形法和采样间隔规则积分 Grace 侧及模块功耗，输出 cpu_power_valid 与 avg_cpu_socket_power_w、 avg_total_cpu_power_w、total_cpu_energy_j 及模块字段；结论独立于 power_valid 和 REQUIRE_POWER，无 cpu/ 的产物包输出逐字节不变（GB200 fixture 回放验证）。 --- .github/AGENT_OPERATIONS.md | 2 + docs/results-and-ingestion.md | 47 +++- docs/results-and-ingestion_zh.md | 42 ++- infx/results/power/__init__.py | 33 ++- infx/results/power/audit.py | 39 ++- infx/results/power/common.py | 14 + infx/results/power/cpu_side.py | 325 ++++++++++++++++++++++++ infx/results/power/multinode.py | 123 +++++++-- utils/test_aggregate_power_multinode.py | 289 +++++++++++++++++++++ 9 files changed, 885 insertions(+), 29 deletions(-) create mode 100644 infx/results/power/cpu_side.py diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 92aa00e9e9..36a35773f7 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -94,6 +94,8 @@ Single-node fixed-sequence results may include `power_valid`, `avg_power_w`, `av Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count. +NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, `total_cpu_energy_j`, and, when the module sensor is exposed on every socket, `avg_total_module_power_w` and `total_module_energy_j`. That verdict is independent of `power_valid` and of `REQUIRE_POWER`; see [`docs/results-and-ingestion.md`](../docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). + Every power result — valid or invalid, single-node or multinode — carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. For srt-slurm recipes, `telemetry.enabled: true` with `telemetry.dcgm_exporter` enables official energy collection. The Git submodule pointer at `utils/srt-slurm` is the source of truth for the shared srt-slurm commit, used by both power and non-power NVIDIA lanes. TileRT is the single documented fork exception. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible recipe-gated `dynamo-sglang` dcgm-power lanes are validated. diff --git a/docs/results-and-ingestion.md b/docs/results-and-ingestion.md index b6791cf6b2..942eaf4340 100644 --- a/docs/results-and-ingestion.md +++ b/docs/results-and-ingestion.md @@ -101,7 +101,7 @@ InferenceX-app treats routing fields as columns or config dimensions and stores The serving client records `benchmark_outcome` before saving its raw result. It retains the existing maximum request-failure rate of 5%, including the requested/completed/failed counts. The processor verifies this record, copies it to the aggregate, and returns failure even when telemetry is valid. Zero successful requests retain a diagnostic aggregate without fabricated reciprocal latency. Invalid request counts retain a failed diagnostic outcome with the raw `requested`/`completed` values and an `error`, without a fabricated failed count or rate; the client saves the raw JSON before exiting and the processor still rejects it. Legacy results without outcome metadata remain distinguishable; power validity alone never establishes benchmark success or answer quality. -`power_invalid_reasons` and `power_audit` carry a bounded summary alongside numeric metrics. The summary includes the available measurement window, expected/observed GPU counts, sampling diagnostics, observed device identifiers and producer pin. Its `source` names the retained `power_validation_*.json` sidecar. Device identifiers retain the collector's semantics; local SMI indices are not physical UUID proof. +`power_invalid_reasons` and `power_audit` carry a bounded summary alongside numeric metrics. The summary includes the available measurement window, expected/observed GPU counts, sampling diagnostics, observed device identifiers and producer pin. Its `source` names the retained `power_validation_*.json` sidecar. Device identifiers retain the collector's semantics; local SMI indices are not physical UUID proof. When the package carries the Grace CPU-side leg, `power_audit.cpu` adds its sensor kind, source, socket counts, row count, and reason codes; see [Measured Grace CPU-side power (NVL72)](#measured-grace-cpu-side-power-nvl72). For multinode fixed-sequence jobs, `utils/process_result.py --all` processes every available result before returning failure. It accepts `_c_gpus_...`, `_conc_gpus_...`, and AMD `_concurrency__req_rate__gpus_...` filenames, including `inf` request rates. It compares result concurrencies with `CONC_LIST`, rejects duplicate or contradictory point identities, and records omissions/errors in `result_processing_.json`. Aggregate workers pass `AGGREGATE_GPUS` with zero role GPU counts to telemetry validation; separate prefill/decode energy remains absent. For a `DISAGG=true` group with zero decode workers, the aggregate row intentionally sets `disagg: false` and reports `num_aggregate_gpu`; the filename, artifact name, and workflow inputs retain the group identity. Downstream consumers should use the row topology to interpret the measurement. @@ -435,6 +435,51 @@ All four values are withheld when telemetry validation fails. Older results rema missing until their original raw traces can be replayed; average watts cannot supply P75 or P90. The validation sidecar records `power_percentile_method`. +## Measured Grace CPU-side power (NVL72) + +GB200 and GB300 NVL72 recipes that enable srt-slurm's `telemetry.cpu_power_exporter` leg write +`LOGS/power/cpu/samples.csv` and a non-authoritative `cpu_manifest.json` beside the GPU DCGM +package. The multinode validator accepts the pinned v2.2.1 long format (one row per sensor +reading, header `schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`) +and the later wide format (one row per scrape, host, and socket, with `cpu_rail_w`, `soc_w`, and +`dram_w` reference columns). It classifies each row by its `sensor` cell: `Module Power Socket N` +is the whole-module reading (Grace, both Blackwell GPUs, HBM, LPDDR5X, and regulator loss); +`Grace Power Socket N` or `CPU:cpuSidePowerUsageW` is the Grace-side socket total (CPU, SoC, +and LPDDR5X); `CPU:cpuPowerUsageW` is DCGM field 1130, the CPU rail only. Component rails +(`CPU Power Socket N`, `SysIO Power Socket N`, DRAM) never feed a published metric. + +Per socket the headline series is chosen in the order module, Grace socket total, DCGM CPU rail, +and one kind must be present for every socket. Every fed series is integrated over the same bound +formal window as GPU energy with the same trapezoid, linear boundary interpolation, and 3.0 s +maximum sample gap. Expected sockets are two per distinct worker host in the manifest topology +(one compute tray per host). The additive keys, rounded like the GPU keys and stripped on re-run, +are: + +| Key | Meaning | +| --- | --- | +| `cpu_power_valid` | `1` or `0` for the CPU-side leg; absent when the package has no `cpu/` | +| `avg_cpu_socket_power_w` | Mean over sockets of each socket's window-mean Grace-side watts | +| `avg_total_cpu_power_w` | Sum over sockets of window-mean Grace-side watts | +| `total_cpu_energy_j` | Grace-side energy over the window, all sockets | +| `avg_total_module_power_w`, `total_module_energy_j` | Module watts and energy; only when a module series exists for every socket | + +The Grace-side keys come from the Grace socket total (or, without ACPI, the DCGM CPU rail) even +when the module sensor is preferred, so a module reading is never published as a Grace-side one. +The sidecar's `cpu` block and the aggregate's `power_audit.cpu` record `sensor_kind` (`module`, +`grace_socket`, or `dcgm_cpu_rail`), `source` (`acpi` or `dcgm`), expected and observed socket +counts, the parsed row count, and reason codes. `power_metric_schema_version` stays `2`. + +The leg is best effort and its verdict is independent of `power_valid`: any failure records +`cpu_power_valid: 0` with no CPU keys, leaves every GPU field byte-identical, and never fails +`REQUIRE_POWER=1`. Reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, +`cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch`, +`cpu_sensor_kind_mixed`, `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, +`cpu_window_unavailable` (no completed window binds to the result, or the window's own contract +checks failed), and `cpu_producer_unverified` (the srt-slurm producer pin failed, so CPU energy is +withheld like GPU energy). At the pinned v2.2.1 the exporter does not classify the Module label, +so current packages yield the Grace socket total; module keys appear once the upstream exporter +classifies it. A package without `cpu/` produces an aggregate and sidecar identical to today. + ## Verification and stop conditions A handoff is verified only when all applicable checks pass. diff --git a/docs/results-and-ingestion_zh.md b/docs/results-and-ingestion_zh.md index 38917f66d7..cc30d55ee3 100644 --- a/docs/results-and-ingestion_zh.md +++ b/docs/results-and-ingestion_zh.md @@ -101,7 +101,7 @@ InferenceX-app 将路由字段作为列或配置维度，并把数值测量存 服务客户端在保存原始结果前写入 `benchmark_outcome`，保留现有的 5% 最大请求失败率，以及请求总数、完成数和失败数。处理器检查该记录并复制到聚合结果中；即使遥测有效，请求失败率超限仍返回失败。零成功请求会保留诊断聚合结果，但不会生成不存在的延迟倒数。请求计数无效时，失败诊断状态保留原始 `requested`/`completed` 值和 `error`，不生成无依据的失败数或失败率；客户端先保存原始 JSON 再退出，处理器仍拒绝该结果。没有状态元数据的历史结果仍可区分；功耗有效不能证明基准成功或答案质量。 -`power_invalid_reasons` 和 `power_audit` 在数值指标旁携带有界摘要，包括可用的测量窗口、预期与观测 GPU 数、采样诊断、观测设备标识和生产者版本。`source` 指向保留的 `power_validation_*.json` 工件名称。设备标识保留采集器原有语义，本地 SMI 序号不是物理 UUID 的证明。 +`power_invalid_reasons` 和 `power_audit` 在数值指标旁携带有界摘要，包括可用的测量窗口、预期与观测 GPU 数、采样诊断、观测设备标识和生产者版本。`source` 指向保留的 `power_validation_*.json` 工件名称。设备标识保留采集器原有语义，本地 SMI 序号不是物理 UUID 的证明。当产物包包含 Grace CPU 侧测量环节时，`power_audit.cpu` 会附带其传感器类型、来源、socket 数、行数和原因码；见 [NVL72 Grace CPU 侧实测功耗](#nvl72-grace-cpu-侧实测功耗)。 对于多节点固定序列任务，`utils/process_result.py --all` 先处理所有已有结果，再返回失败。它接受 `_c_gpus_...`、`_conc_gpus_...` 和 AMD 的 `_concurrency__req_rate__gpus_...` 文件名，也支持 `inf` 请求速率。它将结果并发度与 `CONC_LIST` 比较，拒绝重复或矛盾的点身份，并将遗漏和错误记录到 `result_processing_.json`。共享工作池通过 `AGGREGATE_GPUS` 及零值角色 GPU 数进行遥测验证；独立的 prefill/decode 能耗保持缺失。当 `DISAGG=true` 的配置组中某个点没有 decode worker 时，聚合行会有意设置 `disagg: false` 并记录 `num_aggregate_gpu`；文件名、工件名和工作流输入仍保留配置组身份。下游应按聚合行的拓扑解释测量结果。 @@ -422,6 +422,46 @@ rm -rf -- "$tmp" 各设备分位数的平均值。遥测验证失败时，四项指标都不发布。旧结果需要使用原始 遥测重新计算；不能从平均功耗推算 P75 或 P90。验证 sidecar 会记录 `power_percentile_method`。 +## NVL72 Grace CPU 侧实测功耗 + +启用了 srt-slurm `telemetry.cpu_power_exporter` 的 GB200 和 GB300 NVL72 recipe 会在 GPU DCGM +产物包旁写入 `LOGS/power/cpu/samples.csv` 和一份非权威的 `cpu_manifest.json`。多节点校验器接受 +当前固定版本 v2.2.1 写出的长格式（每个传感器读数一行，表头为 +`schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`），以及后续 +版本的宽格式（每次采集、每台主机、每个 socket 一行，另带 `cpu_rail_w`、`soc_w`、`dram_w` 参考列）。 +它按 `sensor` 单元格对每行分类：`Module Power Socket N` 是整模块读数（Grace、两颗 Blackwell GPU、 +HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePowerUsageW` 是 Grace 侧 socket +总功耗（CPU、SoC 和 LPDDR5X）；`CPU:cpuPowerUsageW` 是 DCGM 字段 1130，仅含 CPU 供电轨。组件供电轨 +（`CPU Power Socket N`、`SysIO Power Socket N`、DRAM）永远不会进入发布指标。 + +每个 socket 的主序列按模块、Grace socket 总功耗、DCGM CPU 供电轨的顺序选取，且所有 socket 必须具备 +同一种传感器类型。每条参与计算的序列都在与 GPU 能耗相同的正式测量窗口内积分，采用同样的梯形法、 +线性边界插值和 3.0 秒最大采样间隔。预期 socket 数为 manifest 拓扑中每台不同 worker 主机两个（每台 +主机一个 compute tray）。新增字段与 GPU 字段采用相同的取整规则，重新运行时会先清除旧值： + +| 字段 | 含义 | +| --- | --- | +| `cpu_power_valid` | CPU 侧测量的有效性，`1` 或 `0`；产物包没有 `cpu/` 时不出现 | +| `avg_cpu_socket_power_w` | 各 socket 窗口平均 Grace 侧功耗的均值 | +| `avg_total_cpu_power_w` | 各 socket 窗口平均 Grace 侧功耗之和 | +| `total_cpu_energy_j` | 窗口内所有 socket 的 Grace 侧能耗 | +| `avg_total_module_power_w`、`total_module_energy_j` | 模块功耗与能耗；仅当每个 socket 都有模块序列时输出 | + +即使优先选用模块传感器，Grace 侧字段仍来自 Grace socket 总功耗（没有 ACPI 时来自 DCGM CPU 供电轨）， +因此模块读数永远不会被当作 Grace 侧读数发布。sidecar 的 `cpu` 块和聚合结果的 `power_audit.cpu` +记录 `sensor_kind`（`module`、`grace_socket` 或 `dcgm_cpu_rail`）、`source`（`acpi` 或 `dcgm`）、 +预期与观测 socket 数、解析行数和原因码。`power_metric_schema_version` 保持为 `2`。 + +该测量环节尽力而为，其结论与 `power_valid` 相互独立：任何失败都会记录 `cpu_power_valid: 0` 且不输出 +CPU 字段，所有 GPU 字段保持逐字节不变，也不会使 `REQUIRE_POWER=1` 失败。原因码包括 +`cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、`cpu_manifest_invalid`、 +`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、`cpu_sample_gap_exceeded`、 +`cpu_window_not_bracketed`、`cpu_window_unavailable`（没有已完成的窗口与该结果绑定，或窗口自身的契约 +检查失败）以及 `cpu_producer_unverified`（srt-slurm producer 固定版本校验失败，CPU 能耗与 GPU 能耗一样 +不予发布）。在当前固定的 v2.2.1 版本中，exporter 尚不识别 Module 标签，因此现有产物包只会得到 Grace +socket 总功耗；上游 exporter 支持该标签后才会出现模块字段。没有 `cpu/` 的产物包产生的聚合结果和 +sidecar 与现在完全一致。 + ## 验证和停止条件 只有全部适用检查通过，交接才算验证完成。 diff --git a/infx/results/power/__init__.py b/infx/results/power/__init__.py index a396f9684b..6e5fb04510 100644 --- a/infx/results/power/__init__.py +++ b/infx/results/power/__init__.py @@ -34,6 +34,21 @@ "decode_joules_per_output_token", ) ALL_POWER_METRIC_KEYS = WHOLE_METRIC_KEYS + ROLE_METRIC_KEYS +# Additive Grace CPU-side metrics from the multinode cpu/ sub-package. They +# share schema version 2 and are governed by cpu_power_valid, not power_valid. +CPU_METRIC_KEYS = ( + "avg_cpu_socket_power_w", + "avg_total_cpu_power_w", + "total_cpu_energy_j", + "avg_total_module_power_w", + "total_module_energy_j", +) + + +def _rounded_metric(key: str, value: float | None) -> float: + if value is None or not math.isfinite(value): + raise ValueError(f"non-finite power metric: {key}") + return round(value, 3 if key.endswith(("_w", "_j")) else 6) def with_power_metrics( @@ -43,17 +58,23 @@ def with_power_metrics( schema_version: int, power_valid: bool, metrics: Mapping[str, float], + cpu_power_valid: bool | None = None, + cpu_metrics: Mapping[str, float] | None = None, ) -> dict[str, Any]: """Return a result with stale metrics replaced by one validated metric set. The caller supplies the metric family and schema version. Detailed invalid reasons belong in its validation sidecar, not the numeric metric payload. Neither the input result nor the supplied metrics is mutated. + + The CPU-side leg is tri-state: ``cpu_power_valid=None`` means the package + carried no ``cpu/`` sub-package, so only stale CPU keys are stripped and the + output stays byte-identical to a run without the leg. """ # Do not coerce malformed JSON (such as a list of pairs) into an object. # Leave non-mappings to fail on the same operations as the existing adapters. data = dict(result) if isinstance(result, Mapping) else result - for key in metric_keys: + for key in (*metric_keys, *CPU_METRIC_KEYS, "cpu_power_valid"): data.pop(key, None) data["power_metric_schema_version"] = schema_version data["power_valid"] = int(power_valid) @@ -61,8 +82,10 @@ def with_power_metrics( data.pop("power_audit", None) if power_valid: for key, value in metrics.items(): - if value is None or not math.isfinite(value): - raise ValueError(f"non-finite power metric: {key}") - precision = 3 if key.endswith(("_w", "_j")) else 6 - data[key] = round(value, precision) + data[key] = _rounded_metric(key, value) + if cpu_power_valid is not None: + data["cpu_power_valid"] = int(cpu_power_valid) + if cpu_power_valid: + for key, value in (cpu_metrics or {}).items(): + data[key] = _rounded_metric(key, value) return data diff --git a/infx/results/power/audit.py b/infx/results/power/audit.py index 443b18f3e5..a42134d852 100644 --- a/infx/results/power/audit.py +++ b/infx/results/power/audit.py @@ -7,6 +7,36 @@ from collections.abc import Mapping from typing import Any +from .cpu_side import HEADLINE_PREFERENCE + +_REASON_CODE = re.compile(r"[a-z][a-z0-9_]{0,63}") +_CPU_COUNT_FIELDS = ("expected_sockets", "observed_sockets", "sample_row_count") + + +def _bounded_reasons(values: Any) -> list[str]: + if not isinstance(values, list): + return [] + return [ + reason for reason in values if isinstance(reason, str) and _REASON_CODE.fullmatch(reason) + ][:32] + + +def _cpu_audit(cpu: Mapping[str, Any]) -> dict[str, Any]: + """Project the CPU-side provenance: sensor kind, source, counts, reasons.""" + summary: dict[str, Any] = {} + kind = cpu.get("sensor_kind") + if kind in HEADLINE_PREFERENCE: + summary["sensor_kind"] = kind + source = cpu.get("source") + if isinstance(source, str) and 0 < len(source) <= 32: + summary["source"] = source + for key in _CPU_COUNT_FIELDS: + value = cpu.get(key) + if type(value) is int and value >= 0: + summary[key] = value + summary["reason_codes"] = _bounded_reasons(cpu.get("reason_codes")) + return summary + def audit_summary(validation: Mapping[str, Any], source: str) -> dict[str, Any]: """Project the shared app audit contract without publishing raw telemetry.""" @@ -49,11 +79,10 @@ def audit_summary(validation: Mapping[str, Any], source: str) -> dict[str, Any]: audit["observed_gpu_ids"] = list( dict.fromkeys(str(value) for value in ids if 0 < len(str(value)) <= 128) )[:1024] + cpu = validation.get("cpu") + if isinstance(cpu, Mapping): + audit["cpu"] = _cpu_audit(cpu) return { - "power_invalid_reasons": [ - reason - for reason in validation.get("reasons", []) - if isinstance(reason, str) and re.fullmatch(r"[a-z][a-z0-9_]{0,63}", reason) - ][:32], + "power_invalid_reasons": _bounded_reasons(validation.get("reasons", [])), "power_audit": audit, } diff --git a/infx/results/power/common.py b/infx/results/power/common.py index 671f215989..f9e333f801 100644 --- a/infx/results/power/common.py +++ b/infx/results/power/common.py @@ -53,6 +53,16 @@ def _interpolate_power(samples: list[tuple[float, float]], timestamp: float) -> return left_power + fraction * (right_power - left_power) +def _bracketing_sequence(times: tuple[float, ...], start: float, end: float) -> list[float] | None: + """Samples covering ``[start, end]``: the last at/before, the inside, the first at/after.""" + before = [value for value in times if value <= start] + after = [value for value in times if value >= end] + if not before or not after: + return None + inside = [value for value in times if start < value < end] + return [before[-1], *inside, after[0]] + + def _integrate_device( samples: list[tuple[float, float]], *, @@ -242,6 +252,8 @@ def patch_power_metrics( metric_keys: Iterable[str], power_valid: bool, metrics: Mapping[str, float], + cpu_power_valid: bool | None = None, + cpu_metrics: Mapping[str, float] | None = None, ) -> None: """Validate replacement metrics before atomically updating an aggregate.""" data = json.loads(path.read_text(encoding="utf-8")) @@ -251,5 +263,7 @@ def patch_power_metrics( schema_version=POWER_METRIC_SCHEMA_VERSION, power_valid=power_valid, metrics=metrics, + cpu_power_valid=cpu_power_valid, + cpu_metrics=cpu_metrics, ) _write_json_atomic(path, data) diff --git a/infx/results/power/cpu_side.py b/infx/results/power/cpu_side.py new file mode 100644 index 0000000000..caf266de41 --- /dev/null +++ b/infx/results/power/cpu_side.py @@ -0,0 +1,325 @@ +"""Best-effort Grace CPU-side power leg of the multinode srt-slurm package. + +srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` and a +non-authoritative ``power/cpu/cpu_manifest.json`` beside the GPU DCGM package +(srt-slurm ``src/srtctl/core/power/cpu_session.py``). The pinned v2.2.1 +producer writes the v1 long format, one row per sensor reading; v2.15.0 +writes the v2 wide format, one row per (scrape, host, socket) with the ACPI +component rails as reference columns. Both are accepted here. + +Each (hostname, socket) headline series is integrated over the GPU leg's +bound formal window with the shared trapezoid and boundary interpolation and +published as additive metrics next to the GPU-board numbers. The leg is +best-effort by contract: any failure records ``cpu_power_valid=0`` with +reason codes and leaves every GPU field untouched, and ``REQUIRE_POWER`` +never fails a run because of it. A package without ``cpu/`` emits nothing. +""" + +from __future__ import annotations + +import csv +import itertools +import json +import math +import re +from collections.abc import Iterable +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from .common import _append_reason, _bracketing_sequence, _integrate_device, audit_metrics + +CPU_DIRNAME = "cpu" +CPU_SAMPLES_FILENAME = "samples.csv" +CPU_MANIFEST_FILENAME = "cpu_manifest.json" + +# srt-slurm contract.py: CPU_SAMPLES_HEADER_V1 (written by v2.2.1) and the +# wide CPU_SAMPLES_HEADER (written by v2.15.0, which reads both). +CPU_SAMPLES_HEADER_V1 = ( + "schema_version", + "timestamp_unix", + "hostname", + "source", + "sensor", + "socket_id", + "power_w", + "total_power_w", +) +CPU_SAMPLES_HEADER_V2 = ( + *CPU_SAMPLES_HEADER_V1[:7], + "cpu_rail_w", + "soc_w", + "dram_w", + "total_power_w", +) +_HEADERS = {1: CPU_SAMPLES_HEADER_V1, 2: CPU_SAMPLES_HEADER_V2} + +# One GB200/GB300 compute tray per worker host, two Grace sockets per tray. +SOCKETS_PER_HOST = 2 +# Shared with the GPU leg (srt-slurm contract.MAX_SAMPLE_GAP_SECONDS). +MAX_SAMPLE_GAP_SECONDS = 3.0 + +SENSOR_MODULE = "module" +SENSOR_GRACE = "grace_socket" +SENSOR_DCGM = "dcgm_cpu_rail" +# Headline preference per socket. Component rails (cpu_rail, soc, dram) are +# reference breakdowns and never feed a published metric. +HEADLINE_PREFERENCE = (SENSOR_MODULE, SENSOR_GRACE, SENSOR_DCGM) +# Grace-side kinds, in fallback order, that feed the ``*_cpu_*`` keys. +_GRACE_SIDE_PREFERENCE = (SENSOR_GRACE, SENSOR_DCGM) + +# ``CPU:`` sensor names (srt-slurm cpu_rails.SENSOR_SUFFIXES) +# and firmware OEM labels (srt-slurm cpu_rails.ACPI_LABEL_PATTERNS for the +# Grace total). The Module label is the NVL72 tray firmware's whole-module +# sensor, which srt-slurm does not classify yet; a v2.2.1 package therefore +# yields the Grace socket total until the exporter learns the label. +_SUFFIX_KINDS = {"cpuSidePowerUsageW": SENSOR_GRACE, "cpuPowerUsageW": SENSOR_DCGM} +_LABEL_KINDS = ( + (SENSOR_MODULE, re.compile(r"\bModule\s+Power\s+Socket\s+\d+\b", re.IGNORECASE)), + (SENSOR_GRACE, re.compile(r"\bGrace\s+Power\s+Socket\s+\d+\b", re.IGNORECASE)), + ( + SENSOR_GRACE, + re.compile(r"\bTotal(?:\s+Input)?\s+Power(?:\s+in\s+uW)?\s+Socket\s+\d+\b", re.IGNORECASE), + ), +) + +_INTEGRATION_METHOD = "per_socket_trapezoidal_with_linear_boundary_interpolation" + + +@dataclass(frozen=True) +class CpuSampleRow: + timestamp_unix: float + hostname: str + source: str + sensor: str + socket_id: int + power_w: float + + +@dataclass +class CpuPowerAudit: + """Everything the sidecar needs to explain the CPU-side verdict.""" + + valid: bool = False + sensor_kind: str | None = None + source: str | None = None + expected_sockets: int = 0 + observed_sockets: int = 0 + sample_row_count: int = 0 + reason_codes: list[str] = field(default_factory=list) + per_series_energy_j: dict[str, float] = field(default_factory=dict) + per_series_max_sample_gap_s: dict[str, float] = field(default_factory=dict) + metrics: dict[str, float] = field(default_factory=dict) + + def invalidate(self, *reasons: str) -> None: + for reason in reasons: + _append_reason(self.reason_codes, reason) + self.valid = False + self.metrics = {} + + def to_payload(self) -> dict[str, Any]: + return { + "cpu_power_valid": self.valid, + "reason_codes": list(self.reason_codes), + "sensor_kind": self.sensor_kind, + "source": self.source, + "sockets_per_host": SOCKETS_PER_HOST, + "expected_sockets": self.expected_sockets, + "observed_sockets": self.observed_sockets, + "sample_row_count": self.sample_row_count, + "headline_preference": list(HEADLINE_PREFERENCE), + "integration_method": _INTEGRATION_METHOD, + "max_sample_gap_seconds": MAX_SAMPLE_GAP_SECONDS, + "per_series_max_sample_gap_s": dict(self.per_series_max_sample_gap_s), + # Overflowed integrations reach here as inf; keep the sidecar strict JSON. + "per_series_energy_j": { + key: value if math.isfinite(value) else None + for key, value in self.per_series_energy_j.items() + }, + "metrics": audit_metrics(self.metrics), + } + + +def classify_sensor(sensor: str) -> str | None: + """Headline kind for a ``sensor`` cell from either writer version, else None.""" + _, _, suffix = sensor.partition(":") + if suffix in _SUFFIX_KINDS: + return _SUFFIX_KINDS[suffix] + return next((kind for kind, pattern in _LABEL_KINDS if pattern.search(sensor)), None) + + +def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: + if len(raw) != len(_HEADERS[version]): + return None + try: + schema_version = int(raw[0]) + timestamp_unix = float(raw[1]) + socket_id = int(raw[5]) + power_w = float(raw[6]) + except ValueError: + return None + hostname, source, sensor = raw[2], raw[3], raw[4] + if schema_version != version or not hostname or not source or not sensor: + return None + if not math.isfinite(timestamp_unix) or not math.isfinite(power_w) or power_w < 0: + return None + if socket_id < 0: + return None + return CpuSampleRow( + timestamp_unix=timestamp_unix, + hostname=hostname, + source=source, + sensor=sensor, + socket_id=socket_id, + power_w=power_w, + ) + + +def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, ...]]: + """Strictly parse either sample generation; malformed rows invalidate the leg.""" + if not path.is_file(): + return (), ("cpu_samples_missing",) + rows: list[CpuSampleRow] = [] + reasons: list[str] = [] + try: + with open(path, newline="", encoding="utf-8") as handle: + reader = csv.reader(handle) + header = next(reader, None) + version = next((v for v, h in _HEADERS.items() if header == list(h)), None) + if version is None: + return (), ("cpu_samples_header_mismatch",) + for raw in reader: + row = _parse_row(raw, version) + if row is None: + _append_reason(reasons, "cpu_samples_malformed") + continue + rows.append(row) + except (OSError, UnicodeDecodeError, csv.Error): + _append_reason(reasons, "cpu_samples_malformed") + return tuple(rows), tuple(reasons) + + +def _manifest_is_object(path: Path) -> bool: + try: + return isinstance(json.loads(path.read_text(encoding="utf-8")), dict) + except (OSError, UnicodeDecodeError, ValueError): + return False + + +SocketKey = tuple[str, int] + + +def _headline_series( + rows: Iterable[CpuSampleRow], +) -> tuple[dict[SocketKey, dict[str, dict[float, float]]], dict[str, set[str]], bool]: + """Group headline rows as (host, socket) -> kind -> {timestamp: watts}.""" + series: dict[SocketKey, dict[str, dict[float, float]]] = {} + sources: dict[str, set[str]] = {} + duplicate = False + for row in rows: + kind = classify_sensor(row.sensor) + if kind is None: + continue + points = series.setdefault((row.hostname, row.socket_id), {}).setdefault(kind, {}) + if row.timestamp_unix in points: + duplicate = True + points[row.timestamp_unix] = row.power_w + sources.setdefault(kind, set()).add(row.source) + return series, sources, duplicate + + +def _select_feeds(common_kinds: set[str]) -> tuple[str | None, dict[str, str]]: + """Pick the headline kind and which kind feeds each metric family.""" + selected = next((kind for kind in HEADLINE_PREFERENCE if kind in common_kinds), None) + if selected is None: + return None, {} + if selected != SENSOR_MODULE: + return selected, {selected: "cpu"} + feeds = {SENSOR_MODULE: "module"} + grace = next((kind for kind in _GRACE_SIDE_PREFERENCE if kind in common_kinds), None) + if grace is not None: + feeds[grace] = "cpu" + return selected, feeds + + +def validate_cpu_leg( + cpu_dir: Path, + *, + window: tuple[float, float] | None, + expected_hosts: Iterable[str], + blocking_reasons: Iterable[str] = (), +) -> CpuPowerAudit | None: + """Validate and integrate the CPU sub-package; None when the run carried none.""" + if not cpu_dir.is_dir(): + return None + audit = CpuPowerAudit() + if not _manifest_is_object(cpu_dir / CPU_MANIFEST_FILENAME): + audit.invalidate("cpu_manifest_invalid") + rows, sample_reasons = read_cpu_samples(cpu_dir / CPU_SAMPLES_FILENAME) + audit.sample_row_count = len(rows) + audit.invalidate(*sample_reasons) + series, sources, duplicate = _headline_series(rows) + if duplicate: + audit.invalidate("cpu_samples_malformed") + expected_keys = { + (host, socket) for host in expected_hosts for socket in range(SOCKETS_PER_HOST) + } + audit.expected_sockets = len(expected_keys) + audit.observed_sockets = len(series) + audit.invalidate(*blocking_reasons) + if window is None: + audit.invalidate("cpu_window_unavailable") + if audit.reason_codes: + return audit + + if not expected_keys or set(series) != expected_keys: + audit.invalidate("cpu_socket_count_mismatch") + return audit + common_kinds = set.intersection(*(set(kinds) for kinds in series.values())) + selected, feeds = _select_feeds(common_kinds) + if selected is None or len(sources[selected]) != 1: + audit.invalidate("cpu_sensor_kind_mixed") + return audit + audit.sensor_kind = selected + audit.source = next(iter(sources[selected])) + + start, end = window + energy: dict[str, float] = {} + for (host, socket), by_kind in sorted(series.items()): + for kind, family in feeds.items(): + label = f"{host}/socket{socket}/{kind}" + samples = sorted(by_kind[kind].items()) + sequence = _bracketing_sequence(tuple(t for t, _ in samples), start, end) + if sequence is None: + audit.invalidate("cpu_window_not_bracketed") + continue + gap = max( + (later - earlier for earlier, later in itertools.pairwise(sequence)), + default=0.0, + ) + audit.per_series_max_sample_gap_s[label] = gap + if gap > MAX_SAMPLE_GAP_SECONDS: + audit.invalidate("cpu_sample_gap_exceeded") + continue + joules = _integrate_device(samples, start_unix=start, end_unix=end) + audit.per_series_energy_j[label] = joules + energy[family] = energy.get(family, 0.0) + joules + if audit.reason_codes: + audit.per_series_energy_j = {} + return audit + + duration = end - start + metrics: dict[str, float] = {} + if "cpu" in energy: + metrics["avg_cpu_socket_power_w"] = energy["cpu"] / duration / len(expected_keys) + metrics["avg_total_cpu_power_w"] = energy["cpu"] / duration + metrics["total_cpu_energy_j"] = energy["cpu"] + if "module" in energy: + metrics["avg_total_module_power_w"] = energy["module"] / duration + metrics["total_module_energy_j"] = energy["module"] + if any(not math.isfinite(value) for value in metrics.values()): + audit.invalidate("non_finite_power_metric") + return audit + audit.metrics = metrics + audit.valid = True + return audit diff --git a/infx/results/power/multinode.py b/infx/results/power/multinode.py index 756449b85a..f5af835b84 100644 --- a/infx/results/power/multinode.py +++ b/infx/results/power/multinode.py @@ -24,6 +24,12 @@ ``power_valid=0`` (and no energy metrics) in the aggregate plus a validation sidecar, but never fails the benchmark. Power studies set ``REQUIRE_POWER=1`` to fail after those audit artifacts exist. + +Grace CPU-side leg: when the package also carries ``power/cpu/`` (see +:mod:`.cpu_side`), each socket's Grace-side (and, when exposed, whole-module) +power is integrated over the same bound formal window and published under +``cpu_power_valid``. That verdict is independent of ``power_valid`` and of +``REQUIRE_POWER``; a package without ``cpu/`` produces identical output. """ from __future__ import annotations @@ -44,12 +50,14 @@ # import them through this module. from . import ( ALL_POWER_METRIC_KEYS as _ALL_POWER_METRIC_KEYS, + CPU_METRIC_KEYS as CPU_METRIC_KEYS, ROLE_METRIC_KEYS as ROLE_METRIC_KEYS, WHOLE_METRIC_KEYS as WHOLE_METRIC_KEYS, ) from .common import ( BenchmarkData, _append_reason, + _bracketing_sequence, _integrate_device, _load_benchmark_data, _percentile_total_power, @@ -58,6 +66,7 @@ benchmark_window_payload, patch_power_metrics, ) +from .cpu_side import CPU_DIRNAME, CpuPowerAudit, validate_cpu_leg # --- srt-slurm dcgm-power v1 wire contract (mirrored constants) ------------- @@ -711,15 +720,6 @@ def _check_window_result(window: ParsedWindow, result_root: Path) -> list[str]: return [] -def _bracketing_sequence(times: tuple[float, ...], start: float, end: float) -> list[float] | None: - before = [value for value in times if value <= start] - after = [value for value in times if value >= end] - if not before or not after: - return None - inside = [value for value in times if start < value < end] - return [before[-1], *inside, after[0]] - - def _check_coverage( start: float, end: float, @@ -896,6 +896,11 @@ class MultinodePowerAudit: observed_gpu_count: int = 0 expected_gpu_count: int = 0 metrics: dict[str, float] = field(default_factory=dict) + # Inputs the CPU-side leg borrows from the GPU package; not serialized. + expected_worker_hosts: tuple[str, ...] = () + formal_window_trusted: bool = False + # None when the package carried no cpu/ sub-package. + cpu: CpuPowerAudit | None = None def _canonical_sha256(payload: dict) -> str: @@ -921,7 +926,48 @@ def validate_and_integrate( aggregate_gpus: int, expected_producer_sha: str | None, ) -> MultinodePowerAudit: - """Recompute package validity, cross-check verdicts, and integrate energy.""" + """Recompute package validity, cross-check verdicts, and integrate energy. + + The GPU leg runs first and binds the formal window; the CPU-side leg then + reuses that window and the manifest topology. Only the producer pin and + the window binding are shared gates: an unpinned producer withholds CPU + energy too, while every other GPU verdict leaves the CPU leg untouched. + """ + audit = _validate_gpu_leg( + power_dir=power_dir, + logs_root=logs_root, + bench_result_path=bench_result_path, + benchmark=benchmark, + prefill_gpus=prefill_gpus, + decode_gpus=decode_gpus, + aggregate_gpus=aggregate_gpus, + expected_producer_sha=expected_producer_sha, + ) + pin_failed = {"producer_pin_missing", "producer_commit_mismatch"} & set(audit.reasons) + blocking = ["cpu_producer_unverified"] if pin_failed else [] + window = None + if audit.window is not None and audit.formal_window_trusted: + window = (audit.window["start_time_unix"], audit.window["end_time_unix"]) + audit.cpu = validate_cpu_leg( + power_dir / CPU_DIRNAME, + window=window, + expected_hosts=audit.expected_worker_hosts, + blocking_reasons=blocking, + ) + return audit + + +def _validate_gpu_leg( + *, + power_dir: Path, + logs_root: Path, + bench_result_path: Path, + benchmark: BenchmarkData | None, + prefill_gpus: int, + decode_gpus: int, + aggregate_gpus: int, + expected_producer_sha: str | None, +) -> MultinodePowerAudit: audit = MultinodePowerAudit( expected_producer_git_commit=expected_producer_sha, ) @@ -972,6 +1018,7 @@ def validate_and_integrate( _add_reason(audit, "manifest_malformed", f"manifest malformed: {exc!r}") audit.recomputed_publication_valid = False return audit + audit.expected_worker_hosts = tuple(sorted({device.hostname for device in expected_devices})) rows, sample_reasons = read_samples(power_dir / SAMPLES_FILENAME) recompute_failures += [f"{reason} in {SAMPLES_FILENAME}" for reason in sample_reasons] @@ -1078,6 +1125,8 @@ def validate_and_integrate( window = _select_window_for_result( audit, parsed_windows, logs_root, bench_result_path, benchmark ) + if window is not None: + audit.formal_window_trusted = _window_contract_holds(window, validations) if audit.reasons or window is None or benchmark is None: return audit @@ -1235,18 +1284,47 @@ def _select_window_for_result( return window +# Per-window audit codes that describe GPU sample coverage rather than the +# window's own contract; the CPU-side leg re-checks its own coverage. +_GPU_COVERAGE_REASONS = frozenset( + {"measurement_window_not_bracketed", "sample_gap_exceeded", "gpu_uuid_changed"} +) + + +def _window_contract_holds(window: ParsedWindow, validations: list[dict]) -> bool: + """True when the bound window passed every check that is not GPU coverage.""" + for validation in validations: + if (validation["benchmark_type"], validation["concurrency"]) == ( + window.benchmark_type, + window.concurrency, + ): + return all(reason in _GPU_COVERAGE_REASONS for reason in validation["reason_codes"]) + return False + + # --- aggregate patch + sidecar + entry point --------------------------------- def _patch_agg(agg_path: Path, audit: MultinodePowerAudit) -> None: + cpu = audit.cpu patch_power_metrics( agg_path, metric_keys=_ALL_POWER_METRIC_KEYS, power_valid=audit.power_valid, metrics=audit.metrics, + cpu_power_valid=None if cpu is None else cpu.valid, + cpu_metrics={} if cpu is None else cpu.metrics, ) +def _withhold_metrics(audit: MultinodePowerAudit, cpu_reason: str | None = None) -> None: + """Drop every publishable number; the CPU leg only when the failure is shared.""" + audit.power_valid = False + audit.metrics = {} + if audit.cpu is not None and cpu_reason is not None: + audit.cpu.invalidate(cpu_reason) + + def _sidecar_payload( *, audit: MultinodePowerAudit, @@ -1254,7 +1332,7 @@ def _sidecar_payload( bench_result: Path, benchmark: BenchmarkData | None, ) -> dict: - return { + payload = { "schema_version": 1, "power_valid": audit.power_valid, "reasons": list(audit.reasons), @@ -1284,6 +1362,10 @@ def _sidecar_payload( }, "metrics": audit_metrics(audit.metrics), } + # Only when the package carried cpu/: sidecars without it stay byte-identical. + if audit.cpu is not None: + payload["cpu"] = audit.cpu.to_payload() + return payload def run( @@ -1322,20 +1404,17 @@ def run( for reason in benchmark_reasons: _add_reason(audit, reason) if benchmark is None or audit.reasons: - audit.power_valid = False - audit.metrics = {} + _withhold_metrics(audit) if not agg_result.is_file(): _add_reason(audit, "aggregate_result_missing", f"aggregate missing: {agg_result}") - audit.power_valid = False - audit.metrics = {} + _withhold_metrics(audit, "aggregate_result_missing") else: try: _patch_agg(agg_result, audit) except (OSError, json.JSONDecodeError, ValueError) as exc: _add_reason(audit, "aggregate_result_unwritable", str(exc)) - audit.power_valid = False - audit.metrics = {} + _withhold_metrics(audit, "aggregate_result_unwritable") print( f"[aggregate_power_multinode] Failed to patch {agg_result}: {exc}", file=sys.stderr, @@ -1359,6 +1438,16 @@ def run( ) return 1 if require_power else 0 + if audit.cpu is not None: + cpu = audit.cpu + print( + f"[aggregate_power_multinode] cpu_power_valid={int(cpu.valid)} " + f"sensor_kind={cpu.sensor_kind} " + f"avg_total_cpu_power_w={cpu.metrics.get('avg_total_cpu_power_w', 0.0):.2f} " + f"total_cpu_energy_j={cpu.metrics.get('total_cpu_energy_j', 0.0):.2f} " + f"reasons={','.join(cpu.reason_codes) or '-'}" + ) + if not audit.power_valid: print( f"[aggregate_power_multinode] Power validation failed: " diff --git a/utils/test_aggregate_power_multinode.py b/utils/test_aggregate_power_multinode.py index 3235b88ebb..3810459cca 100644 --- a/utils/test_aggregate_power_multinode.py +++ b/utils/test_aggregate_power_multinode.py @@ -653,3 +653,292 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) rows, reasons = apm.read_samples(path) assert not rows assert reasons == ("samples_csv_malformed",) + + +# --- Grace CPU-side power leg (power/cpu/) ----------------------------------- + +CPU_HOSTS = tuple(sorted({host for host, *_ in DEVICES})) +CPU_HEADER_V1 = [ + "schema_version", "timestamp_unix", "hostname", "source", "sensor", + "socket_id", "power_w", "total_power_w", +] +CPU_HEADER_V2 = CPU_HEADER_V1[:7] + ["cpu_rail_w", "soc_w", "dram_w", "total_power_w"] +GRACE_W = {"node-d": 500.0, "node-p": 700.0} +MODULE_W = 1500.0 +DCGM_W = 300.0 +# Component rails never feed a headline metric; the values are chosen so any +# leak into a published key is visible against the Grace/module constants. +RAILS_W = {"cpu_rail": 200.0, "soc": 50.0, "dram": 30.0} +GRACE_KINDS = {"grace": None, **RAILS_W} + +_SENSORS = { + # (source, v1 firmware OEM label, v2 collector sensor name) + "grace": ("acpi", "Grace Power Socket {s}", "CPU{s}:cpuSidePowerUsageW"), + "module": ("acpi", "Module Power Socket {s}", "Module Power Socket {s}"), + "dcgm": ("dcgm", "CPU{s}:cpuPowerUsageW", "CPU{s}:cpuPowerUsageW"), + "cpu_rail": ("acpi", "CPU Power Socket {s}", "CPU{s}:cpuRailPowerUsageW"), + "soc": ("acpi", "SysIO Power Socket {s}", "CPU{s}:socPowerUsageW"), + "dram": ("acpi", "DRAM Power Socket {s}", "CPU{s}:dramPowerUsageW"), +} + + +def _cpu_row(ts, host, socket, kind, watts, fmt): + source, label_v1, label_v2 = _SENSORS[kind] + sensor = (label_v1 if fmt == "v1" else label_v2).format(s=socket) + if fmt == "v1": + return [1, repr(ts), host, source, sensor, socket, repr(watts), ""] + rails = ["", "", ""] if source == "dcgm" else [repr(w) for w in RAILS_W.values()] + return [2, repr(ts), host, source, sensor, socket, repr(watts), *rails, ""] + + +def _cpu_rows(kinds_by_host=None, fmt="v1", *, hosts=CPU_HOSTS): + """One row per (scrape, host, socket, kind); ``None`` watts means the Grace constant.""" + kinds_by_host = kinds_by_host or {host: GRACE_KINDS for host in hosts} + rows = [] + ts = FIRST_TS + while ts <= LAST_TS: + for host in hosts: + for socket in (0, 1): + for kind, watts in kinds_by_host.get(host, {}).items(): + watts = GRACE_W[host] if watts is None else watts + rows.append(_cpu_row(ts, host, socket, kind, watts, fmt)) + ts += 1.0 + return rows + + +def add_cpu_package(pkg, rows, header=CPU_HEADER_V1, manifest_text=None): + cpu_dir = pkg.power_dir / "cpu" + cpu_dir.mkdir() + with open(cpu_dir / "samples.csv", "w", newline="") as handle: + writer = csv.writer(handle) + writer.writerow(header) + writer.writerows(rows) + if manifest_text is None: + manifest_text = json.dumps( + { + "schema_version": 1, + "producer": "srt-slurm.cpu-power", + "producer_git_commit": PRODUCER_SHA, + "started_at_unix": 990.0, + "stopped_at_unix": 1070.0, + "nodes": { + host: {"resolved_mode": "acpi", "scrape_count": 65, "error_count": 0} + for host in CPU_HOSTS + }, + }, + indent=2, + ) + (cpu_dir / "cpu_manifest.json").write_text(manifest_text) + return cpu_dir + + +def _gpu_fields(agg): + return {k: v for k, v in agg.items() if k != "cpu_power_valid" and k not in apm.CPU_METRIC_KEYS} + + +def _reference_agg(tmp_path): + """The GPU-only aggregate every CPU-leg outcome must reproduce field for field.""" + reference = build_package(tmp_path / "reference") + assert reference.run() == 0 + return reference.agg() + + +def assert_grace_keys(agg): + assert agg["cpu_power_valid"] == 1 + assert agg["avg_cpu_socket_power_w"] == 600.0 + assert agg["avg_total_cpu_power_w"] == 2400.0 + assert agg["total_cpu_energy_j"] == 144000.0 + + +class TestCpuSidePower: + @pytest.mark.parametrize("fmt, header", [("v1", CPU_HEADER_V1), ("v2", CPU_HEADER_V2)]) + def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): + from infx.results.power.audit import audit_summary + + pkg = build_package(tmp_path) + kinds = None if fmt == "v1" else {host: {"grace": None} for host in CPU_HOSTS} + rows = _cpu_rows(kinds, fmt) + add_cpu_package(pkg, rows, header) + assert pkg.run(require_power=True) == 0 + + agg = pkg.agg() + assert_grace_keys(agg) + assert "avg_total_module_power_w" not in agg + assert "total_module_energy_j" not in agg + assert _gpu_fields(agg) == _reference_agg(tmp_path) + + cpu = pkg.sidecar()["cpu"] + assert cpu["cpu_power_valid"] is True + assert cpu["reason_codes"] == [] + assert cpu["sensor_kind"] == "grace_socket" + assert cpu["source"] == "acpi" + assert cpu["expected_sockets"] == 4 + assert cpu["observed_sockets"] == 4 + assert cpu["sample_row_count"] == len(rows) + assert set(cpu["per_series_energy_j"]) == { + f"{host}/socket{socket}/grace_socket" for host in CPU_HOSTS for socket in (0, 1) + } + assert audit_summary(pkg.sidecar(), "power_validation.json")["power_audit"]["cpu"] == { + "sensor_kind": "grace_socket", + "source": "acpi", + "expected_sockets": 4, + "observed_sockets": 4, + "sample_row_count": len(rows), + "reason_codes": [], + } + + def test_dcgm_only_package_uses_cpu_rail_kind(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) + assert pkg.run() == 0 + agg = pkg.agg() + assert agg["cpu_power_valid"] == 1 + assert agg["avg_cpu_socket_power_w"] == 300.0 + assert agg["avg_total_cpu_power_w"] == 1200.0 + assert agg["total_cpu_energy_j"] == 72000.0 + cpu = pkg.sidecar()["cpu"] + assert (cpu["sensor_kind"], cpu["source"]) == ("dcgm_cpu_rail", "dcgm") + + def test_module_on_every_socket_is_preferred_and_grace_keys_stay_grace(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package( + pkg, _cpu_rows({host: {**GRACE_KINDS, "module": MODULE_W} for host in CPU_HOSTS}) + ) + assert pkg.run() == 0 + agg = pkg.agg() + assert_grace_keys(agg) + assert agg["avg_total_module_power_w"] == 6000.0 + assert agg["total_module_energy_j"] == 360000.0 + cpu = pkg.sidecar()["cpu"] + assert cpu["sensor_kind"] == "module" + assert set(cpu["per_series_energy_j"]) == { + f"{host}/socket{socket}/{kind}" + for host in CPU_HOSTS + for socket in (0, 1) + for kind in ("module", "grace_socket") + } + + def test_module_on_some_sockets_falls_back_to_grace(self, tmp_path): + pkg = build_package(tmp_path) + kinds = {"node-d": {**GRACE_KINDS, "module": MODULE_W}, "node-p": GRACE_KINDS} + add_cpu_package(pkg, _cpu_rows(kinds)) + assert pkg.run() == 0 + agg = pkg.agg() + assert_grace_keys(agg) + assert "avg_total_module_power_w" not in agg + assert "total_module_energy_j" not in agg + assert pkg.sidecar()["cpu"]["sensor_kind"] == "grace_socket" + + def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): + pkg = build_package(tmp_path) + assert pkg.run() == 0 + agg = pkg.agg() + assert "cpu_power_valid" not in agg + assert set(apm.CPU_METRIC_KEYS).isdisjoint(agg) + assert "cpu" not in pkg.sidecar() + + def test_stale_cpu_keys_are_stripped_on_rerun(self, tmp_path): + pkg = build_package(tmp_path) + pkg.agg_result.write_text( + json.dumps({"hw": "gb200", "conc": 4, "cpu_power_valid": 1, "total_cpu_energy_j": 1.0}) + ) + assert pkg.run() == 0 + assert _gpu_fields(pkg.agg()) == _reference_agg(tmp_path) + assert "cpu_power_valid" not in pkg.agg() + assert "total_cpu_energy_j" not in pkg.agg() + + def _drop(self, rows, host, socket, predicate): + return [ + row + for row in rows + if not (row[2] == host and row[5] == socket and predicate(float(row[1]))) + ] + + @pytest.fixture + def tampered(self, request, tmp_path): + kind = request.param + pkg = build_package(tmp_path) + rows = _cpu_rows() + header, manifest_text = CPU_HEADER_V1, None + if kind == "header": + header = [*CPU_HEADER_V1[:6], "watts", "total_power_w"] + elif kind == "socket": + rows = self._drop(rows, "node-p", 1, lambda ts: True) + elif kind == "gap": + rows = self._drop(rows, "node-d", 0, lambda ts: 1020.0 <= ts <= 1024.0) + elif kind == "unbracketed": + rows = self._drop(rows, "node-d", 0, lambda ts: ts <= WINDOW_START) + elif kind == "mixed": + rows = _cpu_rows({"node-d": {"grace": None}, "node-p": {"dcgm": DCGM_W}}) + elif kind == "manifest": + manifest_text = "{broken" + elif kind == "malformed": + rows = [*rows, [1, repr(FIRST_TS), "node-d", "acpi", "Grace Power Socket 0", 0, "n/a", ""]] + cpu_dir = add_cpu_package(pkg, rows, header, manifest_text) + if kind == "samples_missing": + (cpu_dir / "samples.csv").unlink() + return pkg + + @pytest.mark.parametrize( + "tampered, reason", + [ + ("header", "cpu_samples_header_mismatch"), + ("socket", "cpu_socket_count_mismatch"), + ("gap", "cpu_sample_gap_exceeded"), + ("unbracketed", "cpu_window_not_bracketed"), + ("mixed", "cpu_sensor_kind_mixed"), + ("manifest", "cpu_manifest_invalid"), + ("malformed", "cpu_samples_malformed"), + ("samples_missing", "cpu_samples_missing"), + ], + indirect=["tampered"], + ) + def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, reason): + pkg = tampered + # REQUIRE_POWER guards the GPU leg only; a broken CPU leg never fails the run. + assert pkg.run(require_power=True) == 0 + agg = pkg.agg() + assert agg["cpu_power_valid"] == 0 + assert set(apm.CPU_METRIC_KEYS).isdisjoint(agg) + assert _gpu_fields(agg) == _reference_agg(tmp_path) + cpu = pkg.sidecar()["cpu"] + assert cpu["cpu_power_valid"] is False + assert reason in cpu["reason_codes"] + assert pkg.sidecar()["power_valid"] is True + + def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows()) + _rewrite_samples( + pkg, + lambda body: [ + row + for row in body + if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) + ], + ) + assert pkg.run() == 0 + agg = pkg.agg() + assert agg["power_valid"] == 0 + assert "total_gpu_energy_j" not in agg + assert_grace_keys(agg) + + def test_unpinned_producer_withholds_cpu_energy_too(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows()) + assert pkg.run(sha="b" * 40) == 0 + agg = pkg.agg() + assert agg["power_valid"] == 0 + assert agg["cpu_power_valid"] == 0 + assert set(apm.CPU_METRIC_KEYS).isdisjoint(agg) + assert "cpu_producer_unverified" in pkg.sidecar()["cpu"]["reason_codes"] + + def test_window_unavailable_when_result_binds_to_no_window(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows()) + tampered = dict(BENCH_FIELDS, max_concurrency=8) + pkg.original_result.write_text(json.dumps(tampered, indent=2)) + pkg.bench_result.write_text(json.dumps(tampered, indent=2)) + assert pkg.run() == 0 + assert pkg.agg()["cpu_power_valid"] == 0 + assert "cpu_window_unavailable" in pkg.sidecar()["cpu"]["reason_codes"] From f21ddd0d9751bff408aec4337ed4942be5d188f9 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Fri, 18 Sep 2026 15:58:09 -0700 Subject: [PATCH 02/35] fix: keep cpu_power_valid independent of the GPU producer pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit validate_and_integrate() forced cpu_power_valid=0 with cpu_producer_unverified whenever the GPU leg's producer pin failed, so a run with valid Grace samples lost its CPU energy for a GPU-side reason, against the ticket contract ("independent of power_valid") and the spec's independent-leg rule. Drop that gate and the unused blocking_reasons hook: the CPU leg borrows only the bound formal window and the host topology as inputs, and no GPU verdict reaches it. Fold the pin-mismatch and pin-missing cases into the existing invalid-GPU-leg test. Docs (EN/ZH) drop the contradictory sentence and list the codes the leg actually emits, including the shared non_finite_power_metric and aggregate_result_* ones. GB200 fixture replay stays byte-identical. 中文：validate_and_integrate() 原本在 GPU 侧 producer 固定版本校验失败时，用 cpu_producer_unverified 强制 cpu_power_valid=0，导致 Grace 采样有效的运行因 GPU 侧 原因丢失 CPU 能耗，与工单契约（“独立于 power_valid”）和 spec 的独立测量环节规则相悖。 移除该门控及未使用的 blocking_reasons 参数：CPU 侧只借用已绑定的正式窗口和主机拓扑 作为输入，GPU 侧的任何结论都不会传导过来。把固定版本不匹配与缺失两种情形并入现有的 GPU 侧无效测试。中英文文档删除自相矛盾的表述，并列出实际输出的原因码，包括共享的 non_finite_power_metric 与 aggregate_result_*。GB200 fixture 回放输出逐字节不变。 --- docs/results-and-ingestion.md | 20 +++++++----- docs/results-and-ingestion_zh.md | 17 ++++++---- infx/results/power/cpu_side.py | 2 -- infx/results/power/multinode.py | 9 ++---- utils/test_aggregate_power_multinode.py | 43 +++++++++++++------------ 5 files changed, 48 insertions(+), 43 deletions(-) diff --git a/docs/results-and-ingestion.md b/docs/results-and-ingestion.md index 942eaf4340..932cb504b2 100644 --- a/docs/results-and-ingestion.md +++ b/docs/results-and-ingestion.md @@ -469,14 +469,18 @@ The sidecar's `cpu` block and the aggregate's `power_audit.cpu` record `sensor_k `grace_socket`, or `dcgm_cpu_rail`), `source` (`acpi` or `dcgm`), expected and observed socket counts, the parsed row count, and reason codes. `power_metric_schema_version` stays `2`. -The leg is best effort and its verdict is independent of `power_valid`: any failure records -`cpu_power_valid: 0` with no CPU keys, leaves every GPU field byte-identical, and never fails -`REQUIRE_POWER=1`. Reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, -`cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch`, -`cpu_sensor_kind_mixed`, `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, -`cpu_window_unavailable` (no completed window binds to the result, or the window's own contract -checks failed), and `cpu_producer_unverified` (the srt-slurm producer pin failed, so CPU energy is -withheld like GPU energy). At the pinned v2.2.1 the exporter does not classify the Module label, +The leg is best effort and its verdict is independent of `power_valid`: it borrows only the bound +formal window and the worker-host topology from the GPU package, and no GPU verdict reaches it, so +an unpinned producer or failed GPU coverage withholds GPU energy while `cpu_power_valid` still +judges the CPU samples on their own. Any CPU-leg failure records `cpu_power_valid: 0` with no CPU +keys, leaves every GPU field byte-identical, and never fails `REQUIRE_POWER=1`. Reason codes: +`cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, +`cpu_manifest_invalid`, `cpu_socket_count_mismatch`, `cpu_sensor_kind_mixed`, +`cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, and `cpu_window_unavailable` (no completed +window binds to the result, or the window's own contract checks failed). An overflowed CPU +integration reuses the GPU leg's `non_finite_power_metric`, and `aggregate_result_missing` or +`aggregate_result_unwritable` appears in both audits when the aggregate itself cannot be patched. +At the pinned v2.2.1 the exporter does not classify the Module label, so current packages yield the Grace socket total; module keys appear once the upstream exporter classifies it. A package without `cpu/` produces an aggregate and sidecar identical to today. diff --git a/docs/results-and-ingestion_zh.md b/docs/results-and-ingestion_zh.md index cc30d55ee3..f9b32e100a 100644 --- a/docs/results-and-ingestion_zh.md +++ b/docs/results-and-ingestion_zh.md @@ -452,13 +452,16 @@ HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePow 记录 `sensor_kind`（`module`、`grace_socket` 或 `dcgm_cpu_rail`）、`source`（`acpi` 或 `dcgm`）、 预期与观测 socket 数、解析行数和原因码。`power_metric_schema_version` 保持为 `2`。 -该测量环节尽力而为，其结论与 `power_valid` 相互独立：任何失败都会记录 `cpu_power_valid: 0` 且不输出 -CPU 字段，所有 GPU 字段保持逐字节不变，也不会使 `REQUIRE_POWER=1` 失败。原因码包括 -`cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、`cpu_manifest_invalid`、 -`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、`cpu_sample_gap_exceeded`、 -`cpu_window_not_bracketed`、`cpu_window_unavailable`（没有已完成的窗口与该结果绑定，或窗口自身的契约 -检查失败）以及 `cpu_producer_unverified`（srt-slurm producer 固定版本校验失败，CPU 能耗与 GPU 能耗一样 -不予发布）。在当前固定的 v2.2.1 版本中，exporter 尚不识别 Module 标签，因此现有产物包只会得到 Grace +该测量环节尽力而为，其结论与 `power_valid` 相互独立：它只从 GPU 产物包借用已绑定的正式测量窗口和 +worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 producer 固定版本校验失败或 GPU 覆盖不足只会 +使 GPU 能耗不予发布，`cpu_power_valid` 仍按 CPU 采样自身给出结论。CPU 侧的任何失败都会记录 +`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持逐字节不变，也不会使 `REQUIRE_POWER=1` 失败。 +原因码包括 `cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、 +`cpu_manifest_invalid`、`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、 +`cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口 +与该结果绑定，或窗口自身的契约检查失败）。CPU 积分溢出沿用 GPU 侧的 `non_finite_power_metric`；聚合 +结果本身无法写入时，`aggregate_result_missing` 或 `aggregate_result_unwritable` 会同时出现在两侧的审计 +中。在当前固定的 v2.2.1 版本中，exporter 尚不识别 Module 标签，因此现有产物包只会得到 Grace socket 总功耗；上游 exporter 支持该标签后才会出现模块字段。没有 `cpu/` 的产物包产生的聚合结果和 sidecar 与现在完全一致。 diff --git a/infx/results/power/cpu_side.py b/infx/results/power/cpu_side.py index caf266de41..ca9dc07016 100644 --- a/infx/results/power/cpu_side.py +++ b/infx/results/power/cpu_side.py @@ -247,7 +247,6 @@ def validate_cpu_leg( *, window: tuple[float, float] | None, expected_hosts: Iterable[str], - blocking_reasons: Iterable[str] = (), ) -> CpuPowerAudit | None: """Validate and integrate the CPU sub-package; None when the run carried none.""" if not cpu_dir.is_dir(): @@ -266,7 +265,6 @@ def validate_cpu_leg( } audit.expected_sockets = len(expected_keys) audit.observed_sockets = len(series) - audit.invalidate(*blocking_reasons) if window is None: audit.invalidate("cpu_window_unavailable") if audit.reason_codes: diff --git a/infx/results/power/multinode.py b/infx/results/power/multinode.py index f5af835b84..c46758c978 100644 --- a/infx/results/power/multinode.py +++ b/infx/results/power/multinode.py @@ -929,9 +929,9 @@ def validate_and_integrate( """Recompute package validity, cross-check verdicts, and integrate energy. The GPU leg runs first and binds the formal window; the CPU-side leg then - reuses that window and the manifest topology. Only the producer pin and - the window binding are shared gates: an unpinned producer withholds CPU - energy too, while every other GPU verdict leaves the CPU leg untouched. + reuses that window and the manifest topology as inputs. No GPU verdict + reaches it: an unpinned producer or failed GPU coverage withholds GPU + energy only, and ``cpu_power_valid`` judges the CPU samples on their own. """ audit = _validate_gpu_leg( power_dir=power_dir, @@ -943,8 +943,6 @@ def validate_and_integrate( aggregate_gpus=aggregate_gpus, expected_producer_sha=expected_producer_sha, ) - pin_failed = {"producer_pin_missing", "producer_commit_mismatch"} & set(audit.reasons) - blocking = ["cpu_producer_unverified"] if pin_failed else [] window = None if audit.window is not None and audit.formal_window_trusted: window = (audit.window["start_time_unix"], audit.window["end_time_unix"]) @@ -952,7 +950,6 @@ def validate_and_integrate( power_dir / CPU_DIRNAME, window=window, expected_hosts=audit.expected_worker_hosts, - blocking_reasons=blocking, ) return audit diff --git a/utils/test_aggregate_power_multinode.py b/utils/test_aggregate_power_multinode.py index 3810459cca..b7b17468ea 100644 --- a/utils/test_aggregate_power_multinode.py +++ b/utils/test_aggregate_power_multinode.py @@ -906,32 +906,35 @@ def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, r assert reason in cpu["reason_codes"] assert pkg.sidecar()["power_valid"] is True - def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path): + @pytest.mark.parametrize( + "gpu_gap, sha, gpu_reason", + [ + (True, PRODUCER_SHA, "package_recompute_invalid"), + (False, "b" * 40, "producer_commit_mismatch"), + (False, None, "producer_pin_missing"), + ], + ) + def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path, gpu_gap, sha, gpu_reason): + """No GPU verdict, the producer pin included, reaches cpu_power_valid.""" pkg = build_package(tmp_path) add_cpu_package(pkg, _cpu_rows()) - _rewrite_samples( - pkg, - lambda body: [ - row - for row in body - if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) - ], - ) - assert pkg.run() == 0 + if gpu_gap: + _rewrite_samples( + pkg, + lambda body: [ + row + for row in body + if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) + ], + ) + assert pkg.run(sha=sha) == 0 agg = pkg.agg() assert agg["power_valid"] == 0 assert "total_gpu_energy_j" not in agg assert_grace_keys(agg) - - def test_unpinned_producer_withholds_cpu_energy_too(self, tmp_path): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows()) - assert pkg.run(sha="b" * 40) == 0 - agg = pkg.agg() - assert agg["power_valid"] == 0 - assert agg["cpu_power_valid"] == 0 - assert set(apm.CPU_METRIC_KEYS).isdisjoint(agg) - assert "cpu_producer_unverified" in pkg.sidecar()["cpu"]["reason_codes"] + sidecar = pkg.sidecar() + assert gpu_reason in sidecar["reasons"] + assert sidecar["cpu"]["reason_codes"] == [] def test_window_unavailable_when_result_binds_to_no_window(self, tmp_path): pkg = build_package(tmp_path) From cc62f0e01508285c17fdfbb65be500b49bb01c3f Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Fri, 18 Sep 2026 16:10:52 -0700 Subject: [PATCH 03/35] feat: enable srt-slurm cpu power leg on Qwen3.5 FP8 GB200 and GB300 lanes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add telemetry.cpu_power_exporter (port 9405, source acpi) to the 1P1D TP4/TP4 STP recipes of qwen3.5-fp8-gb200-dynamo-sglang and qwen3.5-fp8-gb300-dynamo-sglang, the smallest active non-Kimi-K3 NVL72 power lanes; nothing else in the recipes changes. srt-slurm v2.2.1 runs the exporter on the bare worker host and writes power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package for the multinode validator's CPU leg. Append the perf-changelog entry. DSV4 8k1k lanes are not used because that scenario was deprecated after 2026-09-08. 中文：在 qwen3.5-fp8-gb200-dynamo-sglang 与 qwen3.5-fp8-gb300-dynamo-sglang 的 1P1D TP4/TP4 STP recipe 中加入 telemetry.cpu_power_exporter（port 9405， source acpi），这是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane；recipe 其余内容不变。srt-slurm v2.2.1 在 worker 裸主机上运行 exporter，并在 GPU DCGM 产物包旁写入 power/cpu/samples.csv 与 cpu_manifest.json，供多节点 校验器的 CPU 侧测量环节使用。追加 perf-changelog 条目。未采用 DSV4 8k1k lane，因为该场景已于 2026-09-08 之后弃用。 --- .../gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 3 +++ .../gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 3 +++ perf-changelog.yaml | 14 ++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index d879bc8e45..4acbf80a3e 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -132,3 +132,6 @@ telemetry: dcgm_exporter: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index 2bc2fb0378..afe34a361e 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -134,3 +134,6 @@ telemetry: # 9401 is already bound by the cluster-level exporter on im-gb300 nodes; # use a port outside that range. port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/perf-changelog.yaml b/perf-changelog.yaml index ac580ce184..91ac3f659a 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8051,3 +8051,17 @@ - "Use SGLang nightly-dev-20260914-4358a161 with Dynamo 1.5.0.dev20260913 and retry truncated srt-slurm setup dependency downloads." - "Update telemetry to the current srt-slurm schema, preserving the one-second collection interval and using a schema-compliant 12-second collector join timeout." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3112 + +- config-keys: + - qwen3.5-fp8-gb200-dynamo-sglang + - qwen3.5-fp8-gb300-dynamo-sglang + scenario-type: + - fixed-seq-len + description: + - "Enable srt-slurm's independent, best-effort CPU power leg on the 1P1D TP4/TP4 STP Qwen3.5 FP8 GB200 and GB300 recipes: telemetry.cpu_power_exporter with port 9405 (the schema default) and source acpi. Nothing else in either recipe changes; image, serving flags, topology, concurrency list, and the GPU DCGM exporter (port 9401 on GB200, 19401 on GB300, required: true) are untouched." + - "At the pinned srt-slurm v2.2.1 (984180e5) srtctl launches cpu-power-exporter --port 9405 --source acpi through srun on the bare host of every worker node, falling back to the ACPI-only Python exporter when make setup did not install the release binary, and the head-node collector writes LOGS/power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package. The leg has no required flag: a missing hwmon sensor or a dead exporter only leaves gaps in samples.csv and cannot fail the job or the GPU power verdict." + - "The multinode power validator integrates the Grace Power Socket series over the same formal window as GPU energy and adds cpu_power_valid plus the CPU-side keys (docs/results-and-ingestion.md, Measured Grace CPU-side power); power_valid and every GPU field stay unchanged. The v2.2.1 exporter classifies only the CPU, Grace, and SysIO socket labels, so these runs yield the Grace-side total rather than the module reading." + - "These are the smallest active non-Kimi-K3 NVL72 power lanes (2 nodes, 1 prefill + 1 decode host, 4 Grace sockets). The DSV4 8k1k GB200/GB300 lanes are not used: Single-turn 8k1k on DeepSeek-V4-Pro was deprecated after 2026-09-08 (MODELS.md; configs/deprecated/nvidia-dsv4-8k1k-master.yaml), so no sweep would exercise them." + - "在 Qwen3.5 FP8 GB200 与 GB300 的 1P1D TP4/TP4 STP recipe 上启用 srt-slurm 独立、尽力而为的 CPU 功耗测量环节：telemetry.cpu_power_exporter，port 9405（schema 默认值），source acpi。两份 recipe 其余内容不变：镜像、服务参数、拓扑、并发列表以及 GPU DCGM exporter（GB200 端口 9401、GB300 端口 19401，required: true）均未改动。在固定版本 srt-slurm v2.2.1（984180e5）下，srtctl 通过 srun 在每个 worker 节点的裸主机上启动 cpu-power-exporter --port 9405 --source acpi，若 make setup 未安装发布的二进制则回退到仅支持 ACPI 的 Python exporter；头节点采集器把 LOGS/power/cpu/samples.csv 和 cpu_manifest.json 写在 GPU DCGM 产物包旁。该环节没有 required 开关：缺少 hwmon 传感器或 exporter 失效只会让 samples.csv 出现空缺，不会导致作业失败或改变 GPU 功耗结论。多节点功耗校验器在与 GPU 能耗相同的正式测量窗口内对 Grace Power Socket 序列积分，新增 cpu_power_valid 与 CPU 侧字段（见 docs/results-and-ingestion_zh.md「NVL72 Grace CPU 侧实测功耗」）；power_valid 与所有 GPU 字段保持不变。v2.2.1 的 exporter 只识别 CPU、Grace、SysIO 三种 socket 标签，因此这些运行得到的是 Grace 侧总功耗而非模块读数。" + - "选择这两条 lane 是因为它们是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane（2 个节点，1 台 prefill 主机 + 1 台 decode 主机，4 个 Grace socket）。未采用 DSV4 8k1k 的 GB200/GB300 lane：DeepSeek-V4-Pro 的单轮 8k1k 场景已于 2026-09-08 之后弃用（见 MODELS.md 与 configs/deprecated/nvidia-dsv4-8k1k-master.yaml），sweep 不会再执行它们。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX From d365901f3020bd07417f71ee9784ed64df7d80d7 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 13:29:03 -0700 Subject: [PATCH 04/35] chore: link the cpu power leg changelog entry to PR #3296 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the pr-link placeholder in the appended perf-changelog.yaml entry with the opened PR URL; no other bytes change. 中文：将新增 perf-changelog.yaml 条目中的 pr-link 占位符替换为已开启的 PR 链接，其余内容不变。 --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index cbd5a25796..f15673e6ef 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8372,4 +8372,4 @@ - "These are the smallest active non-Kimi-K3 NVL72 power lanes (2 nodes, 1 prefill + 1 decode host, 4 Grace sockets). The DSV4 8k1k GB200/GB300 lanes are not used: Single-turn 8k1k on DeepSeek-V4-Pro was deprecated after 2026-09-08 (MODELS.md; configs/deprecated/nvidia-dsv4-8k1k-master.yaml), so no sweep would exercise them." - "在 Qwen3.5 FP8 GB200 与 GB300 的 1P1D TP4/TP4 STP recipe 上启用 srt-slurm 独立、尽力而为的 CPU 功耗测量环节：telemetry.cpu_power_exporter，port 9405（schema 默认值），source acpi。两份 recipe 其余内容不变：镜像、服务参数、拓扑、并发列表以及 GPU DCGM exporter（GB200 端口 9401、GB300 端口 19401，required: true）均未改动。在固定版本 srt-slurm v2.2.1（984180e5）下，srtctl 通过 srun 在每个 worker 节点的裸主机上启动 cpu-power-exporter --port 9405 --source acpi，若 make setup 未安装发布的二进制则回退到仅支持 ACPI 的 Python exporter；头节点采集器把 LOGS/power/cpu/samples.csv 和 cpu_manifest.json 写在 GPU DCGM 产物包旁。该环节没有 required 开关：缺少 hwmon 传感器或 exporter 失效只会让 samples.csv 出现空缺，不会导致作业失败或改变 GPU 功耗结论。多节点功耗校验器在与 GPU 能耗相同的正式测量窗口内对 Grace Power Socket 序列积分，新增 cpu_power_valid 与 CPU 侧字段（见 docs/results-and-ingestion_zh.md「NVL72 Grace CPU 侧实测功耗」）；power_valid 与所有 GPU 字段保持不变。v2.2.1 的 exporter 只识别 CPU、Grace、SysIO 三种 socket 标签，因此这些运行得到的是 Grace 侧总功耗而非模块读数。" - "选择这两条 lane 是因为它们是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane（2 个节点，1 台 prefill 主机 + 1 台 decode 主机，4 个 Grace socket）。未采用 DSV4 8k1k 的 GB200/GB300 lane：DeepSeek-V4-Pro 的单轮 8k1k 场景已于 2026-09-08 之后弃用（见 MODELS.md 与 configs/deprecated/nvidia-dsv4-8k1k-master.yaml），sweep 不会再执行它们。" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From b8f85f465bf2455fa374f1c840d88d090c268daa Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Mon, 21 Sep 2026 10:21:14 -0700 Subject: [PATCH 05/35] fix: pin srt-slurm to the hwmon name discovery fix on the gap-budget branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first sweep showed the pinned cpu-power-exporter exiting with "no ACPI power_meter hwmon sensors found" on every GB200 node: on 6.8.0-nvidia-64k the acpi_power_meter hwmon node has no `name` attribute, only `device/name`. Pin utils/srt-slurm to SemiAnalysisAI/srt-slurm#23 head 473ecaa8 (984180e5 plus the gap-budget fix and the discovery fix) so the CPU leg can publish; the wire contract stays v1 long, so no consumer change. 中文：首次 sweep 显示固定版本的 cpu-power-exporter 在所有 GB200 节点上以 "no ACPI power_meter hwmon sensors found" 退出：6.8.0-nvidia-64k 内核下 acpi_power_meter 的 hwmon 节点没有 `name` 属性，只有 `device/name`。将 utils/srt-slurm 固定到 SemiAnalysisAI/srt-slurm#23 的 head 473ecaa8（984180e5 + 采样间隙预算修复 + 传感器发现修复），使 CPU 侧采集可以发布；CSV 契约仍为 v1 长表，消费端无需改动。 --- utils/srt-slurm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/srt-slurm b/utils/srt-slurm index 984180e5b8..473ecaa8fb 160000 --- a/utils/srt-slurm +++ b/utils/srt-slurm @@ -1 +1 @@ -Subproject commit 984180e5b8755aef85e9995048b5a16cb5336bce +Subproject commit 473ecaa8fbd1d409867d492c0271ba154e1dcaa4 From b0fb074bfcb487fa80cbfa027cc5f95a4c124987 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Mon, 21 Sep 2026 13:00:00 -0700 Subject: [PATCH 06/35] feat: prepare Kimi K3 NVL72 CPU telemetry and host power MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enable best-effort CPU telemetry on 13 existing GPU-power recipes and retain validated physical-host GPU power. Keep GPU validation and serving settings intact. 为 13 份已有 GPU 功耗配方准备独立 CPU 遥测，并保留通过验证的逐物理主机 GPU 功耗。保持 GPU 校验与服务设置不变。 --- .../agg-dcp16-dspark4-maxseq2-mooncake.yaml | 4 ++ .../agentx/agg-dcp16-nospec-mooncake.yaml | 4 ++ .../agentx/agg-tp8pp2-mooncake-c16.yaml | 4 ++ .../agentx/agg-tp8pp2-mooncake-c32.yaml | 4 ++ .../agentx/agg-tp8pp2-mooncake-c48.yaml | 4 ++ .../agentx/agg-tp8pp2-mooncake-c72.yaml | 4 ++ .../agentx/agg-tp8pp2-mooncake-c96.yaml | 4 ++ .../agentx/agg-dcp8-dspark4-mooncake.yaml | 4 ++ .../agg-dcp8-dspark7-maxseq2-mooncake.yaml | 4 ++ ...isagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml | 7 +++- ...isagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml | 7 +++- ...isagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml | 7 +++- ...isagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml | 7 +++- configs/nvidia-master.yaml | 5 +++ docs/results-and-ingestion.md | 8 ++++ docs/results-and-ingestion_zh.md | 6 +++ infx/results/power/multinode.py | 38 +++++++++++++++++-- perf-changelog.yaml | 15 ++++++++ utils/test_aggregate_power_multinode.py | 33 ++++++++++++++++ 19 files changed, 158 insertions(+), 11 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml index ceb0efb874..8ac47e3cf6 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml @@ -155,6 +155,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml index 81af8a3005..dbd195a81c 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml @@ -151,6 +151,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml index 33f3115102..3a026dc761 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml @@ -124,6 +124,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml index cf98995eec..7aa094d569 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml @@ -124,6 +124,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml index 57498c8969..8e2f9a064f 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml @@ -123,6 +123,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml index c7e1da7937..cf46403ea4 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml @@ -123,6 +123,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml index 2bb53eaee4..9a87c63ce6 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml @@ -123,6 +123,10 @@ telemetry: container_image: dcgm-exporter port: 9401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml index bac87abe62..22dc3d7df0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml @@ -142,6 +142,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom concurrencies: [48, 52, 56] diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml index 113a947f37..efa89bd84e 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml @@ -140,6 +140,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom concurrencies: [1, 4] diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml index dd9a30f069..3ef238ac2c 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml @@ -174,8 +174,7 @@ srun_options: telemetry: enabled: true - provider: dcgm-power - default_frequency: 1.0 + collect_interval_ms: 1000 storage_subdir: power required: true startup_timeout_seconds: 120 @@ -185,6 +184,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml index 6ea49208c6..e3e6108119 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml @@ -174,8 +174,7 @@ srun_options: telemetry: enabled: true - provider: dcgm-power - default_frequency: 1.0 + collect_interval_ms: 1000 storage_subdir: power required: true startup_timeout_seconds: 120 @@ -185,6 +184,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml index 4bf4ecc656..747fffa25b 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml @@ -174,8 +174,7 @@ srun_options: telemetry: enabled: true - provider: dcgm-power - default_frequency: 1.0 + collect_interval_ms: 1000 storage_subdir: power required: true startup_timeout_seconds: 120 @@ -185,6 +184,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml index 0cd2a16883..f1710d03ad 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml @@ -174,8 +174,7 @@ srun_options: telemetry: enabled: true - provider: dcgm-power - default_frequency: 1.0 + collect_interval_ms: 1000 storage_subdir: power required: true startup_timeout_seconds: 120 @@ -185,6 +184,10 @@ telemetry: container_image: dcgm-exporter port: 19401 + cpu_power_exporter: + port: 9405 + source: acpi + benchmark: type: custom client_placement: head diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index 1215a1a5f7..5651aa87b7 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -6166,6 +6166,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dep16-vllm-simple-offload.yaml" # Kimi-K3 GB200 TP16/DCP16 profiles using Mooncake DRAM offload. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: + # Referenced power recipes also collect best-effort Grace CPU telemetry. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6193,6 +6194,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml" kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: + # Referenced power recipes also collect best-effort Grace CPU telemetry. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6338,6 +6340,7 @@ dsv4-fp4-gb300-dynamo-trt-agentx: # Add GB200 TP8 x PP2/DCP8 AgentX operating points. The runtime topology uses # 16 GPUs total, allocated across four 4-GPU GB200 nodes. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: + # Referenced power recipes also collect best-effort Grace CPU telemetry. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -7594,6 +7597,7 @@ qwen3.5-fp8-gb200-dynamo-sglang-mtp: # Kimi-K3 GB300 points on the measured global p90-ITL Pareto front. Each # DCP8 worker spans two 4-GPU nodes. kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: + # Referenced power recipes also collect best-effort Grace CPU telemetry. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -7664,6 +7668,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: + # Referenced power recipes also collect best-effort Grace CPU telemetry. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 diff --git a/docs/results-and-ingestion.md b/docs/results-and-ingestion.md index 156bb787b0..c938399a8d 100644 --- a/docs/results-and-ingestion.md +++ b/docs/results-and-ingestion.md @@ -194,6 +194,14 @@ When that measurement-window contract is absent, the aggregate records `power_valid: 0` and the audit names `multinode_power_contract_missing`; `REQUIRE_POWER=1` also fails the job after preserving available results. +Validated multinode aggregates also publish `workers` power entries grouped by +physical hostname and role. `num_gpus` counts the audited device UUIDs on that +host; `avg_power_w` is their integrated energy divided by the same formal window +and GPU count. A serving worker spanning two hosts produces two power entries; +these indices do not replace the serving topology's worker counts. Revalidation +removes old entries before publishing the current verdict, and invalid GPU power +publishes no host power. CPU telemetry keeps its independent validity gate. + Treat `power_valid: 1` with `power_metric_schema_version: 2` as a GPU telemetry verdict, not a request-accounting or model-quality verdict. Before using a point as a clean comparison, reconcile issued, completed, cancelled, and errored diff --git a/docs/results-and-ingestion_zh.md b/docs/results-and-ingestion_zh.md index e287ceb223..9514b55407 100644 --- a/docs/results-and-ingestion_zh.md +++ b/docs/results-and-ingestion_zh.md @@ -189,6 +189,12 @@ raw tree: results/**, excluding inputs.json and profile_export_raw.jso 缺少测量窗口接口时，聚合结果记录 `power_valid: 0`，审计原因标记为 `multinode_power_contract_missing`；设置 `REQUIRE_POWER=1` 还会在保留已有结果后使任务失败。 +通过验证的多节点聚合结果还会按物理主机和角色输出 `workers` 功耗记录。 +`num_gpus` 取该主机审计确认的 GPU UUID 数量；`avg_power_w` 为这些 GPU 在 +同一正式测量窗口内的积分能耗除以窗口时长和卡数。跨两台主机的服务 worker 输出 +两条功耗记录，这里的索引不替代服务拓扑中的 worker 数量。重新校验会先删除旧 +记录；GPU 功耗无效时不发布逐主机功耗。CPU 遥测仍使用独立的有效性判断。 + `power_valid: 1` 与 `power_metric_schema_version: 2` 只表示 GPU 遥测有效， 不代表请求计数或模型质量通过验证。用于可靠对比前，应将已发出、已完成、已取消及 出错请求数与原始 profiling 记录和 token 总数核对。GPU 板卡能耗与整机功耗估算分开报告。 diff --git a/infx/results/power/multinode.py b/infx/results/power/multinode.py index c46758c978..9ec7b72a87 100644 --- a/infx/results/power/multinode.py +++ b/infx/results/power/multinode.py @@ -51,8 +51,10 @@ from . import ( ALL_POWER_METRIC_KEYS as _ALL_POWER_METRIC_KEYS, CPU_METRIC_KEYS as CPU_METRIC_KEYS, + POWER_METRIC_SCHEMA_VERSION, ROLE_METRIC_KEYS as ROLE_METRIC_KEYS, WHOLE_METRIC_KEYS as WHOLE_METRIC_KEYS, + with_power_metrics, ) from .common import ( BenchmarkData, @@ -64,7 +66,6 @@ _write_json_atomic, audit_metrics, benchmark_window_payload, - patch_power_metrics, ) from .cpu_side import CPU_DIRNAME, CpuPowerAudit, validate_cpu_leg @@ -1302,16 +1303,47 @@ def _window_contract_holds(window: ParsedWindow, validations: list[dict]) -> boo # --- aggregate patch + sidecar + entry point --------------------------------- +def _host_workers(audit: MultinodePowerAudit) -> list[dict]: + """Project validated device energy onto physical host/role power entries.""" + if not audit.power_valid or audit.window is None: + return [] + energies: dict[tuple[str, str], list[float]] = {} + for device, joules in audit.per_gpu_energy_j.items(): + host = device.rsplit("/", 1)[0] + role = audit.per_gpu_role[device] + energies.setdefault((role, host), []).append(joules) + workers = [] + indices: dict[str, int] = {} + for (role, host), values in sorted(energies.items()): + index = indices.get(role, 0) + workers.append( + { + "role": role, + "worker_idx": index, + "hosts": [host], + "num_gpus": len(values), + "avg_power_w": round(sum(values) / audit.window["duration"] / len(values), 3), + } + ) + indices[role] = index + 1 + return workers + + def _patch_agg(agg_path: Path, audit: MultinodePowerAudit) -> None: cpu = audit.cpu - patch_power_metrics( - agg_path, + data = with_power_metrics( + json.loads(agg_path.read_text(encoding="utf-8")), + schema_version=POWER_METRIC_SCHEMA_VERSION, metric_keys=_ALL_POWER_METRIC_KEYS, power_valid=audit.power_valid, metrics=audit.metrics, cpu_power_valid=None if cpu is None else cpu.valid, cpu_metrics={} if cpu is None else cpu.metrics, ) + data.pop("workers", None) + if workers := _host_workers(audit): + data["workers"] = workers + _write_json_atomic(agg_path, data) def _withhold_metrics(audit: MultinodePowerAudit, cpu_reason: str | None = None) -> None: diff --git a/perf-changelog.yaml b/perf-changelog.yaml index f5574820bc..505d97f689 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8545,3 +8545,18 @@ - "在 Qwen3.5 FP8 GB200 与 GB300 的 1P1D TP4/TP4 STP recipe 上启用 srt-slurm 独立、尽力而为的 CPU 功耗测量环节：telemetry.cpu_power_exporter，port 9405（schema 默认值），source acpi。两份 recipe 其余内容不变：镜像、服务参数、拓扑、并发列表以及 GPU DCGM exporter（GB200 端口 9401、GB300 端口 19401，required: true）均未改动。在固定版本 srt-slurm v2.2.1（984180e5）下，srtctl 通过 srun 在每个 worker 节点的裸主机上启动 cpu-power-exporter --port 9405 --source acpi，若 make setup 未安装发布的二进制则回退到仅支持 ACPI 的 Python exporter；头节点采集器把 LOGS/power/cpu/samples.csv 和 cpu_manifest.json 写在 GPU DCGM 产物包旁。该环节没有 required 开关：缺少 hwmon 传感器或 exporter 失效只会让 samples.csv 出现空缺，不会导致作业失败或改变 GPU 功耗结论。多节点功耗校验器在与 GPU 能耗相同的正式测量窗口内对 Grace Power Socket 序列积分，新增 cpu_power_valid 与 CPU 侧字段（见 docs/results-and-ingestion_zh.md「NVL72 Grace CPU 侧实测功耗」）；power_valid 与所有 GPU 字段保持不变。v2.2.1 的 exporter 只识别 CPU、Grace、SysIO 三种 socket 标签，因此这些运行得到的是 Grace 侧总功耗而非模块读数。" - "选择这两条 lane 是因为它们是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane（2 个节点，1 台 prefill 主机 + 1 台 decode 主机，4 个 Grace socket）。未采用 DSV4 8k1k 的 GB200/GB300 lane：DeepSeek-V4-Pro 的单轮 8k1k 场景已于 2026-09-08 之后弃用（见 MODELS.md 与 configs/deprecated/nvidia-dsv4-8k1k-master.yaml），sweep 不会再执行它们。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 + +- config-keys: + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg + scenario-type: + - agentic-coding + description: + - "Enable the independent best-effort ACPI CPU power exporter on existing Kimi K3 GB200 and GB300 power recipes. Preserve GPU sampling, serving parameters, topology, and concurrency; publish validated per-host GPU power from the same measurement window so cross-host workers can be modeled as physical trays." + - "Replace the four GB300 recipes legacy provider/default_frequency keys with collect_interval_ms: 1000 for the pinned srt-slurm 2 schema; the intended one-second sampling cadence is unchanged." + - "将四份 GB300 配方中的旧 provider/default_frequency 字段改为固定版本 srt-slurm 2 接受的 collect_interval_ms: 1000，保留一秒采样间隔。" + - "在已有 GPU 功耗采集的 Kimi K3 GB200 与 GB300 配方中启用独立、尽力而为的 ACPI CPU 功耗采集；保留 GPU 采样、服务参数、拓扑和并发。根据同一测量窗口的已验证能耗发布逐物理主机 GPU 功耗，使跨主机 worker 可以按实际计算 tray 建模。" + pr-link: "XXX" diff --git a/utils/test_aggregate_power_multinode.py b/utils/test_aggregate_power_multinode.py index b7b17468ea..7b9038cd01 100644 --- a/utils/test_aggregate_power_multinode.py +++ b/utils/test_aggregate_power_multinode.py @@ -256,6 +256,12 @@ def test_emits_all_metrics_exactly(self, tmp_path): assert agg["joules_per_input_token"] == 2.563477 assert agg["joules_per_output_token"] == 20.507812 assert agg["joules_per_total_token"] == 2.278646 + assert agg["workers"] == [ + {"role": "decode", "worker_idx": 0, "hosts": ["node-d"], + "num_gpus": 2, "avg_power_w": 300.0}, + {"role": "prefill", "worker_idx": 0, "hosts": ["node-p"], + "num_gpus": 2, "avg_power_w": 400.0}, + ] assert agg["prefill_gpu_energy_j"] == 48000.0 assert agg["decode_gpu_energy_j"] == 36000.0 assert agg["prefill_avg_power_w"] == 400.0 @@ -945,3 +951,30 @@ def test_window_unavailable_when_result_binds_to_no_window(self, tmp_path): assert pkg.run() == 0 assert pkg.agg()["cpu_power_valid"] == 0 assert "cpu_window_unavailable" in pkg.sidecar()["cpu"]["reason_codes"] + + +def test_invalid_revalidation_removes_stale_host_power(tmp_path): + pkg = build_package(tmp_path) + assert pkg.run() == 0 + assert len(pkg.agg()["workers"]) == 2 + assert pkg.run(sha="b" * 40) == 0 + assert pkg.agg()["power_valid"] == 0 + assert "workers" not in pkg.agg() + + +def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): + pkg = build_package(tmp_path) + manifest_path = pkg.power_dir / "manifest.json" + manifest = json.loads(manifest_path.read_text()) + for device in manifest["expected_devices"]: + for assignment in device["assignments"]: + assignment["worker_role"] = "agg" + assignment["het_group"] = None + manifest_path.write_text(json.dumps(manifest)) + assert pkg.run(prefill_gpus=0, decode_gpus=0, aggregate_gpus=4) == 0 + assert pkg.agg()["workers"] == [ + {"role": "agg", "worker_idx": 0, "hosts": ["node-d"], + "num_gpus": 2, "avg_power_w": 300.0}, + {"role": "agg", "worker_idx": 1, "hosts": ["node-p"], + "num_gpus": 2, "avg_power_w": 400.0}, + ] From 15dbee4a2c06a06aaaef456518555b44b4fd5050 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 30 Sep 2026 22:05:06 -0700 Subject: [PATCH 07/35] fix: enforce NVL72 CPU exporter preflight MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 应用上游 #553 的提交检查修复，令启用 CPU 采集的配方在 exporter 缺失、不可执行或已知架构不匹配时于 sbatch 前失败，并明确源码采集链路与二进制版本的区别。 --- inferencex-e2e/docs/results-and-ingestion.md | 3 ++- inferencex-e2e/docs/results-and-ingestion_zh.md | 7 ++++--- inferencex-e2e/perf-changelog.yaml | 12 ++++++++++++ .../patches/553-cpu-exporter-submit-preflight.patch | 13 +++++++++++++ inferencex-e2e/runners/srt-slurm/patches/README.md | 1 + 5 files changed, 32 insertions(+), 4 deletions(-) create mode 100644 inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index e00ca0a796..a1ba216171 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -505,7 +505,8 @@ that classifies that label. The selected Qwen3.5 and Kimi-K3 CPU-telemetry recip `CPU_POWER_EXPORTER_RELEASE=v2.40.2` from their master-config `additional-settings` to `make setup`. This release includes legacy ACPI hwmon discovery. Setup verifies the downloaded asset checksum, replaces a cached binary with a different release marker, and fails on a pinned -download error; submission also rejects a missing, non-executable, or wrong-architecture binary. +download error. The local srt-slurm submission patch passes the resolved recipe to setup +validation, which rejects a missing, non-executable, or wrong-architecture binary. Qualification still needs the setup log and actual exporter identity, complete same-window samples, and verified firmware sensor boundaries. A source pin or successful setup alone does not establish valid measurements. A package without `cpu/` adds no CPU metrics or CPU verdict. diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index e0735ca21f..757eed7447 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -477,12 +477,13 @@ worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 produ `cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口 与该结果绑定，或窗口自身的契约检查失败）。CPU 积分溢出沿用 GPU 侧的 `non_finite_power_metric`；聚合 结果本身无法写入时，`aggregate_result_missing` 或 `aggregate_result_unwritable` 会同时出现在两侧的审计 -中。固定的 srt-slurm 源码版本（`098e15ac`）写出宽格式，exporter 尚不识别 Module 标签，因此 +中。固定的 srt-slurm 源码版本（`098e15ac`）写出宽格式；采集与写出链路尚不识别 Module 标签，因此 这些产物包只会得到 Grace socket 总功耗；需要 producer 支持该标签后才会出现模块字段。 选中的 Qwen3.5 与 Kimi-K3 CPU 采集配方通过主配置的 `additional-settings`，向 `make setup` 传入 `CPU_POWER_EXPORTER_RELEASE=v2.40.2`。该版本包含旧版 ACPI hwmon 传感器发现修复。 -安装过程校验下载工件的校验和，替换版本标记不匹配的缓存二进制，并在固定版本下载失败时退出； -提交前还会拒绝缺失、不可执行或架构不匹配的二进制。验收仍需保留安装日志与实际 exporter +安装过程校验下载工件的校验和，替换版本标记不匹配的缓存二进制，并在固定版本下载失败时退出。 +本地 srt-slurm 提交补丁将已解析的配方传入安装检查，从而在提交前拒绝缺失、不可执行或架构 +不匹配的二进制。验收仍需保留安装日志与实际 exporter 身份，检查完整的同窗口样本，并核实固件传感器的测量范围。源码版本或安装成功均不能证明 测量有效。没有 `cpu/` 的产物包不新增 CPU 指标或 CPU 有效性结论；通过 GPU 校验的产物包 仍发布上文所述的逐主机 `workers` 条目。 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 4029463850..a1aa4cb824 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9209,3 +9209,15 @@ description: - "Pin cpu-power-exporter v2.40.2 for the existing Qwen3.5 and Kimi-K3 NVL72 CPU-telemetry recipes. The fixed binary discovers legacy ACPI hwmon meters; setup verifies the release asset checksum and fails if the pinned download cannot be installed. Preserve the merged noprof GPU profile and independent best-effort CPU verdict. Other recipe variants retain their current runtime settings." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 + +- config-keys: + - qwen3.5-fp8-gb200-dynamo-sglang + - qwen3.5-fp8-gb300-dynamo-sglang + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg + description: + - "Apply NVIDIA/srt-slurm#553 to the pinned submitter: pass the resolved recipe into setup validation so CPU-enabled jobs fail before sbatch when the exporter is missing, non-executable, or built for the wrong compute architecture." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 diff --git a/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch b/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch new file mode 100644 index 0000000000..d19797214c --- /dev/null +++ b/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch @@ -0,0 +1,13 @@ +diff --git a/src/srtctl/cli/submit.py b/src/srtctl/cli/submit.py +index 5c213209..c21a6047 100755 +--- a/src/srtctl/cli/submit.py ++++ b/src/srtctl/cli/submit.py +@@ -1214,7 +1214,7 @@ def submit_with_orchestrator( + # Validate setup before submitting (not during dry-run) + srtctl_root = get_srtslurm_setting("srtctl_root") + srtctl_source = Path(srtctl_root) if srtctl_root else Path(__file__).parent.parent.parent.parent +- validate_setup(srtctl_source) ++ validate_setup(srtctl_source, config) + + if render_dir is not None: + return _render_to_dir( diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index e6ba038f31..c5dc8e7257 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -9,3 +9,4 @@ Each patch is a temporary fix for an open upstream PR. When the PR merges and th | Patch | Upstream PR | Fix | |-------|-------------|-----| | `548-atom-served-model-name.patch` | [NVIDIA/srt-slurm#548](https://github.com/NVIDIA/srt-slurm/pull/548) | ATOM serves evals the role's `served-model-name` instead of the literal `--model` path, so AToMesh accepts eval requests | +| `553-cpu-exporter-submit-preflight.patch` | [NVIDIA/srt-slurm#553](https://github.com/NVIDIA/srt-slurm/pull/553) | Pass the resolved recipe into setup validation so CPU-enabled jobs reject unusable exporters before submission | From 8bd735e551a2d552d82705ae73b9185c0767b65a Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 1 Oct 2026 20:39:54 -0700 Subject: [PATCH 08/35] fix: require complete CPU power for NVL72 sweeps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 要求 NVL72 扫描提供完整 CPU 功耗，并补齐 Qwen 采集与时钟主机配置。 --- .github/AGENT_OPERATIONS.md | 2 +- .../workflows/benchmark-multinode-tmpl.yml | 1 + .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 3 + .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 20 +++- .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 3 + .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 21 +++- inferencex-e2e/configs/nvidia-master.yaml | 6 + inferencex-e2e/docs/results-and-ingestion.md | 11 +- .../docs/results-and-ingestion_zh.md | 10 +- inferencex-e2e/infx/launch/artifacts.py | 5 +- .../infx/launch/drivers/srt/__init__.py | 3 + .../infx/launch/drivers/srt/collect.py | 14 ++- .../infx/launch/drivers/srt/power.py | 15 ++- .../infx/results/agentic/power_adapter.py | 4 + inferencex-e2e/infx/results/fixed_sequence.py | 1 + inferencex-e2e/infx/results/power/cpu_side.py | 18 ++- .../infx/results/power/multinode.py | 21 +++- .../tests/launch/test_launch_artifacts.py | 67 +++++++++++ .../infx/tests/launch/test_launch_policy.py | 22 ++++ .../infx/tests/launch/test_srt_config.py | 34 ++++++ .../infx/tests/launch/test_srt_driver.py | 46 ++++++++ .../results/agentic/test_power_adapter.py | 106 ------------------ .../power/test_aggregate_power_multinode.py | 27 +++++ inferencex-e2e/perf-changelog.yaml | 13 +++ 24 files changed, 340 insertions(+), 133 deletions(-) diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 0340b3c1bb..06a0515d49 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -99,7 +99,7 @@ Single-node fixed-sequence results may include `power_valid`, `avg_power_w`, `av Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count. -NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, `total_cpu_energy_j`, and, when the module sensor is exposed on every socket, `avg_total_module_power_w` and `total_module_energy_j`. That verdict is independent of `power_valid` and of `REQUIRE_POWER`; see [`inferencex-e2e/docs/results-and-ingestion.md`](../inferencex-e2e/docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). +NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, `total_cpu_energy_j`, and, when the module sensor is exposed on every socket, `avg_total_module_power_w` and `total_module_energy_j`. That verdict is independent of `power_valid`; strict `REQUIRE_POWER` qualification also requires any CPU source declared by the recipe; see [`inferencex-e2e/docs/results-and-ingestion.md`](../inferencex-e2e/docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). Every power result — valid or invalid, single-node or multinode — carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. diff --git a/.github/workflows/benchmark-multinode-tmpl.yml b/.github/workflows/benchmark-multinode-tmpl.yml index d495876be1..1d4b78b9a5 100644 --- a/.github/workflows/benchmark-multinode-tmpl.yml +++ b/.github/workflows/benchmark-multinode-tmpl.yml @@ -367,6 +367,7 @@ jobs: fi export EVAL_CONC export IS_MULTINODE=true + echo "POWER_EXPECTED_CPU_SOURCE=" >> "$GITHUB_ENV" "$INFERENCEX_LAUNCH_PYTHON" -m infx.launch run if [ "${EVAL_ONLY}" = "true" ]; then echo "Eval-only mode: skipping benchmark result file check" diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index 6a33277513..7e8a331b53 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -170,3 +170,6 @@ telemetry: container_image: dcgm-exporter port: 9401 command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index a81e258e7a..ca331ce895 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -10,11 +10,11 @@ services: - name: etcd type: etcd placement: - node: dedicated + node: head - name: nats type: nats placement: - node: dedicated + node: head dynamo: install: true @@ -22,6 +22,7 @@ dynamo: wheel: "1.5.0.dev20260917" frontend: type: dynamo + dedicated_node: true enable_multiple_frontends: true num_additional_frontends: 4 nginx_container: nginx @@ -166,3 +167,18 @@ benchmark: TOKENIZER: /model ISL: '8192' OSL: '1024' + +telemetry: + enabled: true + collect_interval_ms: 1000 + storage_subdir: power + required: true + startup_timeout_seconds: 120 + request_timeout_seconds: 2 + dcgm_exporter: + container_image: dcgm-exporter + port: 9401 + command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index 6de94aaded..0bd60e0e9b 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -169,3 +169,6 @@ telemetry: container_image: dcgm-exporter port: 19401 command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index 0ae0af5a67..1f4b417757 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -9,11 +9,11 @@ services: - name: etcd type: etcd placement: - node: dedicated + node: head - name: nats type: nats placement: - node: dedicated + node: head dynamo: install: true @@ -21,6 +21,7 @@ dynamo: wheel: "1.5.0.dev20260917" frontend: type: dynamo + dedicated_node: true enable_multiple_frontends: true num_additional_frontends: 4 nginx_container: nginx @@ -165,3 +166,19 @@ benchmark: TOKENIZER: /model ISL: '8192' OSL: '1024' + +telemetry: + enabled: true + collect_interval_ms: 1000 + storage_subdir: power + required: true + startup_timeout_seconds: 120 + request_timeout_seconds: 2 + dcgm_exporter: + container_image: dcgm-exporter + # The cluster exporter uses 9401; use a separate port. + port: 19401 + command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" + cpu_power_exporter: + port: 9405 + source: acpi diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 40e1c1a862..3f751e5499 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -4091,6 +4091,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: fixed-seq-len: - isl: 8192 osl: 1024 + require-power: true search-space: # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1). - spec-decoding: "none" @@ -4118,6 +4119,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" + - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" decode: num-worker: 1 tp: 16 @@ -4133,6 +4135,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" + - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" decode: num-worker: 1 tp: 16 @@ -6777,6 +6780,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: fixed-seq-len: - isl: 8192 osl: 1024 + require-power: true search-space: # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1). - spec-decoding: "none" @@ -6804,6 +6808,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" + - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" decode: num-worker: 1 tp: 16 @@ -6819,6 +6824,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" + - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" decode: num-worker: 1 tp: 16 diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index a1ba216171..08d4e62226 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -476,7 +476,7 @@ are: | Key | Meaning | | --- | --- | -| `cpu_power_valid` | `1` or `0` for the CPU-side leg; absent when the package has no `cpu/` | +| `cpu_power_valid` | `1` or `0` for the CPU-side leg; absent only when CPU is undeclared and the package has no `cpu/` | | `avg_cpu_socket_power_w` | Mean over sockets of each socket's window-mean Grace-side watts | | `avg_total_cpu_power_w` | Sum over sockets of window-mean Grace-side watts | | `total_cpu_energy_j` | Grace-side energy over the window, all sockets | @@ -488,11 +488,12 @@ The sidecar's `cpu` block and the aggregate's `power_audit.cpu` record `sensor_k `grace_socket`, or `dcgm_cpu_rail`), `source` (`acpi` or `dcgm`), expected and observed socket counts, the parsed row count, and reason codes. `power_metric_schema_version` stays `2`. -The leg is best effort and its verdict is independent of `power_valid`: it borrows only the bound +The CPU verdict is independent of `power_valid`: it borrows only the bound formal window and the worker-host topology from the GPU package, and no GPU verdict reaches it, so an unpinned producer or failed GPU coverage withholds GPU energy while `cpu_power_valid` still judges the CPU samples on their own. Any CPU-leg failure records `cpu_power_valid: 0` with no CPU -keys, leaves every GPU field byte-identical, and never fails `REQUIRE_POWER=1`. Reason codes: +keys and leaves every GPU field unchanged. With `REQUIRE_POWER=1`, a recipe-declared CPU source must be valid; ACPI requires complete Grace socket or module totals, and missing CPU artifacts produce an independent invalid CPU audit. GPU-only recipes retain their existing behavior. Reason codes: +`cpu_artifacts_missing`, `cpu_sensor_source_mismatch`, `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch`, `cpu_sensor_kind_mixed`, `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, and `cpu_window_unavailable` (no completed @@ -503,13 +504,13 @@ The srt-slurm source pin (`098e15ac`) writes the wide format and does not classi Module label, so these packages yield the Grace socket total; module keys require a producer that classifies that label. The selected Qwen3.5 and Kimi-K3 CPU-telemetry recipes pass `CPU_POWER_EXPORTER_RELEASE=v2.40.2` from their master-config `additional-settings` to -`make setup`. This release includes legacy ACPI hwmon discovery. Setup verifies the downloaded +`make setup`. Qwen3.5 8P1D keeps its frontend, benchmark client, and infra on the reserved head so their timestamps share the collector clock; its 13-node allocation still contains 12 GPU-worker nodes. This release includes legacy ACPI hwmon discovery. Setup verifies the downloaded asset checksum, replaces a cached binary with a different release marker, and fails on a pinned download error. The local srt-slurm submission patch passes the resolved recipe to setup validation, which rejects a missing, non-executable, or wrong-architecture binary. Qualification still needs the setup log and actual exporter identity, complete same-window samples, and verified firmware sensor boundaries. A source pin or successful setup alone does -not establish valid measurements. A package without `cpu/` adds no CPU metrics or CPU verdict. +not establish valid measurements. Without an expected CPU source, a package without `cpu/` adds no CPU metrics or CPU verdict. Validated GPU packages still publish the per-host `workers` entries described above. ## Verification and stop conditions diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index 757eed7447..5c7eee8b4c 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -457,7 +457,7 @@ HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePow | 字段 | 含义 | | --- | --- | -| `cpu_power_valid` | CPU 侧测量的有效性，`1` 或 `0`；产物包没有 `cpu/` 时不出现 | +| `cpu_power_valid` | CPU 侧测量的有效性，`1` 或 `0`；仅在未声明 CPU 来源且没有 `cpu/` 时不出现 | | `avg_cpu_socket_power_w` | 各 socket 窗口平均 Grace 侧功耗的均值 | | `avg_total_cpu_power_w` | 各 socket 窗口平均 Grace 侧功耗之和 | | `total_cpu_energy_j` | 窗口内所有 socket 的 Grace 侧能耗 | @@ -471,8 +471,8 @@ HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePow 该测量环节尽力而为，其结论与 `power_valid` 相互独立：它只从 GPU 产物包借用已绑定的正式测量窗口和 worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 producer 固定版本校验失败或 GPU 覆盖不足只会 使 GPU 能耗不予发布，`cpu_power_valid` 仍按 CPU 采样自身给出结论。CPU 侧的任何失败都会记录 -`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持逐字节不变，也不会使 `REQUIRE_POWER=1` 失败。 -原因码包括 `cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、 +`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持不变。设置 `REQUIRE_POWER=1` 时，配方声明的 CPU 来源必须有效；ACPI 必须提供完整的 Grace socket 或模块总功耗，缺失 CPU 产物会生成独立的无效 CPU 审计。仅采集 GPU 的配方保留原有行为。 +原因码包括 `cpu_artifacts_missing`、`cpu_sensor_source_mismatch`、`cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、 `cpu_manifest_invalid`、`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、 `cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口 与该结果绑定，或窗口自身的契约检查失败）。CPU 积分溢出沿用 GPU 侧的 `non_finite_power_metric`；聚合 @@ -480,12 +480,12 @@ worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 produ 中。固定的 srt-slurm 源码版本（`098e15ac`）写出宽格式；采集与写出链路尚不识别 Module 标签，因此 这些产物包只会得到 Grace socket 总功耗；需要 producer 支持该标签后才会出现模块字段。 选中的 Qwen3.5 与 Kimi-K3 CPU 采集配方通过主配置的 `additional-settings`，向 `make setup` -传入 `CPU_POWER_EXPORTER_RELEASE=v2.40.2`。该版本包含旧版 ACPI hwmon 传感器发现修复。 +传入 `CPU_POWER_EXPORTER_RELEASE=v2.40.2`。Qwen3.5 8P1D 将前端、压测客户端和基础服务放在预留的 head 节点，与采集器共用时钟；仍分配 13 个节点，其中 12 个为 GPU worker。该版本包含旧版 ACPI hwmon 传感器发现修复。 安装过程校验下载工件的校验和，替换版本标记不匹配的缓存二进制，并在固定版本下载失败时退出。 本地 srt-slurm 提交补丁将已解析的配方传入安装检查，从而在提交前拒绝缺失、不可执行或架构 不匹配的二进制。验收仍需保留安装日志与实际 exporter 身份，检查完整的同窗口样本，并核实固件传感器的测量范围。源码版本或安装成功均不能证明 -测量有效。没有 `cpu/` 的产物包不新增 CPU 指标或 CPU 有效性结论；通过 GPU 校验的产物包 +测量有效。未声明 CPU 来源时，没有 `cpu/` 的产物包不新增 CPU 指标或 CPU 有效性结论；通过 GPU 校验的产物包 仍发布上文所述的逐主机 `workers` 条目。 ## 验证和停止条件 diff --git a/inferencex-e2e/infx/launch/artifacts.py b/inferencex-e2e/infx/launch/artifacts.py index deaa823ebb..ba66f5a38a 100644 --- a/inferencex-e2e/infx/launch/artifacts.py +++ b/inferencex-e2e/infx/launch/artifacts.py @@ -153,6 +153,7 @@ def collect_agentic_power_results( concurrencies: Sequence[int], *, results_python: str | None, + expected_cpu_source: str | None = None, ) -> int: """Stage the AgentX power audit inputs and validate each concurrency's power window. @@ -175,7 +176,7 @@ def collect_agentic_power_results( if step := validate_agentic_power( logs_dir, workspace, result_filename, producer_sha, concurrencies, - results_python=results_python, require_power=True, + results_python=results_python, require_power=True, expected_cpu_source=expected_cpu_source, ): # fmt: skip rc = step return rc @@ -190,6 +191,7 @@ def validate_agentic_power( *, results_python: str | None, require_power: bool, + expected_cpu_source: str | None = None, ) -> int: """Run the AgentX power adapter for every concurrency; return the last failing code. @@ -216,6 +218,7 @@ def validate_agentic_power( "--logs-root", str(logs_dir), "--expected-producer-sha", producer_sha, *(["--require-power"] if require_power else []), + *(["--expected-cpu-source", expected_cpu_source] if expected_cpu_source else []), ] # fmt: skip step = proc.run(argv, env=env, cwd=workspace).returncode if step: diff --git a/inferencex-e2e/infx/launch/drivers/srt/__init__.py b/inferencex-e2e/infx/launch/drivers/srt/__init__.py index 4f7099498e..ec2838f2d1 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/__init__.py +++ b/inferencex-e2e/infx/launch/drivers/srt/__init__.py @@ -124,6 +124,9 @@ def run_multinode(launch: Launch) -> int: lanes.check_request(lane, request) config_file = lanes.config_file(request) decision = power.resolve_power(launch.cluster.id, launch.path, request) + if github_env := request.env.get("GITHUB_ENV"): + with Path(github_env).open("a") as handle: + handle.write(f"POWER_EXPECTED_CPU_SOURCE={decision.expected_cpu_source or ''}\n") model = models.checkpoint(launch.cluster, request) served = models.served_path(launch.cluster, request, model) model_paths = models.model_paths(launch.cluster, request, config_file, served) diff --git a/inferencex-e2e/infx/launch/drivers/srt/collect.py b/inferencex-e2e/infx/launch/drivers/srt/collect.py index 0324f4e775..8b89281506 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/collect.py +++ b/inferencex-e2e/infx/launch/drivers/srt/collect.py @@ -148,11 +148,21 @@ def stage_logs_on_exit() -> None: python = request.inferencex_results_python if power.agentx: power_rc = collect_agentic_power_results( - status, job.id, logs, infmax, *audit, results_python=python + status, + job.id, + logs, + infmax, + *audit, + results_python=python, + expected_cpu_source=power.expected_cpu_source, ) else: power_rc = validate_agentic_power( - logs, *audit, results_python=python, require_power=request.require_power + logs, + *audit, + results_python=python, + require_power=request.require_power, + expected_cpu_source=power.expected_cpu_source, ) if power_rc: print( diff --git a/inferencex-e2e/infx/launch/drivers/srt/power.py b/inferencex-e2e/infx/launch/drivers/srt/power.py index 29eb324dbb..6985dd987e 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/power.py +++ b/inferencex-e2e/infx/launch/drivers/srt/power.py @@ -7,7 +7,7 @@ from __future__ import annotations import fnmatch -from dataclasses import dataclass +from dataclasses import dataclass, replace from typing import TYPE_CHECKING import yaml @@ -159,6 +159,7 @@ class PowerDecision: dcgm: bool agentx: bool adapter: bool = False + expected_cpu_source: str | None = None NO_POWER = PowerDecision(dcgm=False, agentx=False) @@ -197,7 +198,15 @@ def resolve_power(cluster_id: str, path: LaunchPath, request: LaunchRequest) -> if not config_file: return NO_POWER mirror = recipe_mirror_path(request.workspace, config_file) - dcgm = mirror.is_file() and recipe_enables_dcgm_power(mirror.read_text()) - return decide_power( + text = mirror.read_text() if mirror.is_file() else "" + dcgm = recipe_enables_dcgm_power(text) + decision = decide_power( cluster_id, path, dcgm=dcgm, request=request, recipe=recipe_relpath(config_file) ) + if decision.dcgm: + telemetry = yaml.safe_load(text)["telemetry"] + cpu = telemetry.get("cpu_power_exporter") + source = cpu.get("source") if isinstance(cpu, dict) else None + if source in {"acpi", "dcgm"}: + decision = replace(decision, expected_cpu_source=source) + return decision diff --git a/inferencex-e2e/infx/results/agentic/power_adapter.py b/inferencex-e2e/infx/results/agentic/power_adapter.py index 2a92044293..e65725b472 100644 --- a/inferencex-e2e/infx/results/agentic/power_adapter.py +++ b/inferencex-e2e/infx/results/agentic/power_adapter.py @@ -420,6 +420,7 @@ def run_multinode_agentic_power( logs_root: Path, expected_producer_sha: str, require_power: bool = False, + expected_cpu_source: str | None = None, ) -> int: """Join one AgentX aggregate to the finalized central multinode package.""" validation_result = result_dir / "power_validation.json" @@ -491,6 +492,7 @@ def run_multinode_agentic_power( logs_root=logs_root, validation_result=validation_result, require_power=require_power, + expected_cpu_source=expected_cpu_source, ) @@ -510,6 +512,7 @@ def main() -> int: action="store_true", default=os.environ.get("REQUIRE_POWER", "").lower() in {"1", "true", "yes"}, ) + parser.add_argument("--expected-cpu-source", choices=("acpi", "dcgm")) args = parser.parse_args() if args.multinode_contract_missing: if args.agg_result is None: @@ -560,6 +563,7 @@ def main() -> int: logs_root=args.logs_root, expected_producer_sha=args.expected_producer_sha, require_power=args.require_power, + expected_cpu_source=args.expected_cpu_source, ) if args.agg_result is None: parser.error("--agg-result is required for single-node AgentX power") diff --git a/inferencex-e2e/infx/results/fixed_sequence.py b/inferencex-e2e/infx/results/fixed_sequence.py index d6dd547e0d..edf9f23bb2 100644 --- a/inferencex-e2e/infx/results/fixed_sequence.py +++ b/inferencex-e2e/infx/results/fixed_sequence.py @@ -316,6 +316,7 @@ def aggregate_power_result( aggregate_gpus=aggregate_gpus, expected_producer_sha=env.get("POWER_PRODUCER_SHA") or None, logs_root=Path(env.get("POWER_RESULT_ROOT", "LOGS")), + expected_cpu_source=env.get("POWER_EXPECTED_CPU_SOURCE") or None, validation_result=validation_path, require_power=require_power, ) diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index 69dc45043b..6ec7a8628f 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -11,8 +11,9 @@ bound formal window with the shared trapezoid and boundary interpolation and published as additive metrics next to the GPU-board numbers. The leg is best-effort by contract: any failure records ``cpu_power_valid=0`` with -reason codes and leaves every GPU field untouched, and ``REQUIRE_POWER`` -never fails a run because of it. A package without ``cpu/`` emits nothing. +reason codes and leaves every GPU field untouched. Strict qualification also +requires a recipe-declared CPU source; GPU-only callers keep best-effort CPU +behavior. A package without ``cpu/`` emits nothing unless a source is expected. """ from __future__ import annotations @@ -247,10 +248,15 @@ def validate_cpu_leg( *, window: tuple[float, float] | None, expected_hosts: Iterable[str], + expected_source: str | None = None, ) -> CpuPowerAudit | None: """Validate and integrate the CPU sub-package; None when the run carried none.""" if not cpu_dir.is_dir(): - return None + if expected_source is None: + return None + audit = CpuPowerAudit() + audit.invalidate("cpu_artifacts_missing") + return audit audit = CpuPowerAudit() if not _manifest_is_object(cpu_dir / CPU_MANIFEST_FILENAME): audit.invalidate("cpu_manifest_invalid") @@ -280,6 +286,12 @@ def validate_cpu_leg( return audit audit.sensor_kind = selected audit.source = next(iter(sources[selected])) + if expected_source is not None and ( + audit.source != expected_source + or (expected_source == "acpi" and selected not in {SENSOR_GRACE, SENSOR_MODULE}) + ): + audit.invalidate("cpu_sensor_source_mismatch") + return audit start, end = window energy: dict[str, float] = {} diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 80fa68da19..acf405878c 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -31,8 +31,9 @@ Grace CPU-side leg: when the package also carries ``power/cpu/`` (see :mod:`.cpu_side`), each socket's Grace-side (and, when exposed, whole-module) power is integrated over the same bound formal window and published under -``cpu_power_valid``. That verdict is independent of ``power_valid`` and of -``REQUIRE_POWER``; a package without ``cpu/`` adds no CPU metrics or verdict. +``cpu_power_valid``. That verdict is independent of ``power_valid``. Strict +power qualification fails when a recipe-declared CPU source is missing or invalid; +GPU-only callers retain best-effort CPU behavior. """ from __future__ import annotations @@ -933,6 +934,7 @@ def validate_and_integrate( decode_gpus: int, aggregate_gpus: int, expected_producer_sha: str | None, + expected_cpu_source: str | None = None, ) -> MultinodePowerAudit: """Recompute package validity, cross-check verdicts, and integrate energy. @@ -958,6 +960,7 @@ def validate_and_integrate( power_dir / CPU_DIRNAME, window=window, expected_hosts=audit.expected_worker_hosts, + expected_source=expected_cpu_source, ) return audit @@ -1456,6 +1459,7 @@ def run( aggregate_gpus: int = 0, validation_result: Path | None = None, require_power: bool = False, + expected_cpu_source: str | None = None, ) -> int: """Validate multinode power artifacts, patch metrics, and audit the verdict.""" validation_result = validation_result or bench_result.with_name( @@ -1476,6 +1480,7 @@ def run( decode_gpus=decode_gpus, aggregate_gpus=aggregate_gpus, expected_producer_sha=expected_producer_sha, + expected_cpu_source=expected_cpu_source, ) for reason in benchmark_reasons: _add_reason(audit, reason) @@ -1524,6 +1529,14 @@ def run( f"reasons={','.join(cpu.reason_codes) or '-'}" ) + if ( + require_power + and expected_cpu_source is not None + and (audit.cpu is None or not audit.cpu.valid) + ): + print("[aggregate_power_multinode] Required CPU power is invalid", file=sys.stderr) + return 1 + if not audit.power_valid: print( f"[aggregate_power_multinode] Power validation failed: " @@ -1602,8 +1615,9 @@ def main() -> int: "--require-power", action="store_true", default=os.environ.get("REQUIRE_POWER", "").lower() in {"1", "true", "yes"}, - help="Fail when power telemetry is invalid (also enabled by REQUIRE_POWER=1)", + help="Fail invalid GPU power and any expected CPU source (also REQUIRE_POWER=1)", ) + parser.add_argument("--expected-cpu-source", choices=("acpi", "dcgm")) args = parser.parse_args() return run( args.power_dir, @@ -1616,6 +1630,7 @@ def main() -> int: logs_root=args.logs_root, validation_result=args.validation_result, require_power=args.require_power, + expected_cpu_source=args.expected_cpu_source, ) diff --git a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py index e82f1b37dc..e30f55f978 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py @@ -1,7 +1,9 @@ """Workspace staging behavior of the launcher artifact helpers.""" +import json import os import stat +import sys import tarfile import pytest @@ -112,3 +114,68 @@ def test_power_collection_records_the_job_status_and_stages_results_either_way(t assert (logs / "power" / "native-job-status.txt").read_text() == f"42|{status.raw}\n" assert (workspace / "point_conc4.json").exists() + + +@pytest.mark.parametrize("has_cpu", [False, True]) +@pytest.mark.parametrize("disagg", [False, True]) +def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path, has_cpu, disagg): + from infx.tests.results.power.test_aggregate_power_multinode import ( + PRODUCER_SHA, + _cpu_rows, + add_cpu_package, + build_package, + ) + + pkg = build_package(tmp_path / "captured") + result_dir = pkg.logs_root / "agentic/conc_4" + result_dir.mkdir(parents=True) + formal_result = result_dir / "agentic_power_concurrency_4.json" + formal_result.write_bytes(pkg.original_result.read_bytes()) + [window_path] = pkg.windows_dir.glob("*.json") + window = json.loads(window_path.read_text()) + window["result_path"] = str(formal_result.relative_to(pkg.logs_root)) + window_path.unlink() + window_path = pkg.windows_dir / formal_result.name + window_path.write_text(json.dumps(window)) + manifest_path = pkg.power_dir / "manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["window_validations"][0]["window_file"] = f"windows/{formal_result.name}" + manifest_path.write_text(json.dumps(manifest)) + if not disagg: + manifest_path = pkg.power_dir / "manifest.json" + manifest = json.loads(manifest_path.read_text()) + for device in manifest["expected_devices"]: + device["assignments"][0].update(worker_role="agg", het_group=0) + manifest_path.write_text(json.dumps(manifest)) + if has_cpu: + add_cpu_package(pkg, _cpu_rows()) + source, workspace = tmp_path / "source", tmp_path / "output" + source.mkdir() + workspace.mkdir() + (source / "point_conc4.json").write_text( + json.dumps( + { + "disagg": disagg, + "num_prefill_gpu": 2 if disagg else 4, + "num_decode_gpu": 2 if disagg else 0, + } + ) + ) + status = JobStatus(JobState.SUCCEEDED, "COMPLETED|0:0", 0) + assert collect_agentic_power_results( + status, + "42", + pkg.logs_root, + source, + workspace, + "point", + PRODUCER_SHA, + [4], + results_python=sys.executable, + expected_cpu_source="acpi", + ) == int(not has_cpu) + result = json.loads((workspace / "point_conc4.json").read_text()) + assert result["power_valid"] == 1 + assert result["cpu_power_valid"] == int(has_cpu) + assert result["total_gpu_energy_j"] == 84000.0 + assert (result_dir / "power_validation.json").is_file() diff --git a/inferencex-e2e/infx/tests/launch/test_launch_policy.py b/inferencex-e2e/infx/tests/launch/test_launch_policy.py index a3d8d128d3..c227f8d798 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_policy.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_policy.py @@ -199,3 +199,25 @@ def test_cluster_settings_beat_the_host_but_never_the_points_settings(chosen, ex env = policy.runtime_env(cluster, request, {"LATER": "later", "SHARED": "later"}) assert {name: env[name] for name in HOST} == expected + + +@pytest.mark.parametrize( + "enabled,source,expected", + [ + (True, "acpi", "acpi"), + (False, "acpi", None), + (True, "dcgm", "dcgm"), + ], +) +def test_power_recipe_resolves_enabled_cpu_source(tmp_path, enabled, source, expected): + telemetry = POWER_RECIPE.replace("enabled: true", f"enabled: {str(enabled).lower()}") + _mirror( + tmp_path, "qwen/cpu.yaml", telemetry + (f" cpu_power_exporter:\n source: {source}\n") + ) + request = _request( + GITHUB_WORKSPACE=str(tmp_path), + CONFIG_FILE="recipes/qwen/cpu.yaml", + IS_AGENTIC="0", + FRAMEWORK="dynamo-sglang", + ) + assert resolve_power("gb300-nv", MULTI, request).expected_cpu_source == expected diff --git a/inferencex-e2e/infx/tests/launch/test_srt_config.py b/inferencex-e2e/infx/tests/launch/test_srt_config.py index 193281657e..db231e960d 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_config.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_config.py @@ -229,3 +229,37 @@ def rsync(source: Path, staging: Path, *, exclude: tuple[str, ...]) -> Path: assert staged == tmp_path / "runs/infmax-workspace-1-1-abc" assert sorted(str(p.relative_to(staged)) for p in staged.rglob("*") if p.is_file()) == sorted(kept) + + +@pytest.mark.parametrize("sku", ["gb200", "gb300"]) +def test_qwen_8p1d_power_keeps_the_collector_and_benchmark_on_the_reserved_head( + tmp_path, monkeypatch, sku +): + from srtctl.core.config import load_config + from srtctl.core.runtime import Nodes + + monkeypatch.setattr("srtctl.core.config.load_cluster_config", lambda: None) + recipe = ROOT / ( + f"benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/{sku}-fp8/8k1k/" + "disagg-8p1d-dep4-dep16-stp.yaml" + ) + staged = tmp_path / "recipes" / recipe.name + staged.parent.mkdir() + shutil.copyfile(recipe, staged) + prepare_recipe(tmp_path, f"recipes/{recipe.name}", "qwen-8p1d", None, "2048 4096") + config = load_config(staged) + allocation = [f"node-{index}" for index in range(13)] + monkeypatch.setattr("srtctl.core.runtime.get_slurm_nodelist", lambda: allocation) + monkeypatch.setattr("srtctl.core.runtime.get_slurm_het_nodelists", lambda: None) + nodes = Nodes.from_slurm( + frontend_dedicated_node=config.frontend.dedicated_node, + client_dedicated_node=config.benchmark.client_dedicated_node, + etcd_nats_dedicated_node=config.infra.etcd_nats_dedicated_node, + colocate_dedicated_nodes=config.benchmark.colocate_with_frontend, + ) + assert config.engine_node_count == 12 + assert not config.pool_services + assert nodes.head == nodes.bench == nodes.infra == allocation[0] + assert nodes.worker == nodes.compute == tuple(allocation[1:]) + assert config.benchmark.client_placement == "head" + assert config.telemetry.enabled and config.telemetry.cpu_power_exporter.source == "acpi" diff --git a/inferencex-e2e/infx/tests/launch/test_srt_driver.py b/inferencex-e2e/infx/tests/launch/test_srt_driver.py index 18118f7bc4..33b539e7bc 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_driver.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_driver.py @@ -570,3 +570,49 @@ def test_multinode_eval_overrides_image_offline_mode_and_host_model_path(harness "http://worker:8000", str(harness.workspace), ] + + +@pytest.mark.parametrize("has_cpu", [False, True]) +def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkeypatch, has_cpu): + from infx.results.fixed_sequence import aggregate_power_result + from infx.tests.results.power.test_aggregate_power_multinode import ( + PRODUCER_SHA, + _cpu_rows, + add_cpu_package, + build_package, + ) + + github_env = harness.tmp / "github-env" + recipe = LANE_RECIPE + POWER_TELEMETRY + " cpu_power_exporter:\n source: acpi\n" + env = lane_env( + harness, + "gb300-nv", + recipe, + MODEL_PREFIX="qwen3.5", + PRECISION="fp8", + FRAMEWORK="dynamo-sglang", + MODEL="Qwen/Qwen3.5-397B-A17B-FP8", + REQUIRE_POWER="1", + GITHUB_ENV=str(github_env), + ) + assert_ok(launch(env, harness.config, harness.workspace)) + exported = dict(line.split("=", 1) for line in github_env.read_text().splitlines()) + pkg = build_package(harness.tmp / "power-input") + if has_cpu: + add_cpu_package(pkg, _cpu_rows()) + monkeypatch.chdir(harness.tmp) + result_env = { + **exported, + "REQUIRE_POWER": "1", + "IS_MULTINODE": "true", + "RESULT_FILENAME": "fixture", + "PREFILL_GPUS": "2", + "DECODE_GPUS": "2", + "POWER_PRODUCER_SHA": PRODUCER_SHA, + "POWER_ARTIFACT_DIR": str(pkg.power_dir), + "POWER_RESULT_ROOT": str(pkg.logs_root), + } + assert aggregate_power_result(result_env, pkg.bench_result, pkg.agg_result) == int(not has_cpu) + aggregate = pkg.agg() + assert aggregate["power_valid"] == 1 + assert aggregate["cpu_power_valid"] == int(has_cpu) diff --git a/inferencex-e2e/infx/tests/results/agentic/test_power_adapter.py b/inferencex-e2e/infx/tests/results/agentic/test_power_adapter.py index 30f2351480..7a4b644a54 100644 --- a/inferencex-e2e/infx/tests/results/agentic/test_power_adapter.py +++ b/inferencex-e2e/infx/tests/results/agentic/test_power_adapter.py @@ -492,112 +492,6 @@ def test_multinode_window_writer_fails_closed_on_invalid_window_environment( assert "formal measurement-window contract" in capsys.readouterr().err -def test_multinode_aggregation_uses_central_package_and_aggregate_topology( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -): - from infx.results.agentic import power_adapter - - logs_root = tmp_path / "logs" - result_dir = logs_root / "agentic" / "conc_8" - result_dir.mkdir(parents=True) - bench_result = result_dir / "agentic_power_concurrency_8.json" - bench_result.write_text(json.dumps({"max_concurrency": 8})) - agg_result = tmp_path / "agg_agentx_conc8.json" - agg_result.write_text( - json.dumps( - {"disagg": True, "num_prefill_gpu": 16, "num_decode_gpu": 16} - ), - encoding="utf-8", - ) - power_dir = logs_root / "power" - power_dir.mkdir(parents=True) - calls: list[dict] = [] - - def fake_run(**kwargs) -> int: - calls.append(kwargs) - return 0 - - monkeypatch.setattr(power_adapter, "run_multinode_power", fake_run) - - exit_code = power_adapter.run_multinode_agentic_power( - result_dir=result_dir, - agg_result=agg_result, - power_dir=power_dir, - logs_root=logs_root, - expected_producer_sha="a1b8c7af10c00e5ea40074aebdc0086189bbc064", - require_power=True, - ) - - assert exit_code == 0 - assert calls == [ - { - "power_dir": power_dir, - "bench_result": bench_result, - "agg_result": agg_result, - "prefill_gpus": 16, - "decode_gpus": 16, - "aggregate_gpus": 0, - "expected_producer_sha": "a1b8c7af10c00e5ea40074aebdc0086189bbc064", - "logs_root": logs_root, - "validation_result": result_dir / "power_validation.json", - "require_power": True, - } - ] - - -def test_multinode_aggregation_maps_aggregate_deployment_to_agg_role( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -): - from infx.results.agentic import power_adapter - - logs_root = tmp_path / "logs" - result_dir = logs_root / "agentic" / "conc_8" - result_dir.mkdir(parents=True) - bench_result = result_dir / "agentic_power_concurrency_8.json" - bench_result.write_text(json.dumps({"max_concurrency": 8})) - agg_result = tmp_path / "agg_agentx_conc8.json" - agg_result.write_text( - json.dumps( - {"disagg": False, "num_prefill_gpu": 8, "num_decode_gpu": 0} - ), - encoding="utf-8", - ) - power_dir = logs_root / "power" - power_dir.mkdir(parents=True) - calls: list[dict] = [] - - def fake_run(**kwargs) -> int: - calls.append(kwargs) - return 0 - - monkeypatch.setattr(power_adapter, "run_multinode_power", fake_run) - - assert power_adapter.run_multinode_agentic_power( - result_dir=result_dir, - agg_result=agg_result, - power_dir=power_dir, - logs_root=logs_root, - expected_producer_sha="a" * 40, - require_power=True, - ) == 0 - assert calls == [ - { - "power_dir": power_dir, - "bench_result": bench_result, - "agg_result": agg_result, - "prefill_gpus": 0, - "decode_gpus": 0, - "aggregate_gpus": 8, - "expected_producer_sha": "a" * 40, - "logs_root": logs_root, - "validation_result": result_dir / "power_validation.json", - "require_power": True, - } - ] - - @pytest.mark.parametrize( "payload", [ diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index eb3d40f940..1abee4cad0 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -62,6 +62,7 @@ def run( aggregate_gpus=0, sha=PRODUCER_SHA, require_power=False, + expected_cpu_source=None, ): return apm.run( self.power_dir, @@ -74,6 +75,7 @@ def run( logs_root=self.logs_root, validation_result=self.validation_result, require_power=require_power, + expected_cpu_source=expected_cpu_source, ) def agg(self): @@ -942,6 +944,31 @@ def test_module_on_some_sockets_falls_back_to_grace(self, tmp_path): assert "total_module_energy_j" not in agg assert pkg.sidecar()["cpu"]["sensor_kind"] == "grace_socket" + @pytest.mark.parametrize("require_power", [False, True]) + @pytest.mark.parametrize("cpu", ["complete", "missing", "rail_only", "corrupt"]) + def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( + self, tmp_path, cpu, require_power + ): + pkg = build_package(tmp_path) + if cpu == "complete": + add_cpu_package(pkg, _cpu_rows()) + elif cpu == "rail_only": + add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) + elif cpu == "corrupt": + cpu_dir = add_cpu_package(pkg, _cpu_rows()) + (cpu_dir / "samples.csv").write_text("broken\n") + assert pkg.run(require_power=require_power, expected_cpu_source="acpi") == int( + require_power and cpu != "complete" + ) + assert _gpu_fields(pkg.agg()) == _reference_agg(tmp_path) + assert pkg.sidecar()["power_valid"] is True + assert pkg.agg()["cpu_power_valid"] == int(cpu == "complete") + if cpu == "complete": + assert_grace_keys(pkg.agg()) + else: + assert set(apm.CPU_METRIC_KEYS).isdisjoint(pkg.agg()) + assert pkg.sidecar()["cpu"]["reason_codes"] + def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): pkg = build_package(tmp_path) assert pkg.run() == 0 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 5549e231b3..2bad1e01f8 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9264,3 +9264,16 @@ description: - "Apply NVIDIA/srt-slurm#553 to the pinned submitter: pass the resolved recipe into setup validation so CPU-enabled jobs fail before sbatch when the exporter is missing, non-executable, or built for the wrong compute architecture." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 + +- config-keys: + - qwen3.5-fp8-gb200-dynamo-sglang + - qwen3.5-fp8-gb300-dynamo-sglang + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg + - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg + - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg + description: + - "Collect ACPI CPU power on Qwen3.5 NVL72 4P1D and 8P1D recipes, including noprof DCGM collection on 8P1D, with the pinned v2.40.2 CPU exporter. Place the 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock; retain 13 allocated nodes and 12 GPU workers. Require valid power for all selected Qwen3.5 fixed-sequence points." + - "During strict power qualification, require the CPU source declared by each selected recipe; reject missing or invalid CPU totals without changing the independent GPU verdict. Preserve best-effort behavior when no CPU source is declared." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From 94a7f53ba825f463c0e2643f4f3eaae298ec8906 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Fri, 2 Oct 2026 01:27:54 -0700 Subject: [PATCH 09/35] fix: retain AgentX CPU power audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：在 AgentX 结果中保留 CPU 功耗审计信息。 --- .../infx/results/agentic/power_adapter.py | 13 ++++++++++++- .../infx/tests/launch/test_launch_artifacts.py | 9 +++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/infx/results/agentic/power_adapter.py b/inferencex-e2e/infx/results/agentic/power_adapter.py index e65725b472..2d32dfa925 100644 --- a/inferencex-e2e/infx/results/agentic/power_adapter.py +++ b/inferencex-e2e/infx/results/agentic/power_adapter.py @@ -20,6 +20,7 @@ POWER_METRIC_SCHEMA_VERSION, with_power_metrics, ) +from infx.results.power.audit import audit_summary from infx.results.power.multinode import WINDOWS_DIRNAME, run as run_multinode_power from infx.results.power.single_node import ( _patch_power_result, @@ -481,7 +482,7 @@ def run_multinode_agentic_power( aggregate_gpus = prefill_gpus + decode_gpus prefill_gpus = 0 decode_gpus = 0 - return run_multinode_power( + status = run_multinode_power( power_dir=power_dir, bench_result=bench_result, agg_result=agg_result, @@ -494,6 +495,16 @@ def run_multinode_agentic_power( require_power=require_power, expected_cpu_source=expected_cpu_source, ) + try: + result = json.loads(agg_result.read_text()) + validation = json.loads(validation_result.read_text()) + source = "LOGS/" + validation_result.resolve().relative_to(logs_root.resolve()).as_posix() + result.update(audit_summary(validation, source)) + _write_json_atomic(agg_result, result) + except (OSError, ValueError, TypeError) as exc: + print(f"[agentx_power] Audit summary unavailable: {exc}", file=sys.stderr) + status = max(status, int(require_power)) + return status def main() -> int: diff --git a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py index e30f55f978..1f5ec3e094 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py @@ -179,3 +179,12 @@ def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path, assert result["cpu_power_valid"] == int(has_cpu) assert result["total_gpu_energy_j"] == 84000.0 assert (result_dir / "power_validation.json").is_file() + assert result["power_audit"]["source"] == "LOGS/agentic/conc_4/power_validation.json" + cpu = result["power_audit"]["cpu"] + if has_cpu: + assert cpu["sensor_kind"] == "grace_socket" + assert cpu["source"] == "acpi" + assert cpu["expected_sockets"] == cpu["observed_sockets"] == 4 + assert cpu["reason_codes"] == [] + else: + assert cpu["reason_codes"] == ["cpu_artifacts_missing"] From 508a892cc15c0b074d9f8f9f4c79d0c2d0bd9ccf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 15:54:48 -0700 Subject: [PATCH 10/35] chore: consolidate NVL72 power changelog entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 NVL72 功耗变更记录合并为一个条目。 --- inferencex-e2e/perf-changelog.yaml | 57 +++--------------------------- 1 file changed, 4 insertions(+), 53 deletions(-) diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index a2d8eb3187..e8a906b419 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9323,65 +9323,16 @@ - config-keys: - qwen3.5-fp8-gb200-dynamo-sglang - qwen3.5-fp8-gb300-dynamo-sglang - scenario-type: - - fixed-seq-len - description: - - "Enable srt-slurm's independent, best-effort CPU power leg on the 1P1D TP4/TP4 STP Qwen3.5 FP8 GB200 and GB300 recipes: telemetry.cpu_power_exporter with port 9405 (the schema default) and source acpi. Nothing else in either recipe changes; image, serving flags, topology, concurrency list, and the GPU DCGM exporter (port 9401 on GB200, 19401 on GB300, required: true) are untouched." - - "At the pinned srt-slurm v2.2.1 (984180e5) srtctl launches cpu-power-exporter --port 9405 --source acpi through srun on the bare host of every worker node, falling back to the ACPI-only Python exporter when make setup did not install the release binary, and the head-node collector writes LOGS/power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package. The leg has no required flag: a missing hwmon sensor or a dead exporter only leaves gaps in samples.csv and cannot fail the job or the GPU power verdict." - - "The multinode power validator integrates the Grace Power Socket series over the same formal window as GPU energy and adds cpu_power_valid plus the CPU-side keys (docs/results-and-ingestion.md, Measured Grace CPU-side power); power_valid and every GPU field stay unchanged. The v2.2.1 exporter classifies only the CPU, Grace, and SysIO socket labels, so these runs yield the Grace-side total rather than the module reading." - - "These are the smallest active non-Kimi-K3 NVL72 power lanes (2 nodes, 1 prefill + 1 decode host, 4 Grace sockets). The DSV4 8k1k GB200/GB300 lanes are not used: Single-turn 8k1k on DeepSeek-V4-Pro was deprecated after 2026-09-08 (MODELS.md; configs/deprecated/nvidia-dsv4-8k1k-master.yaml), so no sweep would exercise them." - - "在 Qwen3.5 FP8 GB200 与 GB300 的 1P1D TP4/TP4 STP recipe 上启用 srt-slurm 独立、尽力而为的 CPU 功耗测量环节：telemetry.cpu_power_exporter，port 9405（schema 默认值），source acpi。两份 recipe 其余内容不变：镜像、服务参数、拓扑、并发列表以及 GPU DCGM exporter（GB200 端口 9401、GB300 端口 19401，required: true）均未改动。在固定版本 srt-slurm v2.2.1（984180e5）下，srtctl 通过 srun 在每个 worker 节点的裸主机上启动 cpu-power-exporter --port 9405 --source acpi，若 make setup 未安装发布的二进制则回退到仅支持 ACPI 的 Python exporter；头节点采集器把 LOGS/power/cpu/samples.csv 和 cpu_manifest.json 写在 GPU DCGM 产物包旁。该环节没有 required 开关：缺少 hwmon 传感器或 exporter 失效只会让 samples.csv 出现空缺，不会导致作业失败或改变 GPU 功耗结论。多节点功耗校验器在与 GPU 能耗相同的正式测量窗口内对 Grace Power Socket 序列积分，新增 cpu_power_valid 与 CPU 侧字段（见 docs/results-and-ingestion_zh.md「NVL72 Grace CPU 侧实测功耗」）；power_valid 与所有 GPU 字段保持不变。v2.2.1 的 exporter 只识别 CPU、Grace、SysIO 三种 socket 标签，因此这些运行得到的是 Grace 侧总功耗而非模块读数。" - - "选择这两条 lane 是因为它们是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane（2 个节点，1 台 prefill 主机 + 1 台 decode 主机，4 个 Grace socket）。未采用 DSV4 8k1k 的 GB200/GB300 lane：DeepSeek-V4-Pro 的单轮 8k1k 场景已于 2026-09-08 之后弃用（见 MODELS.md 与 configs/deprecated/nvidia-dsv4-8k1k-master.yaml），sweep 不会再执行它们。" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 - -- config-keys: - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg scenario-type: + - fixed-seq-len - agentic-coding description: - - "Enable the independent best-effort ACPI CPU power exporter on existing Kimi K3 GB200 and GB300 power recipes. Preserve GPU sampling, serving parameters, topology, and concurrency; publish validated per-host GPU power from the same measurement window so cross-host workers can be modeled as physical trays." - - "Replace the four GB300 recipes legacy provider/default_frequency keys with collect_interval_ms: 1000 for the pinned srt-slurm 2 schema; the intended one-second sampling cadence is unchanged." - - "将四份 GB300 配方中的旧 provider/default_frequency 字段改为固定版本 srt-slurm 2 接受的 collect_interval_ms: 1000，保留一秒采样间隔。" - - "在已有 GPU 功耗采集的 Kimi K3 GB200 与 GB300 配方中启用独立、尽力而为的 ACPI CPU 功耗采集；保留 GPU 采样、服务参数、拓扑和并发。根据同一测量窗口的已验证能耗发布逐物理主机 GPU 功耗，使跨主机 worker 可以按实际计算 tray 建模。" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 - -- config-keys: - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 - - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg - - qwen3.5-fp8-gb200-dynamo-sglang - - qwen3.5-fp8-gb300-dynamo-sglang - description: - - "Pin cpu-power-exporter v2.40.2 for the existing Qwen3.5 and Kimi-K3 NVL72 CPU-telemetry recipes. The fixed binary discovers legacy ACPI hwmon meters; setup verifies the release asset checksum and fails if the pinned download cannot be installed. Preserve the merged noprof GPU profile and independent best-effort CPU verdict. Other recipe variants retain their current runtime settings." - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 - -- config-keys: - - qwen3.5-fp8-gb200-dynamo-sglang - - qwen3.5-fp8-gb300-dynamo-sglang - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg - - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg - description: - - "Apply NVIDIA/srt-slurm#553 to the pinned submitter: pass the resolved recipe into setup validation so CPU-enabled jobs fail before sbatch when the exporter is missing, non-executable, or built for the wrong compute architecture." - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 - -- config-keys: - - qwen3.5-fp8-gb200-dynamo-sglang - - qwen3.5-fp8-gb300-dynamo-sglang - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg - - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg - - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2 - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg - - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg - description: - - "Collect ACPI CPU power on Qwen3.5 NVL72 4P1D and 8P1D recipes, including noprof DCGM collection on 8P1D, with the pinned v2.40.2 CPU exporter. Place the 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock; retain 13 allocated nodes and 12 GPU workers. Require valid power for all selected Qwen3.5 fixed-sequence points." - - "During strict power qualification, require the CPU source declared by each selected recipe; reject missing or invalid CPU totals without changing the independent GPU verdict. Preserve best-effort behavior when no CPU source is declared." + - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.40.2. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." + - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." + - "Use collect_interval_ms: 1000 in the four Kimi K3 GB300 recipes to preserve their one-second GPU sampling cadence under the pinned srt-slurm 2 schema. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From a02c8aedb50c8cab28bab57925ef3b7b78afb1f2 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 16:05:58 -0700 Subject: [PATCH 11/35] fix: migrate Qwen NVL72 head placement to srt schema 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将两份 Qwen NVL72 配方的前端与基准客户端位置改为当前 schema。 --- .../sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 5 ++++- .../sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 5 ++++- inferencex-e2e/infx/tests/launch/test_srt_config.py | 8 ++++---- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index ca331ce895..d30f703786 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -22,7 +22,8 @@ dynamo: wheel: "1.5.0.dev20260917" frontend: type: dynamo - dedicated_node: true + placement: + node: dedicated enable_multiple_frontends: true num_additional_frontends: 4 nginx_container: nginx @@ -163,6 +164,8 @@ roles: benchmark: type: custom command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh + placement: + node: head env: TOKENIZER: /model ISL: '8192' diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index 1f4b417757..e5b6f0fa08 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -21,7 +21,8 @@ dynamo: wheel: "1.5.0.dev20260917" frontend: type: dynamo - dedicated_node: true + placement: + node: dedicated enable_multiple_frontends: true num_additional_frontends: 4 nginx_container: nginx @@ -162,6 +163,8 @@ roles: benchmark: type: custom command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh + placement: + node: head env: TOKENIZER: /model ISL: '8192' diff --git a/inferencex-e2e/infx/tests/launch/test_srt_config.py b/inferencex-e2e/infx/tests/launch/test_srt_config.py index e456ad8b7f..e6f94ff0c8 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_config.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_config.py @@ -252,14 +252,14 @@ def test_qwen_8p1d_power_keeps_the_collector_and_benchmark_on_the_reserved_head( monkeypatch.setattr("srtctl.core.runtime.get_slurm_nodelist", lambda: allocation) monkeypatch.setattr("srtctl.core.runtime.get_slurm_het_nodelists", lambda: None) nodes = Nodes.from_slurm( - frontend_dedicated_node=config.frontend.dedicated_node, - client_dedicated_node=config.benchmark.client_dedicated_node, - etcd_nats_dedicated_node=config.infra.etcd_nats_dedicated_node, + frontend_dedicated_node=config.frontend.placement.dedicated, + client_dedicated_node=config.benchmark.placement.dedicated, + etcd_nats_dedicated_node=False, colocate_dedicated_nodes=config.benchmark.colocate_with_frontend, ) assert config.engine_node_count == 12 assert not config.pool_services assert nodes.head == nodes.bench == nodes.infra == allocation[0] assert nodes.worker == nodes.compute == tuple(allocation[1:]) - assert config.benchmark.client_placement == "head" + assert config.benchmark.placement.location == "head" assert config.telemetry.enabled and config.telemetry.cpu_power_exporter.source == "acpi" From bbf1ad75659bde5b7f5abb7db9d7a612072079fb Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 16:13:40 -0700 Subject: [PATCH 12/35] chore: drop upstreamed srt-slurm 553 patch and pin exporter v2.47.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：移除已合入上游的 srt-slurm 553 补丁，exporter 固定为 v2.47.2。 --- inferencex-e2e/configs/nvidia-master.yaml | 38 +++++++++---------- .../553-cpu-exporter-submit-preflight.patch | 13 ------- .../runners/srt-slurm/patches/README.md | 1 - 3 files changed, 19 insertions(+), 33 deletions(-) delete mode 100644 inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 00dc3ce8a0..cc7b28eebb 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -4103,7 +4103,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 4 @@ -4119,7 +4119,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 16 @@ -4135,7 +4135,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 16 @@ -6581,7 +6581,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: # Referenced power recipes also collect best-effort Grace CPU telemetry. @@ -6609,7 +6609,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" dsv4-fp4-gb300-dynamo-trt-agentx: image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24 @@ -6758,7 +6758,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } @@ -6773,7 +6773,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [48] @@ -6787,7 +6787,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [72] @@ -6801,7 +6801,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [96] @@ -6815,7 +6815,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" dsv4-fp4-gb200-dynamo-sglang-agentic-agg: image: lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 model: deepseek-ai/DeepSeek-V4-Pro-0813 @@ -7094,7 +7094,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 4 @@ -7110,7 +7110,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 16 @@ -7126,7 +7126,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: num-worker: 1 tp: 16 @@ -8174,7 +8174,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: { num-worker: 1, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8188,7 +8188,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: { num-worker: 2, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8202,7 +8202,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8216,7 +8216,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: @@ -8246,7 +8246,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } @@ -8260,7 +8260,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.40.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" # Kimi-K3 MXFP4 B200 aggregated vLLM (TP8 x PP2, 2 nodes / 16 GPUs), agentic # coding. The native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does diff --git a/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch b/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch deleted file mode 100644 index d19797214c..0000000000 --- a/inferencex-e2e/runners/srt-slurm/patches/553-cpu-exporter-submit-preflight.patch +++ /dev/null @@ -1,13 +0,0 @@ -diff --git a/src/srtctl/cli/submit.py b/src/srtctl/cli/submit.py -index 5c213209..c21a6047 100755 ---- a/src/srtctl/cli/submit.py -+++ b/src/srtctl/cli/submit.py -@@ -1214,7 +1214,7 @@ def submit_with_orchestrator( - # Validate setup before submitting (not during dry-run) - srtctl_root = get_srtslurm_setting("srtctl_root") - srtctl_source = Path(srtctl_root) if srtctl_root else Path(__file__).parent.parent.parent.parent -- validate_setup(srtctl_source) -+ validate_setup(srtctl_source, config) - - if render_dir is not None: - return _render_to_dir( diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index eea40d2122..fcadc7bf88 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -8,4 +8,3 @@ Each patch is a temporary fix for an open upstream PR. When the PR merges and th | Patch | Upstream PR | Fix | |-------|-------------|-----| -| `553-cpu-exporter-submit-preflight.patch` | [NVIDIA/srt-slurm#553](https://github.com/NVIDIA/srt-slurm/pull/553) | Pass the resolved recipe into setup validation so CPU-enabled jobs reject unusable exporters before submission | From 93950d3bda789f1762991d5307e859eba945e810 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 16:13:44 -0700 Subject: [PATCH 13/35] feat: publish Grace CPU-only and SysIO rail power MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：发布 Grace CPU 核心与 SysIO 供电轨功耗。 --- inferencex-e2e/docs/results-and-ingestion.md | 15 ++-- .../docs/results-and-ingestion_zh.md | 11 +-- inferencex-e2e/infx/results/power/__init__.py | 4 ++ inferencex-e2e/infx/results/power/cpu_side.py | 68 ++++++++++++++----- .../power/test_aggregate_power_multinode.py | 28 +++++++- inferencex-e2e/perf-changelog.yaml | 4 +- 6 files changed, 100 insertions(+), 30 deletions(-) diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index 16ba8d8411..7fccb483ed 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -476,6 +476,8 @@ are: | `avg_total_cpu_power_w` | Sum over sockets of window-mean Grace-side watts | | `total_cpu_energy_j` | Grace-side energy over the window, all sockets | | `avg_total_module_power_w`, `total_module_energy_j` | Module watts and energy; only when a module series exists for every socket | +| `avg_total_cpu_rail_power_w`, `total_cpu_rail_energy_j` | ACPI `CPU Power Socket N` (cores, no SysIO) watts and energy; only when every Grace socket reports a gap-free rail | +| `avg_total_cpu_sysio_power_w`, `total_cpu_sysio_energy_j` | ACPI `SysIO Power Socket N` watts and energy; same condition | The Grace-side keys come from the Grace socket total (or, without ACPI, the DCGM CPU rail) even when the module sensor is preferred, so a module reading is never published as a Grace-side one. @@ -495,14 +497,15 @@ keys and leaves every GPU field unchanged. With `REQUIRE_POWER=1`, a recipe-decl window binds to the result, or the window's own contract checks failed). An overflowed CPU integration reuses the GPU leg's `non_finite_power_metric`, and `aggregate_result_missing` or `aggregate_result_unwritable` appears in both audits when the aggregate itself cannot be patched. -The srt-slurm source pin (`098e15ac`) writes the wide format and does not classify the -Module label, so these packages yield the Grace socket total; module keys require a producer -that classifies that label. The selected Qwen3.5 and Kimi-K3 CPU-telemetry recipes pass -`CPU_POWER_EXPORTER_RELEASE=v2.40.2` from their master-config `additional-settings` to +The srt-slurm source pin (`9d65c0ae`, v2.47.2) writes the wide format and does not classify the +Module label, so these packages yield the Grace socket total plus its CPU-only and SysIO rails; +module keys require a producer that classifies that label. A rail gap drops only that rail's keys +and never changes `cpu_power_valid`. The selected Qwen3.5 and Kimi-K3 CPU-telemetry recipes pass +`CPU_POWER_EXPORTER_RELEASE=v2.47.2` from their master-config `additional-settings` to `make setup`. Qwen3.5 8P1D keeps its frontend, benchmark client, and infra on the reserved head so their timestamps share the collector clock; its 13-node allocation still contains 12 GPU-worker nodes. This release includes legacy ACPI hwmon discovery. Setup verifies the downloaded asset checksum, replaces a cached binary with a different release marker, and fails on a pinned -download error. The local srt-slurm submission patch passes the resolved recipe to setup -validation, which rejects a missing, non-executable, or wrong-architecture binary. +download error. The pinned srt-slurm submit preflight (NVIDIA/srt-slurm#553) rejects a missing, +non-executable, or wrong-architecture binary before `sbatch`. Qualification still needs the setup log and actual exporter identity, complete same-window samples, and verified firmware sensor boundaries. A source pin or successful setup alone does not establish valid measurements. Without an expected CPU source, a package without `cpu/` adds no CPU metrics or CPU verdict. diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index be6c523b84..2379f2024d 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -458,6 +458,8 @@ HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePow | `avg_total_cpu_power_w` | 各 socket 窗口平均 Grace 侧功耗之和 | | `total_cpu_energy_j` | 窗口内所有 socket 的 Grace 侧能耗 | | `avg_total_module_power_w`、`total_module_energy_j` | 模块功耗与能耗；仅当每个 socket 都有模块序列时输出 | +| `avg_total_cpu_rail_power_w`、`total_cpu_rail_energy_j` | ACPI `CPU Power Socket N`（仅 CPU 核心，不含 SysIO）的功耗与能耗；仅当每个 Grace socket 都有无断档的该供电轨时输出 | +| `avg_total_cpu_sysio_power_w`、`total_cpu_sysio_energy_j` | ACPI `SysIO Power Socket N` 的功耗与能耗；输出条件相同 | 即使优先选用模块传感器，Grace 侧字段仍来自 Grace socket 总功耗（没有 ACPI 时来自 DCGM CPU 供电轨）， 因此模块读数永远不会被当作 Grace 侧读数发布。sidecar 的 `cpu` 块和聚合结果的 `power_audit.cpu` @@ -473,12 +475,13 @@ worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 produ `cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口 与该结果绑定，或窗口自身的契约检查失败）。CPU 积分溢出沿用 GPU 侧的 `non_finite_power_metric`；聚合 结果本身无法写入时，`aggregate_result_missing` 或 `aggregate_result_unwritable` 会同时出现在两侧的审计 -中。固定的 srt-slurm 源码版本（`098e15ac`）写出宽格式；采集与写出链路尚不识别 Module 标签，因此 -这些产物包只会得到 Grace socket 总功耗；需要 producer 支持该标签后才会出现模块字段。 +中。固定的 srt-slurm 源码版本（`9d65c0ae`，v2.47.2）写出宽格式；采集与写出链路尚不识别 Module 标签，因此 +这些产物包只会得到 Grace socket 总功耗及其 CPU 核心与 SysIO 供电轨；需要 producer 支持该标签后才会出现模块字段。 +某条供电轨出现断档时只会去掉该供电轨的字段，不影响 `cpu_power_valid`。 选中的 Qwen3.5 与 Kimi-K3 CPU 采集配方通过主配置的 `additional-settings`，向 `make setup` -传入 `CPU_POWER_EXPORTER_RELEASE=v2.40.2`。Qwen3.5 8P1D 将前端、压测客户端和基础服务放在预留的 head 节点，与采集器共用时钟；仍分配 13 个节点，其中 12 个为 GPU worker。该版本包含旧版 ACPI hwmon 传感器发现修复。 +传入 `CPU_POWER_EXPORTER_RELEASE=v2.47.2`。Qwen3.5 8P1D 将前端、压测客户端和基础服务放在预留的 head 节点，与采集器共用时钟；仍分配 13 个节点，其中 12 个为 GPU worker。该版本包含旧版 ACPI hwmon 传感器发现修复。 安装过程校验下载工件的校验和，替换版本标记不匹配的缓存二进制，并在固定版本下载失败时退出。 -本地 srt-slurm 提交补丁将已解析的配方传入安装检查，从而在提交前拒绝缺失、不可执行或架构 +固定的 srt-slurm 已包含提交前检查（NVIDIA/srt-slurm#553），会在 `sbatch` 前拒绝缺失、不可执行或架构 不匹配的二进制。验收仍需保留安装日志与实际 exporter 身份，检查完整的同窗口样本，并核实固件传感器的测量范围。源码版本或安装成功均不能证明 测量有效。未声明 CPU 来源时，没有 `cpu/` 的产物包不新增 CPU 指标或 CPU 有效性结论；通过 GPU 校验的产物包 diff --git a/inferencex-e2e/infx/results/power/__init__.py b/inferencex-e2e/infx/results/power/__init__.py index 6e5fb04510..fe796f0368 100644 --- a/inferencex-e2e/infx/results/power/__init__.py +++ b/inferencex-e2e/infx/results/power/__init__.py @@ -42,6 +42,10 @@ "total_cpu_energy_j", "avg_total_module_power_w", "total_module_energy_j", + "avg_total_cpu_rail_power_w", + "total_cpu_rail_energy_j", + "avg_total_cpu_sysio_power_w", + "total_cpu_sysio_energy_j", ) diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index 6ec7a8628f..bcf84ecfaf 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -9,7 +9,9 @@ Each (hostname, socket) headline series is integrated over the GPU leg's bound formal window with the shared trapezoid and boundary interpolation and -published as additive metrics next to the GPU-board numbers. The leg is +published as additive metrics next to the GPU-board numbers. The v2 ACPI +CPU-only and SysIO rails are integrated the same way when every socket reports +them; a rail with gaps drops only its own metrics. The leg is best-effort by contract: any failure records ``cpu_power_valid=0`` with reason codes and leaves every GPU field untouched. Strict qualification also requires a recipe-declared CPU source; GPU-only callers keep best-effort CPU @@ -63,9 +65,12 @@ SENSOR_MODULE = "module" SENSOR_GRACE = "grace_socket" SENSOR_DCGM = "dcgm_cpu_rail" -# Headline preference per socket. Component rails (cpu_rail, soc, dram) are -# reference breakdowns and never feed a published metric. +# Headline preference per socket. ACPI component rails ride on the Grace row +# and never select the headline or its verdict. HEADLINE_PREFERENCE = (SENSOR_MODULE, SENSOR_GRACE, SENSOR_DCGM) +# v2 Grace-row column -> published rail family: "CPU Power Socket N" (cores, +# no SysIO) and "SysIO Power Socket N". dram_w stays blank on NVL72 firmware. +_RAIL_COLUMNS = {"cpu_rail_w": "cpu_rail", "soc_w": "cpu_sysio"} # Grace-side kinds, in fallback order, that feed the ``*_cpu_*`` keys. _GRACE_SIDE_PREFERENCE = (SENSOR_GRACE, SENSOR_DCGM) @@ -95,6 +100,7 @@ class CpuSampleRow: sensor: str socket_id: int power_w: float + rails: tuple[tuple[str, float], ...] = () @dataclass @@ -166,6 +172,18 @@ def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: return None if socket_id < 0: return None + rails = [] + for column, family in _RAIL_COLUMNS.items(): + cell = raw[_HEADERS[version].index(column)] if column in _HEADERS[version] else "" + if not cell: + continue + try: + watts = float(cell) + except ValueError: + return None + if not math.isfinite(watts) or watts < 0: + return None + rails.append((family, watts)) return CpuSampleRow( timestamp_unix=timestamp_unix, hostname=hostname, @@ -173,6 +191,7 @@ def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: sensor=sensor, socket_id=socket_id, power_w=power_w, + rails=tuple(rails), ) @@ -221,11 +240,15 @@ def _headline_series( kind = classify_sensor(row.sensor) if kind is None: continue - points = series.setdefault((row.hostname, row.socket_id), {}).setdefault(kind, {}) + by_kind = series.setdefault((row.hostname, row.socket_id), {}) + points = by_kind.setdefault(kind, {}) if row.timestamp_unix in points: duplicate = True points[row.timestamp_unix] = row.power_w sources.setdefault(kind, set()).add(row.source) + if kind == SENSOR_GRACE: + for family, watts in row.rails: + by_kind.setdefault(family, {})[row.timestamp_unix] = watts return series, sources, duplicate @@ -293,23 +316,33 @@ def validate_cpu_leg( audit.invalidate("cpu_sensor_source_mismatch") return audit + if SENSOR_GRACE in feeds: + feeds.update({rail: rail for rail in _RAIL_COLUMNS.values() if rail in common_kinds}) start, end = window energy: dict[str, float] = {} + dropped_rails: set[str] = set() for (host, socket), by_kind in sorted(series.items()): for kind, family in feeds.items(): label = f"{host}/socket{socket}/{kind}" samples = sorted(by_kind[kind].items()) sequence = _bracketing_sequence(tuple(t for t, _ in samples), start, end) - if sequence is None: - audit.invalidate("cpu_window_not_bracketed") - continue - gap = max( - (later - earlier for earlier, later in itertools.pairwise(sequence)), - default=0.0, + gap = ( + None + if sequence is None + else max( + (later - earlier for earlier, later in itertools.pairwise(sequence)), + default=0.0, + ) ) - audit.per_series_max_sample_gap_s[label] = gap - if gap > MAX_SAMPLE_GAP_SECONDS: - audit.invalidate("cpu_sample_gap_exceeded") + if gap is not None: + audit.per_series_max_sample_gap_s[label] = gap + if gap is None or gap > MAX_SAMPLE_GAP_SECONDS: + if family in _RAIL_COLUMNS.values(): + dropped_rails.add(family) + elif gap is None: + audit.invalidate("cpu_window_not_bracketed") + else: + audit.invalidate("cpu_sample_gap_exceeded") continue joules = _integrate_device(samples, start_unix=start, end_unix=end) audit.per_series_energy_j[label] = joules @@ -317,6 +350,8 @@ def validate_cpu_leg( if audit.reason_codes: audit.per_series_energy_j = {} return audit + for family in dropped_rails: + energy.pop(family, None) duration = end - start metrics: dict[str, float] = {} @@ -324,9 +359,10 @@ def validate_cpu_leg( metrics["avg_cpu_socket_power_w"] = energy["cpu"] / duration / len(expected_keys) metrics["avg_total_cpu_power_w"] = energy["cpu"] / duration metrics["total_cpu_energy_j"] = energy["cpu"] - if "module" in energy: - metrics["avg_total_module_power_w"] = energy["module"] / duration - metrics["total_module_energy_j"] = energy["module"] + for family in ("module", *_RAIL_COLUMNS.values()): + if family in energy: + metrics[f"avg_total_{family}_power_w"] = energy[family] / duration + metrics[f"total_{family}_energy_j"] = energy[family] if any(not math.isfinite(value) for value in metrics.values()): audit.invalidate("non_finite_power_metric") return audit diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index 3b73b28c26..d91ec4fa91 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -785,7 +785,7 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) MODULE_W = 1500.0 DCGM_W = 300.0 # Component rails never feed a headline metric; the values are chosen so any -# leak into a published key is visible against the Grace/module constants. +# leak into a headline key is visible against the Grace/module constants. RAILS_W = {"cpu_rail": 200.0, "soc": 50.0, "dram": 30.0} GRACE_KINDS = {"grace": None, **RAILS_W} @@ -883,6 +883,13 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): assert_grace_keys(agg) assert "avg_total_module_power_w" not in agg assert "total_module_energy_j" not in agg + rails = { + "avg_total_cpu_rail_power_w": 800.0, + "total_cpu_rail_energy_j": 48000.0, + "avg_total_cpu_sysio_power_w": 200.0, + "total_cpu_sysio_energy_j": 12000.0, + } + assert {k: agg[k] for k in rails if k in agg} == (rails if fmt == "v2" else {}) assert _gpu_fields(agg) == _reference_agg(tmp_path) cpu = pkg.sidecar()["cpu"] @@ -893,8 +900,12 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): assert cpu["expected_sockets"] == 4 assert cpu["observed_sockets"] == 4 assert cpu["sample_row_count"] == len(rows) + series_kinds = ("grace_socket", "cpu_rail", "cpu_sysio") if fmt == "v2" else ("grace_socket",) assert set(cpu["per_series_energy_j"]) == { - f"{host}/socket{socket}/grace_socket" for host in CPU_HOSTS for socket in (0, 1) + f"{host}/socket{socket}/{kind}" + for host in CPU_HOSTS + for socket in (0, 1) + for kind in series_kinds } assert audit_summary(pkg.sidecar(), "power_validation.json")["power_audit"]["cpu"] == { "sensor_kind": "grace_socket", @@ -905,6 +916,19 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): "reason_codes": [], } + def test_rail_gap_drops_only_that_rail(self, tmp_path): + pkg = build_package(tmp_path) + rows = _cpu_rows({host: {"grace": None} for host in CPU_HOSTS}, "v2") + for row in rows: + if row[2] == "node-d" and row[5] == 0 and 1020.0 <= float(row[1]) <= 1030.0: + row[7] = "" + add_cpu_package(pkg, rows, CPU_HEADER_V2) + assert pkg.run(require_power=True) == 0 + agg = pkg.agg() + assert_grace_keys(agg) + assert "avg_total_cpu_rail_power_w" not in agg + assert agg["avg_total_cpu_sysio_power_w"] == 200.0 + def test_dcgm_only_package_uses_cpu_rail_kind(self, tmp_path): pkg = build_package(tmp_path) add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 6d1cccc206..a2ea67b390 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9488,7 +9488,7 @@ - fixed-seq-len - agentic-coding description: - - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.40.2. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." - - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." + - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.47.2, matching the srt-slurm v2.47.2 source pin. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." + - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict, including the ACPI Grace socket total and its CPU-only and SysIO rails. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." - "Use collect_interval_ms: 1000 in the four Kimi K3 GB300 recipes to preserve their one-second GPU sampling cadence under the pinned srt-slurm 2 schema. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From 2a02d1de9fbf5f8b6f897d9d5f064d5699262a89 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 16:18:18 -0700 Subject: [PATCH 14/35] test: prune redundant NVL72 power tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：精简 NVL72 功耗冗余测试。 --- .../infx/tests/launch/fake_slurm.py | 1 - .../tests/launch/test_launch_artifacts.py | 38 +--- .../infx/tests/launch/test_launch_policy.py | 22 -- .../infx/tests/launch/test_srt_config.py | 34 ---- .../infx/tests/launch/test_srt_driver.py | 21 +- .../results/power/test_aggregate_power.py | 1 + .../power/test_aggregate_power_multinode.py | 192 +++--------------- 7 files changed, 38 insertions(+), 271 deletions(-) diff --git a/inferencex-e2e/infx/tests/launch/fake_slurm.py b/inferencex-e2e/infx/tests/launch/fake_slurm.py index 27b76db231..004e91e4d6 100644 --- a/inferencex-e2e/infx/tests/launch/fake_slurm.py +++ b/inferencex-e2e/infx/tests/launch/fake_slurm.py @@ -38,7 +38,6 @@ """, "make": r""" printf '%s\n' "$*" >> "$FAKE_LOG_DIR/make.log" -printf '%s\n' "${CPU_POWER_EXPORTER_RELEASE-}" >> "$FAKE_LOG_DIR/cpu-exporter-release.log" if [[ -n "${FAKE_MAKE_RC:-}" ]]; then echo "make setup broke"; exit "$FAKE_MAKE_RC"; fi # First run leaves a truncated NATS download behind, as flaky GitHub releases do. if [[ -n "${FAKE_TRUNCATED_NATS:-}" && ! -e configs/.downloaded ]]; then diff --git a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py index 1f5ec3e094..d18d4cba38 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py @@ -116,13 +116,9 @@ def test_power_collection_records_the_job_status_and_stages_results_either_way(t assert (workspace / "point_conc4.json").exists() -@pytest.mark.parametrize("has_cpu", [False, True]) -@pytest.mark.parametrize("disagg", [False, True]) -def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path, has_cpu, disagg): +def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path): from infx.tests.results.power.test_aggregate_power_multinode import ( PRODUCER_SHA, - _cpu_rows, - add_cpu_package, build_package, ) @@ -140,26 +136,14 @@ def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path, manifest_path = pkg.power_dir / "manifest.json" manifest = json.loads(manifest_path.read_text()) manifest["window_validations"][0]["window_file"] = f"windows/{formal_result.name}" + for device in manifest["expected_devices"]: + device["assignments"][0].update(worker_role="agg", het_group=0) manifest_path.write_text(json.dumps(manifest)) - if not disagg: - manifest_path = pkg.power_dir / "manifest.json" - manifest = json.loads(manifest_path.read_text()) - for device in manifest["expected_devices"]: - device["assignments"][0].update(worker_role="agg", het_group=0) - manifest_path.write_text(json.dumps(manifest)) - if has_cpu: - add_cpu_package(pkg, _cpu_rows()) source, workspace = tmp_path / "source", tmp_path / "output" source.mkdir() workspace.mkdir() (source / "point_conc4.json").write_text( - json.dumps( - { - "disagg": disagg, - "num_prefill_gpu": 2 if disagg else 4, - "num_decode_gpu": 2 if disagg else 0, - } - ) + json.dumps({"disagg": False, "num_prefill_gpu": 4, "num_decode_gpu": 0}) ) status = JobStatus(JobState.SUCCEEDED, "COMPLETED|0:0", 0) assert collect_agentic_power_results( @@ -173,18 +157,10 @@ def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path, [4], results_python=sys.executable, expected_cpu_source="acpi", - ) == int(not has_cpu) + ) == 1 result = json.loads((workspace / "point_conc4.json").read_text()) assert result["power_valid"] == 1 - assert result["cpu_power_valid"] == int(has_cpu) + assert result["cpu_power_valid"] == 0 assert result["total_gpu_energy_j"] == 84000.0 - assert (result_dir / "power_validation.json").is_file() assert result["power_audit"]["source"] == "LOGS/agentic/conc_4/power_validation.json" - cpu = result["power_audit"]["cpu"] - if has_cpu: - assert cpu["sensor_kind"] == "grace_socket" - assert cpu["source"] == "acpi" - assert cpu["expected_sockets"] == cpu["observed_sockets"] == 4 - assert cpu["reason_codes"] == [] - else: - assert cpu["reason_codes"] == ["cpu_artifacts_missing"] + assert result["power_audit"]["cpu"]["reason_codes"] == ["cpu_artifacts_missing"] diff --git a/inferencex-e2e/infx/tests/launch/test_launch_policy.py b/inferencex-e2e/infx/tests/launch/test_launch_policy.py index c227f8d798..a3d8d128d3 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_policy.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_policy.py @@ -199,25 +199,3 @@ def test_cluster_settings_beat_the_host_but_never_the_points_settings(chosen, ex env = policy.runtime_env(cluster, request, {"LATER": "later", "SHARED": "later"}) assert {name: env[name] for name in HOST} == expected - - -@pytest.mark.parametrize( - "enabled,source,expected", - [ - (True, "acpi", "acpi"), - (False, "acpi", None), - (True, "dcgm", "dcgm"), - ], -) -def test_power_recipe_resolves_enabled_cpu_source(tmp_path, enabled, source, expected): - telemetry = POWER_RECIPE.replace("enabled: true", f"enabled: {str(enabled).lower()}") - _mirror( - tmp_path, "qwen/cpu.yaml", telemetry + (f" cpu_power_exporter:\n source: {source}\n") - ) - request = _request( - GITHUB_WORKSPACE=str(tmp_path), - CONFIG_FILE="recipes/qwen/cpu.yaml", - IS_AGENTIC="0", - FRAMEWORK="dynamo-sglang", - ) - assert resolve_power("gb300-nv", MULTI, request).expected_cpu_source == expected diff --git a/inferencex-e2e/infx/tests/launch/test_srt_config.py b/inferencex-e2e/infx/tests/launch/test_srt_config.py index e6f94ff0c8..273ae27422 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_config.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_config.py @@ -229,37 +229,3 @@ def rsync(source: Path, staging: Path, *, exclude: tuple[str, ...]) -> Path: assert staged == tmp_path / "runs/infmax-workspace-1-1-abc" assert sorted(str(p.relative_to(staged)) for p in staged.rglob("*") if p.is_file()) == sorted(kept) - - -@pytest.mark.parametrize("sku", ["gb200", "gb300"]) -def test_qwen_8p1d_power_keeps_the_collector_and_benchmark_on_the_reserved_head( - tmp_path, monkeypatch, sku -): - from srtctl.core.config import load_config - from srtctl.core.runtime import Nodes - - monkeypatch.setattr("srtctl.core.config.load_cluster_config", lambda: None) - recipe = ROOT / ( - f"benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/{sku}-fp8/8k1k/" - "disagg-8p1d-dep4-dep16-stp.yaml" - ) - staged = tmp_path / "recipes" / recipe.name - staged.parent.mkdir() - shutil.copyfile(recipe, staged) - prepare_recipe(tmp_path, f"recipes/{recipe.name}", "qwen-8p1d", None, "2048 4096") - config = load_config(staged) - allocation = [f"node-{index}" for index in range(13)] - monkeypatch.setattr("srtctl.core.runtime.get_slurm_nodelist", lambda: allocation) - monkeypatch.setattr("srtctl.core.runtime.get_slurm_het_nodelists", lambda: None) - nodes = Nodes.from_slurm( - frontend_dedicated_node=config.frontend.placement.dedicated, - client_dedicated_node=config.benchmark.placement.dedicated, - etcd_nats_dedicated_node=False, - colocate_dedicated_nodes=config.benchmark.colocate_with_frontend, - ) - assert config.engine_node_count == 12 - assert not config.pool_services - assert nodes.head == nodes.bench == nodes.infra == allocation[0] - assert nodes.worker == nodes.compute == tuple(allocation[1:]) - assert config.benchmark.placement.location == "head" - assert config.telemetry.enabled and config.telemetry.cpu_power_exporter.source == "acpi" diff --git a/inferencex-e2e/infx/tests/launch/test_srt_driver.py b/inferencex-e2e/infx/tests/launch/test_srt_driver.py index af963b065c..ee46cc4655 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_driver.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_driver.py @@ -456,16 +456,6 @@ def test_eval_only_runs_the_eval_recipe_with_real_verification(harness): assert list(workspace.glob("point-identity_*.json")) == [] -def test_point_cpu_exporter_release_reaches_setup(harness): - env = lane_env( - harness, "gb200-nv", MODEL_PREFIX="qwen3.5", PRECISION="fp8", FRAMEWORK="dynamo-sglang", - MODEL="Qwen/Qwen3.5-397B-A17B-FP8", CPU_POWER_EXPORTER_RELEASE="fixture-release", - PREFILL_ADDITIONAL_SETTINGS='["CPU_POWER_EXPORTER_RELEASE=fixture-release"]', - ) # fmt: skip - assert_ok(launch(env, harness.config, harness.workspace)) - assert lines(harness.logs, "cpu-exporter-release") == ["fixture-release"] - - def test_setup_retries_only_after_discarding_a_truncated_archive(harness): env = lane_env( harness, "h200-dgxc", MODEL_PREFIX="dsr1", PRECISION="fp8", FRAMEWORK="dynamo-sglang", @@ -585,13 +575,10 @@ def test_multinode_eval_overrides_image_offline_mode_and_host_model_path(harness ] -@pytest.mark.parametrize("has_cpu", [False, True]) -def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkeypatch, has_cpu): +def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkeypatch): from infx.results.fixed_sequence import aggregate_power_result from infx.tests.results.power.test_aggregate_power_multinode import ( PRODUCER_SHA, - _cpu_rows, - add_cpu_package, build_package, ) @@ -611,8 +598,6 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey assert_ok(launch(env, harness.config, harness.workspace)) exported = dict(line.split("=", 1) for line in github_env.read_text().splitlines()) pkg = build_package(harness.tmp / "power-input") - if has_cpu: - add_cpu_package(pkg, _cpu_rows()) monkeypatch.chdir(harness.tmp) result_env = { **exported, @@ -625,7 +610,7 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey "POWER_ARTIFACT_DIR": str(pkg.power_dir), "POWER_RESULT_ROOT": str(pkg.logs_root), } - assert aggregate_power_result(result_env, pkg.bench_result, pkg.agg_result) == int(not has_cpu) + assert aggregate_power_result(result_env, pkg.bench_result, pkg.agg_result) == 1 aggregate = pkg.agg() assert aggregate["power_valid"] == 1 - assert aggregate["cpu_power_valid"] == int(has_cpu) + assert aggregate["cpu_power_valid"] == 0 diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power.py index 6f483d8799..49e0906036 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power.py @@ -33,6 +33,7 @@ def test_power_replacement_removes_stale_metrics(tmp_path, patch_validated_power "hw": "fixture", "avg_power_w": 99, "total_gpu_energy_j": 50, "power_invalid_reasons": ["stale"], "power_audit": {"source": "previous-run.json"}, + "cpu_power_valid": 1, "total_cpu_energy_j": 1.0, "workers": [{"role": "agg"}], })) patch_validated_power(path, power_valid=valid, metrics={ "avg_power_w": 12.34567, "joules_per_output_token": 0.12345678, diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index d91ec4fa91..ce87eae105 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -782,18 +782,14 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) ] CPU_HEADER_V2 = CPU_HEADER_V1[:7] + ["cpu_rail_w", "soc_w", "dram_w", "total_power_w"] GRACE_W = {"node-d": 500.0, "node-p": 700.0} -MODULE_W = 1500.0 -DCGM_W = 300.0 # Component rails never feed a headline metric; the values are chosen so any -# leak into a headline key is visible against the Grace/module constants. +# leak into a headline key is visible against the Grace constants. RAILS_W = {"cpu_rail": 200.0, "soc": 50.0, "dram": 30.0} GRACE_KINDS = {"grace": None, **RAILS_W} _SENSORS = { # (source, v1 firmware OEM label, v2 collector sensor name) "grace": ("acpi", "Grace Power Socket {s}", "CPU{s}:cpuSidePowerUsageW"), - "module": ("acpi", "Module Power Socket {s}", "Module Power Socket {s}"), - "dcgm": ("dcgm", "CPU{s}:cpuPowerUsageW", "CPU{s}:cpuPowerUsageW"), "cpu_rail": ("acpi", "CPU Power Socket {s}", "CPU{s}:cpuRailPowerUsageW"), "soc": ("acpi", "SysIO Power Socket {s}", "CPU{s}:socPowerUsageW"), "dram": ("acpi", "DRAM Power Socket {s}", "CPU{s}:dramPowerUsageW"), @@ -805,8 +801,7 @@ def _cpu_row(ts, host, socket, kind, watts, fmt): sensor = (label_v1 if fmt == "v1" else label_v2).format(s=socket) if fmt == "v1": return [1, repr(ts), host, source, sensor, socket, repr(watts), ""] - rails = ["", "", ""] if source == "dcgm" else [repr(w) for w in RAILS_W.values()] - return [2, repr(ts), host, source, sensor, socket, repr(watts), *rails, ""] + return [2, repr(ts), host, source, sensor, socket, repr(watts), *map(repr, RAILS_W.values()), ""] def _cpu_rows(kinds_by_host=None, fmt="v1", *, hosts=CPU_HOSTS): @@ -869,27 +864,20 @@ def assert_grace_keys(agg): class TestCpuSidePower: - @pytest.mark.parametrize("fmt, header", [("v1", CPU_HEADER_V1), ("v2", CPU_HEADER_V2)]) - def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): + def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path): from infx.results.power.audit import audit_summary pkg = build_package(tmp_path) - kinds = None if fmt == "v1" else {host: {"grace": None} for host in CPU_HOSTS} - rows = _cpu_rows(kinds, fmt) - add_cpu_package(pkg, rows, header) + rows = _cpu_rows({host: {"grace": None} for host in CPU_HOSTS}, "v2") + add_cpu_package(pkg, rows, CPU_HEADER_V2) assert pkg.run(require_power=True) == 0 agg = pkg.agg() assert_grace_keys(agg) - assert "avg_total_module_power_w" not in agg - assert "total_module_energy_j" not in agg - rails = { - "avg_total_cpu_rail_power_w": 800.0, - "total_cpu_rail_energy_j": 48000.0, - "avg_total_cpu_sysio_power_w": 200.0, - "total_cpu_sysio_energy_j": 12000.0, - } - assert {k: agg[k] for k in rails if k in agg} == (rails if fmt == "v2" else {}) + assert agg["avg_total_cpu_rail_power_w"] == 800.0 + assert agg["total_cpu_rail_energy_j"] == 48000.0 + assert agg["avg_total_cpu_sysio_power_w"] == 200.0 + assert agg["total_cpu_sysio_energy_j"] == 12000.0 assert _gpu_fields(agg) == _reference_agg(tmp_path) cpu = pkg.sidecar()["cpu"] @@ -900,12 +888,11 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path, fmt, header): assert cpu["expected_sockets"] == 4 assert cpu["observed_sockets"] == 4 assert cpu["sample_row_count"] == len(rows) - series_kinds = ("grace_socket", "cpu_rail", "cpu_sysio") if fmt == "v2" else ("grace_socket",) assert set(cpu["per_series_energy_j"]) == { f"{host}/socket{socket}/{kind}" for host in CPU_HOSTS for socket in (0, 1) - for kind in series_kinds + for kind in ("grace_socket", "cpu_rail", "cpu_sysio") } assert audit_summary(pkg.sidecar(), "power_validation.json")["power_audit"]["cpu"] == { "sensor_kind": "grace_socket", @@ -929,61 +916,15 @@ def test_rail_gap_drops_only_that_rail(self, tmp_path): assert "avg_total_cpu_rail_power_w" not in agg assert agg["avg_total_cpu_sysio_power_w"] == 200.0 - def test_dcgm_only_package_uses_cpu_rail_kind(self, tmp_path): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) - assert pkg.run() == 0 - agg = pkg.agg() - assert agg["cpu_power_valid"] == 1 - assert agg["avg_cpu_socket_power_w"] == 300.0 - assert agg["avg_total_cpu_power_w"] == 1200.0 - assert agg["total_cpu_energy_j"] == 72000.0 - cpu = pkg.sidecar()["cpu"] - assert (cpu["sensor_kind"], cpu["source"]) == ("dcgm_cpu_rail", "dcgm") - - def test_module_on_every_socket_is_preferred_and_grace_keys_stay_grace(self, tmp_path): - pkg = build_package(tmp_path) - add_cpu_package( - pkg, _cpu_rows({host: {**GRACE_KINDS, "module": MODULE_W} for host in CPU_HOSTS}) - ) - assert pkg.run() == 0 - agg = pkg.agg() - assert_grace_keys(agg) - assert agg["avg_total_module_power_w"] == 6000.0 - assert agg["total_module_energy_j"] == 360000.0 - cpu = pkg.sidecar()["cpu"] - assert cpu["sensor_kind"] == "module" - assert set(cpu["per_series_energy_j"]) == { - f"{host}/socket{socket}/{kind}" - for host in CPU_HOSTS - for socket in (0, 1) - for kind in ("module", "grace_socket") - } - - def test_module_on_some_sockets_falls_back_to_grace(self, tmp_path): - pkg = build_package(tmp_path) - kinds = {"node-d": {**GRACE_KINDS, "module": MODULE_W}, "node-p": GRACE_KINDS} - add_cpu_package(pkg, _cpu_rows(kinds)) - assert pkg.run() == 0 - agg = pkg.agg() - assert_grace_keys(agg) - assert "avg_total_module_power_w" not in agg - assert "total_module_energy_j" not in agg - assert pkg.sidecar()["cpu"]["sensor_kind"] == "grace_socket" - - @pytest.mark.parametrize("require_power", [False, True]) - @pytest.mark.parametrize("cpu", ["complete", "missing", "rail_only", "corrupt"]) + @pytest.mark.parametrize( + "cpu, require_power", [("missing", True), ("complete", True), ("missing", False)] + ) def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( self, tmp_path, cpu, require_power ): pkg = build_package(tmp_path) if cpu == "complete": add_cpu_package(pkg, _cpu_rows()) - elif cpu == "rail_only": - add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) - elif cpu == "corrupt": - cpu_dir = add_cpu_package(pkg, _cpu_rows()) - (cpu_dir / "samples.csv").write_text("broken\n") assert pkg.run(require_power=require_power, expected_cpu_source="acpi") == int( require_power and cpu != "complete" ) @@ -1004,16 +945,6 @@ def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): assert set(apm.CPU_METRIC_KEYS).isdisjoint(agg) assert "cpu" not in pkg.sidecar() - def test_stale_cpu_keys_are_stripped_on_rerun(self, tmp_path): - pkg = build_package(tmp_path) - pkg.agg_result.write_text( - json.dumps({"hw": "gb200", "conc": 4, "cpu_power_valid": 1, "total_cpu_energy_j": 1.0}) - ) - assert pkg.run() == 0 - assert _gpu_fields(pkg.agg()) == _reference_agg(tmp_path) - assert "cpu_power_valid" not in pkg.agg() - assert "total_cpu_energy_j" not in pkg.agg() - def _drop(self, rows, host, socket, predicate): return [ row @@ -1026,37 +957,18 @@ def tampered(self, request, tmp_path): kind = request.param pkg = build_package(tmp_path) rows = _cpu_rows() - header, manifest_text = CPU_HEADER_V1, None - if kind == "header": - header = [*CPU_HEADER_V1[:6], "watts", "total_power_w"] - elif kind == "socket": + if kind == "socket": rows = self._drop(rows, "node-p", 1, lambda ts: True) elif kind == "gap": rows = self._drop(rows, "node-d", 0, lambda ts: 1020.0 <= ts <= 1024.0) - elif kind == "unbracketed": - rows = self._drop(rows, "node-d", 0, lambda ts: ts <= WINDOW_START) - elif kind == "mixed": - rows = _cpu_rows({"node-d": {"grace": None}, "node-p": {"dcgm": DCGM_W}}) - elif kind == "manifest": - manifest_text = "{broken" - elif kind == "malformed": - rows = [*rows, [1, repr(FIRST_TS), "node-d", "acpi", "Grace Power Socket 0", 0, "n/a", ""]] - cpu_dir = add_cpu_package(pkg, rows, header, manifest_text) - if kind == "samples_missing": - (cpu_dir / "samples.csv").unlink() + add_cpu_package(pkg, rows) return pkg @pytest.mark.parametrize( "tampered, reason", [ - ("header", "cpu_samples_header_mismatch"), ("socket", "cpu_socket_count_mismatch"), ("gap", "cpu_sample_gap_exceeded"), - ("unbracketed", "cpu_window_not_bracketed"), - ("mixed", "cpu_sensor_kind_mixed"), - ("manifest", "cpu_manifest_invalid"), - ("malformed", "cpu_samples_malformed"), - ("samples_missing", "cpu_samples_missing"), ], indirect=["tampered"], ) @@ -1073,56 +985,27 @@ def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, r assert reason in cpu["reason_codes"] assert pkg.sidecar()["power_valid"] is True - @pytest.mark.parametrize( - "gpu_gap, sha, gpu_reason", - [ - (True, PRODUCER_SHA, "package_recompute_invalid"), - (False, "b" * 40, "producer_commit_mismatch"), - (False, None, "producer_pin_missing"), - ], - ) - def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path, gpu_gap, sha, gpu_reason): - """No GPU verdict, the producer pin included, reaches cpu_power_valid.""" + def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path): + """A GPU coverage failure withholds GPU energy but not cpu_power_valid.""" pkg = build_package(tmp_path) add_cpu_package(pkg, _cpu_rows()) - if gpu_gap: - _rewrite_samples( - pkg, - lambda body: [ - row - for row in body - if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) - ], - ) - assert pkg.run(sha=sha) == 0 + _rewrite_samples( + pkg, + lambda body: [ + row + for row in body + if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) + ], + ) + assert pkg.run() == 0 agg = pkg.agg() assert agg["power_valid"] == 0 assert "total_gpu_energy_j" not in agg assert_grace_keys(agg) sidecar = pkg.sidecar() - assert gpu_reason in sidecar["reasons"] + assert "package_recompute_invalid" in sidecar["reasons"] assert sidecar["cpu"]["reason_codes"] == [] - def test_window_unavailable_when_result_binds_to_no_window(self, tmp_path): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows()) - tampered = dict(BENCH_FIELDS, max_concurrency=8) - pkg.original_result.write_text(json.dumps(tampered, indent=2)) - pkg.bench_result.write_text(json.dumps(tampered, indent=2)) - assert pkg.run() == 0 - assert pkg.agg()["cpu_power_valid"] == 0 - assert "cpu_window_unavailable" in pkg.sidecar()["cpu"]["reason_codes"] - - -def test_invalid_revalidation_removes_stale_host_power(tmp_path): - pkg = build_package(tmp_path) - assert pkg.run() == 0 - assert len(pkg.agg()["workers"]) == 2 - assert pkg.run(sha="b" * 40) == 0 - assert pkg.agg()["power_valid"] == 0 - assert "workers" not in pkg.agg() - - def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): pkg = build_package(tmp_path) manifest_path = pkg.power_dir / "manifest.json" @@ -1141,27 +1024,6 @@ def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): ] -def test_host_power_uses_timestamp_window_when_reported_duration_is_skewed(tmp_path): - pkg = build_package(tmp_path, bench_extra={"duration": 60.1}) - window_path = pkg.windows_dir / f"{RESULT_STEM}.json" - window = json.loads(window_path.read_text()) - window["duration"] = 60.1 - window_path.write_text(json.dumps(window)) - - assert pkg.run() == 0 - agg = pkg.agg() - assert agg["power_valid"] == 1 - assert agg["avg_total_gpu_power_w"] == 1400.0 - assert agg["prefill_avg_power_w"] == 400.0 - assert agg["decode_avg_power_w"] == 300.0 - assert agg["workers"] == [ - {"role": "decode", "worker_idx": 0, "hosts": ["node-d"], - "num_gpus": 2, "avg_power_w": 300.0}, - {"role": "prefill", "worker_idx": 0, "hosts": ["node-p"], - "num_gpus": 2, "avg_power_w": 400.0}, - ] - - @pytest.mark.parametrize("temperature", ["nan", "inf", "bad", "-300", "9223372036854775794"]) def test_v3_samples_reject_invalid_temperature(tmp_path, temperature): path = tmp_path / "samples.csv" From 65c6d40b4cbbf96c2dab04c13d6655cfb6747b79 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 16:24:14 -0700 Subject: [PATCH 15/35] test: restore distinct CPU power regression coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 恢复 CPU 电源来源选择、错误隔离和时间窗口的回归测试。 --- .../power/test_aggregate_power_multinode.py | 104 +++++++++++++++--- 1 file changed, 89 insertions(+), 15 deletions(-) diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index ce87eae105..2a2d9be39b 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -782,8 +782,10 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) ] CPU_HEADER_V2 = CPU_HEADER_V1[:7] + ["cpu_rail_w", "soc_w", "dram_w", "total_power_w"] GRACE_W = {"node-d": 500.0, "node-p": 700.0} +MODULE_W = 1500.0 +DCGM_W = 300.0 # Component rails never feed a headline metric; the values are chosen so any -# leak into a headline key is visible against the Grace constants. +# leak into a headline key is visible against the Grace and module constants. RAILS_W = {"cpu_rail": 200.0, "soc": 50.0, "dram": 30.0} GRACE_KINDS = {"grace": None, **RAILS_W} @@ -793,6 +795,8 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) "cpu_rail": ("acpi", "CPU Power Socket {s}", "CPU{s}:cpuRailPowerUsageW"), "soc": ("acpi", "SysIO Power Socket {s}", "CPU{s}:socPowerUsageW"), "dram": ("acpi", "DRAM Power Socket {s}", "CPU{s}:dramPowerUsageW"), + "module": ("acpi", "Module Power Socket {s}", "Module Power Socket {s}"), + "dcgm": ("dcgm", "CPU{s}:cpuPowerUsageW", "CPU{s}:cpuPowerUsageW"), } @@ -801,7 +805,8 @@ def _cpu_row(ts, host, socket, kind, watts, fmt): sensor = (label_v1 if fmt == "v1" else label_v2).format(s=socket) if fmt == "v1": return [1, repr(ts), host, source, sensor, socket, repr(watts), ""] - return [2, repr(ts), host, source, sensor, socket, repr(watts), *map(repr, RAILS_W.values()), ""] + rails = ["", "", ""] if source == "dcgm" else [repr(w) for w in RAILS_W.values()] + return [2, repr(ts), host, source, sensor, socket, repr(watts), *rails, ""] def _cpu_rows(kinds_by_host=None, fmt="v1", *, hosts=CPU_HOSTS): @@ -917,7 +922,37 @@ def test_rail_gap_drops_only_that_rail(self, tmp_path): assert agg["avg_total_cpu_sysio_power_w"] == 200.0 @pytest.mark.parametrize( - "cpu, require_power", [("missing", True), ("complete", True), ("missing", False)] + "kinds, expected_kind, expected_power, module_power", + [ + ({host: {"dcgm": DCGM_W} for host in CPU_HOSTS}, "dcgm_cpu_rail", 1200.0, None), + ( + {host: {"grace": None, "module": MODULE_W} for host in CPU_HOSTS}, + "module", 2400.0, 6000.0, + ), + ( + {"node-d": {"grace": None, "module": MODULE_W}, "node-p": {"grace": None}}, + "grace_socket", 2400.0, None, + ), + ], + ) + def test_cpu_source_selection( + self, tmp_path, kinds, expected_kind, expected_power, module_power + ): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows(kinds)) + assert pkg.run() == 0 + agg = pkg.agg() + assert agg["cpu_power_valid"] == 1 + assert agg["avg_total_cpu_power_w"] == expected_power + assert pkg.sidecar()["cpu"]["sensor_kind"] == expected_kind + if module_power is None: + assert "avg_total_module_power_w" not in agg + else: + assert agg["avg_total_module_power_w"] == module_power + + @pytest.mark.parametrize( + "cpu, require_power", + [("missing", True), ("complete", True), ("missing", False), ("dcgm_only", True)], ) def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( self, tmp_path, cpu, require_power @@ -925,6 +960,8 @@ def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( pkg = build_package(tmp_path) if cpu == "complete": add_cpu_package(pkg, _cpu_rows()) + elif cpu == "dcgm_only": + add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) assert pkg.run(require_power=require_power, expected_cpu_source="acpi") == int( require_power and cpu != "complete" ) @@ -936,6 +973,8 @@ def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( else: assert set(apm.CPU_METRIC_KEYS).isdisjoint(pkg.agg()) assert pkg.sidecar()["cpu"]["reason_codes"] + if cpu == "dcgm_only": + assert "cpu_sensor_source_mismatch" in pkg.sidecar()["cpu"]["reason_codes"] def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): pkg = build_package(tmp_path) @@ -961,6 +1000,10 @@ def tampered(self, request, tmp_path): rows = self._drop(rows, "node-p", 1, lambda ts: True) elif kind == "gap": rows = self._drop(rows, "node-d", 0, lambda ts: 1020.0 <= ts <= 1024.0) + elif kind == "mixed": + rows = _cpu_rows({"node-d": {"grace": None}, "node-p": {"dcgm": DCGM_W}}) + elif kind == "malformed": + rows.append([1, repr(FIRST_TS), "node-d", "acpi", "Grace Power Socket 0", 0, "n/a", ""]) add_cpu_package(pkg, rows) return pkg @@ -969,6 +1012,8 @@ def tampered(self, request, tmp_path): [ ("socket", "cpu_socket_count_mismatch"), ("gap", "cpu_sample_gap_exceeded"), + ("mixed", "cpu_sensor_kind_mixed"), + ("malformed", "cpu_samples_malformed"), ], indirect=["tampered"], ) @@ -985,27 +1030,42 @@ def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, r assert reason in cpu["reason_codes"] assert pkg.sidecar()["power_valid"] is True - def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path): - """A GPU coverage failure withholds GPU energy but not cpu_power_valid.""" + @pytest.mark.parametrize("gpu_failure", ["coverage", "producer_pin"]) + def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path, gpu_failure): pkg = build_package(tmp_path) add_cpu_package(pkg, _cpu_rows()) - _rewrite_samples( - pkg, - lambda body: [ - row - for row in body - if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) - ], - ) - assert pkg.run() == 0 + if gpu_failure == "coverage": + _rewrite_samples( + pkg, + lambda body: [ + row for row in body + if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) + ], + ) + assert pkg.run(sha="b" * 40 if gpu_failure == "producer_pin" else PRODUCER_SHA) == 0 agg = pkg.agg() assert agg["power_valid"] == 0 assert "total_gpu_energy_j" not in agg assert_grace_keys(agg) sidecar = pkg.sidecar() - assert "package_recompute_invalid" in sidecar["reasons"] + gpu_reason = ( + "producer_commit_mismatch" if gpu_failure == "producer_pin" + else "package_recompute_invalid" + ) + assert gpu_reason in sidecar["reasons"] assert sidecar["cpu"]["reason_codes"] == [] + def test_result_without_window_invalidates_cpu(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _cpu_rows()) + result = dict(BENCH_FIELDS, max_concurrency=8) + pkg.original_result.write_text(json.dumps(result)) + pkg.bench_result.write_text(json.dumps(result)) + assert pkg.run() == 0 + assert pkg.agg()["cpu_power_valid"] == 0 + assert "cpu_window_unavailable" in pkg.sidecar()["cpu"]["reason_codes"] + + def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): pkg = build_package(tmp_path) manifest_path = pkg.power_dir / "manifest.json" @@ -1024,6 +1084,20 @@ def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): ] +def test_host_power_uses_timestamp_window_when_reported_duration_is_skewed(tmp_path): + pkg = build_package(tmp_path, bench_extra={"duration": 60.1}) + window_path = pkg.windows_dir / f"{RESULT_STEM}.json" + window = json.loads(window_path.read_text()) + window["duration"] = 60.1 + window_path.write_text(json.dumps(window)) + + assert pkg.run() == 0 + workers = pkg.agg()["workers"] + assert [(worker["role"], worker["avg_power_w"]) for worker in workers] == [ + ("decode", 300.0), ("prefill", 400.0) + ] + + @pytest.mark.parametrize("temperature", ["nan", "inf", "bad", "-300", "9223372036854775794"]) def test_v3_samples_reject_invalid_temperature(tmp_path, temperature): path = tmp_path / "samples.csv" From 4954627d0082324083b5d9247d21beb92b7b5f61 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 18:02:44 -0700 Subject: [PATCH 16/35] docs: define the Grace CPU power sensors and their published keys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：定义 Grace CPU 各功耗传感器的范围及对应发布字段。 --- inferencex-e2e/docs/results-and-ingestion.md | 23 +++++++++++++++---- .../docs/results-and-ingestion_zh.md | 20 ++++++++++++---- 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index 7fccb483ed..59a2e1af9f 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -456,11 +456,24 @@ GB200 and GB300 NVL72 recipes that enable srt-slurm's `telemetry.cpu_power_expor package. The multinode validator accepts the historical v2.2.1 long format (one row per sensor reading, header `schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`) and the current wide format (one row per scrape, host, and socket, with `cpu_rail_w`, `soc_w`, and -`dram_w` reference columns). It classifies each row by its `sensor` cell: `Module Power Socket N` -is the whole-module reading (Grace, both Blackwell GPUs, HBM, LPDDR5X, and regulator loss); -`Grace Power Socket N` or `CPU:cpuSidePowerUsageW` is the Grace-side socket total (CPU, SoC, -and LPDDR5X); `CPU:cpuPowerUsageW` is DCGM field 1130, the CPU rail only. Component rails -(`CPU Power Socket N`, `SysIO Power Socket N`, DRAM) never feed a published metric. +`dram_w` reference columns). It classifies each row by its `sensor` cell. The firmware meters +nest inside one another; none of them is a Bianca board or compute-tray figure, and the socket +total is not the sum of its component rails +([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): + +| Sensor (ACPI OEM label) | Contains | Does not contain | Published as | +| --- | --- | --- | --- | +| `Module Power Socket N` | The whole GB200/GB300 superchip module: Grace, its two Blackwell GPUs, HBM, LPDDR5X, and on-module regulator loss | Everything else on the tray (NICs, NVMe, fans, the tray converter) | `*_module_*` keys, only when every socket reports it | +| `Grace Power Socket N` (`CPU:cpuSidePowerUsageW` in the wide format) | The whole Grace socket: CPU rail, SoC rail, LPDDR5X, and regulator loss | The GPUs and HBM | The Grace-side `*_cpu_*` headline keys | +| `CPU Power Socket N` (`cpu_rail_w`) | The CPU rail only (cores) | SysIO, LPDDR5X, regulator loss | `*_cpu_rail_*` keys | +| `SysIO Power Socket N` (`soc_w`) | The SoC rail (system I/O) | The CPU rail and LPDDR5X | `*_cpu_sysio_*` keys | +| `DRAM Power Socket N` (`dram_w`) | The LPDDR5X rail where firmware exposes it | | Never; blank on the current NVL72 firmware | +| `CPU:cpuPowerUsageW` | DCGM field 1130, the CPU rail only | SysIO and LPDDR5X | The Grace-side keys only when ACPI is unavailable | + +On a GB300 tray the socket total reads about 95 W per socket while its CPU and SysIO rails sum +to about 51 W; the remainder is LPDDR5X and regulator loss, so rails are never added to or +subtracted from a total. The CI clusters' NVL72 firmware currently binds the Grace, CPU, and +SysIO meters only: no module meter was found on GB200, and the DRAM rail is blank. Per socket the headline series is chosen in the order module, Grace socket total, DCGM CPU rail, and one kind must be present for every socket. Every fed series is integrated over the same bound diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index 2379f2024d..794867241a 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -441,10 +441,22 @@ rm -rf -- "$tmp" 历史版本 v2.2.1 写出的长格式（每个传感器读数一行，表头为 `schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`），以及当前 版本的宽格式（每次采集、每台主机、每个 socket 一行，另带 `cpu_rail_w`、`soc_w`、`dram_w` 参考列）。 -它按 `sensor` 单元格对每行分类：`Module Power Socket N` 是整模块读数（Grace、两颗 Blackwell GPU、 -HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU:cpuSidePowerUsageW` 是 Grace 侧 socket -总功耗（CPU、SoC 和 LPDDR5X）；`CPU:cpuPowerUsageW` 是 DCGM 字段 1130，仅含 CPU 供电轨。组件供电轨 -（`CPU Power Socket N`、`SysIO Power Socket N`、DRAM）永远不会进入发布指标。 +它按 `sensor` 单元格对每行分类。这些固件功耗计层层嵌套，都不是 Bianca 板级或 compute tray 级的数字， +socket 总功耗也不等于各组件供电轨之和 +（[NVIDIA Grace 功耗指南](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)）： + +| 传感器（ACPI OEM 标签） | 包含 | 不包含 | 发布为 | +| --- | --- | --- | --- | +| `Module Power Socket N` | 整个 GB200/GB300 superchip 模块：Grace、所属的两颗 Blackwell GPU、HBM、LPDDR5X 以及模块内稳压损耗 | tray 上的其他部件（NIC、NVMe、风扇、tray 电源转换器） | `*_module_*` 字段，仅当每个 socket 都有该读数时输出 | +| `Grace Power Socket N`（宽格式中为 `CPU:cpuSidePowerUsageW`） | 整个 Grace socket：CPU 供电轨、SoC 供电轨、LPDDR5X 及稳压损耗 | GPU 与 HBM | Grace 侧 `*_cpu_*` 主字段 | +| `CPU Power Socket N`（`cpu_rail_w`） | 仅 CPU 供电轨（核心） | SysIO、LPDDR5X、稳压损耗 | `*_cpu_rail_*` 字段 | +| `SysIO Power Socket N`（`soc_w`） | SoC 供电轨（系统 I/O） | CPU 供电轨与 LPDDR5X | `*_cpu_sysio_*` 字段 | +| `DRAM Power Socket N`（`dram_w`） | 固件提供时的 LPDDR5X 供电轨 | | 不发布；当前 NVL72 固件下为空 | +| `CPU:cpuPowerUsageW` | DCGM 字段 1130，仅 CPU 供电轨 | SysIO 与 LPDDR5X | 仅在没有 ACPI 时作为 Grace 侧字段 | + +在 GB300 tray 上，socket 总功耗约为每 socket 95 W，而 CPU 与 SysIO 两条供电轨之和约为 51 W；差值是 +LPDDR5X 与稳压损耗，因此供电轨读数永远不会与总功耗相加或相减。CI 集群的 NVL72 固件目前只绑定 Grace、CPU +和 SysIO 三个功耗计：GB200 上没有发现模块功耗计，DRAM 供电轨为空。 每个 socket 的主序列按模块、Grace socket 总功耗、DCGM CPU 供电轨的顺序选取，且所有 socket 必须具备 同一种传感器类型。每条参与计算的序列都在与 GPU 能耗相同的正式测量窗口内积分，采用同样的梯形法、 From 25f465a12c9a240c1cb08a905c2ea2e1636eed0f Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 8 Oct 2026 18:21:30 -0700 Subject: [PATCH 17/35] docs: note that GB200 exposes module power through NVML, not ACPI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：说明 GB200 的模块功耗由 NVML 提供而非 ACPI 功耗计，srt-slurm 尚未读取。 --- inferencex-e2e/docs/results-and-ingestion.md | 6 +++++- inferencex-e2e/docs/results-and-ingestion_zh.md | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index 59a2e1af9f..8a9a6d234b 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -473,7 +473,11 @@ total is not the sum of its component rails On a GB300 tray the socket total reads about 95 W per socket while its CPU and SysIO rails sum to about 51 W; the remainder is LPDDR5X and regulator loss, so rails are never added to or subtracted from a total. The CI clusters' NVL72 firmware currently binds the Grace, CPU, and -SysIO meters only: no module meter was found on GB200, and the DRAM rail is blank. +SysIO meters only: the module ACPI meters on GB200 are unbound, and the DRAM rail is blank. The +module reading is available there through NVML instead (`nvidia-smi +--query-gpu=module.power.draw.average`), where both GPUs of one superchip report the same module +value, about 475 W at idle against roughly 95 W Grace plus two GPUs near 170 W each. srt-slurm's +CPU leg does not read NVML yet, so no package carries a module series today. Per socket the headline series is chosen in the order module, Grace socket total, DCGM CPU rail, and one kind must be present for every socket. Every fed series is integrated over the same bound diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index 794867241a..18deca6378 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -456,7 +456,10 @@ socket 总功耗也不等于各组件供电轨之和 在 GB300 tray 上，socket 总功耗约为每 socket 95 W，而 CPU 与 SysIO 两条供电轨之和约为 51 W；差值是 LPDDR5X 与稳压损耗，因此供电轨读数永远不会与总功耗相加或相减。CI 集群的 NVL72 固件目前只绑定 Grace、CPU -和 SysIO 三个功耗计：GB200 上没有发现模块功耗计，DRAM 供电轨为空。 +和 SysIO 三个功耗计：GB200 上的模块 ACPI 功耗计未绑定，DRAM 供电轨为空。模块读数在那里改由 NVML 提供 +（`nvidia-smi --query-gpu=module.power.draw.average`），同一颗 superchip 的两颗 GPU 报出相同的模块值，空闲时 +约 475 W，对应约 95 W 的 Grace 加两颗各约 170 W 的 GPU。srt-slurm 的 CPU 采集环节尚未读取 NVML，因此目前 +没有任何产物包带有模块序列。 每个 socket 的主序列按模块、Grace socket 总功耗、DCGM CPU 供电轨的顺序选取，且所有 socket 必须具备 同一种传感器类型。每条参与计算的序列都在与 GPU 能耗相同的正式测量窗口内积分，采用同样的梯形法、 From 6d87ea2c35b05f30235e692305f5e8c31bcd6161 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Fri, 9 Oct 2026 13:16:51 -0700 Subject: [PATCH 18/35] docs: correct NVL72 CPU power policy, rail keys, and changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：修正 NVL72 CPU 功耗失败策略说明，补充供电轨字段与实测数值，并删除 changelog 中不属于本 PR 的 Kimi K3 采样间隔描述。 --- .github/AGENT_OPERATIONS.md | 4 ++-- inferencex-e2e/configs/nvidia-master.yaml | 10 +++++----- inferencex-e2e/docs/results-and-ingestion.md | 16 ++++++++++++---- inferencex-e2e/docs/results-and-ingestion_zh.md | 13 +++++++++---- inferencex-e2e/infx/results/power/cpu_side.py | 12 ++++++------ inferencex-e2e/infx/results/power/multinode.py | 6 +++--- .../power/test_aggregate_power_multinode.py | 2 +- inferencex-e2e/perf-changelog.yaml | 2 +- 8 files changed, 39 insertions(+), 26 deletions(-) diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 036f2b261d..2275d32cb1 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -68,11 +68,11 @@ Eval selection, the `--no-evals` / `--evals-only` / `--all-evals` flags, and cha ## Power telemetry -Multinode srt-slurm results may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`. Single-node results carry no power fields until srt-slurm telemetry covers those lanes. +Multinode srt-slurm results may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power. Single-node results carry no power fields until srt-slurm telemetry covers those lanes. Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count. -NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, `total_cpu_energy_j`, and, when the module sensor is exposed on every socket, `avg_total_module_power_w` and `total_module_energy_j`. That verdict is independent of `power_valid`; strict `REQUIRE_POWER` qualification also requires any CPU source declared by the recipe; see [`inferencex-e2e/docs/results-and-ingestion.md`](../inferencex-e2e/docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). +NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j`. They add `avg_total_module_power_w` and `total_module_energy_j` when the module sensor is exposed on every socket. They add `avg_total_cpu_rail_power_w` and `total_cpu_rail_energy_j` (ACPI CPU rail), and `avg_total_cpu_sysio_power_w` and `total_cpu_sysio_energy_j` (ACPI SysIO rail), each pair only when every Grace socket reports that rail without gaps. The rails are a non-additive breakdown inside the Grace socket total: never add them to it or subtract them from it. The CPU verdict is independent of `power_valid`. An invalid CPU leg fails the job only when power is required and the recipe declares `telemetry.cpu_power_exporter.source`; AgentX lanes whose launcher validates power always require it, and other lanes require it through `REQUIRE_POWER=1`; see [`inferencex-e2e/docs/results-and-ingestion.md`](../inferencex-e2e/docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index cc7b28eebb..4fb66a3795 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -6555,7 +6555,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-variants.yaml:override_dep16_vllm_simple_offload" # Kimi-K3 GB200 TP16/DCP16 profiles using Mooncake DRAM offload. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: - # Referenced power recipes also collect best-effort Grace CPU telemetry. + # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6584,7 +6584,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: - # Referenced power recipes also collect best-effort Grace CPU telemetry. + # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6731,7 +6731,7 @@ dsv4-fp4-gb300-dynamo-trt-agentx: # Add GB200 TP8 x PP2/DCP8 AgentX operating points. The runtime topology uses # 16 GPUs total, allocated across four 4-GPU GB200 nodes. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: - # Referenced power recipes also collect best-effort Grace CPU telemetry. + # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -8145,7 +8145,7 @@ qwen3.5-fp8-gb200-dynamo-sglang-mtp: # Kimi-K3 GB300 points on the measured global p90-ITL Pareto front. Each # DCP8 worker spans two 4-GPU nodes. kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: - # Referenced power recipes also collect best-effort Grace CPU telemetry. + # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -8220,7 +8220,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: - # Referenced power recipes also collect best-effort Grace CPU telemetry. + # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index 8a9a6d234b..cf565f841a 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -93,7 +93,7 @@ The fixed-sequence transformer requires runner, framework, precision, speculativ | Latency and interactivity | Each benchmark input key ending in `ms` is converted from milliseconds to seconds with `_ms` removed. Keys containing `tpot` also produce an `intvty` reciprocal. | | Optional runtime metadata | `router` as exactly `{name, version}`, `kv_p2p_transfer`, and, for multinode results, measured power from the srt-slurm telemetry package | -Single-node GPU count is `tp * pp * pcp_size`. DCP does not multiply the physical GPU count. Multinode per-GPU denominators use the declared prefill and decode GPU counts. Invalid or missing required metadata fails transformation. Only multinode results are power-aggregated; single-node results carry no power fields or verdict. Power aggregation is best effort by default; `REQUIRE_POWER=1` fails the job after preserving available results and audits when power validation fails. +Single-node GPU count is `tp * pp * pcp_size`. DCP does not multiply the physical GPU count. Multinode per-GPU denominators use the declared prefill and decode GPU counts. Invalid or missing required metadata fails transformation. Only multinode results are power-aggregated; single-node results carry no power fields or verdict. Power aggregation is best effort by default; when power validation fails, `REQUIRE_POWER=1` or an AgentX lane whose launcher validates power fails the job after preserving available results and audits. InferenceX-app treats routing fields as columns or config dimensions and stores numeric measurements in `benchmark_results.metrics` JSONB. The mapper supports v1 shared topology, v2 split prefill/decode topology, and nested v3 AgentX metrics. Unknown numeric metrics are retained and warned about, which permits schema growth without silently losing numeric data. @@ -470,8 +470,8 @@ total is not the sum of its component rails | `DRAM Power Socket N` (`dram_w`) | The LPDDR5X rail where firmware exposes it | | Never; blank on the current NVL72 firmware | | `CPU:cpuPowerUsageW` | DCGM field 1130, the CPU rail only | SysIO and LPDDR5X | The Grace-side keys only when ACPI is unavailable | -On a GB300 tray the socket total reads about 95 W per socket while its CPU and SysIO rails sum -to about 51 W; the remainder is LPDDR5X and regulator loss, so rails are never added to or +On a GB300 tray each socket total reads about 97 W while its CPU and SysIO rails sum to about +55 W; the remaining ~42 W is LPDDR5X and regulator loss, so rails are never added to or subtracted from a total. The CI clusters' NVL72 firmware currently binds the Grace, CPU, and SysIO meters only: the module ACPI meters on GB200 are unbound, and the DRAM rail is blank. The module reading is available there through NVML instead (`nvidia-smi @@ -506,7 +506,15 @@ The CPU verdict is independent of `power_valid`: it borrows only the bound formal window and the worker-host topology from the GPU package, and no GPU verdict reaches it, so an unpinned producer or failed GPU coverage withholds GPU energy while `cpu_power_valid` still judges the CPU samples on their own. Any CPU-leg failure records `cpu_power_valid: 0` with no CPU -keys and leaves every GPU field unchanged. With `REQUIRE_POWER=1`, a recipe-declared CPU source must be valid; ACPI requires complete Grace socket or module totals, and missing CPU artifacts produce an independent invalid CPU audit. GPU-only recipes retain their existing behavior. Reason codes: +keys and leaves every GPU field unchanged. It fails the job only when power is required +and the recipe declares `telemetry.cpu_power_exporter.source`. AgentX lanes whose launcher +validates power (`agentx=True` in [`power.py`](../infx/launch/drivers/srt/power.py)), including +the Kimi-K3 GB200/GB300 recipes with DCGM and ACPI telemetry, always require power; other lanes +require it through `REQUIRE_POWER=1`, set by the `require-power` master-config field (as on the +Qwen3.5 GB200/GB300 8k1k sequences) or workflow input. A declared `acpi` source requires complete +Grace socket or module totals, and missing CPU artifacts produce an invalid CPU audit instead of no +verdict. +Without a declared source, the CPU verdict never changes the job's exit code. Reason codes: `cpu_artifacts_missing`, `cpu_sensor_source_mismatch`, `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch`, `cpu_sensor_kind_mixed`, diff --git a/inferencex-e2e/docs/results-and-ingestion_zh.md b/inferencex-e2e/docs/results-and-ingestion_zh.md index 18deca6378..304762ce04 100644 --- a/inferencex-e2e/docs/results-and-ingestion_zh.md +++ b/inferencex-e2e/docs/results-and-ingestion_zh.md @@ -93,7 +93,7 @@ shape: array of benchmark row objects | 延迟和交互性 | 基准输入中每个以 `ms` 结尾的键都会从毫秒换算为秒，并移除 `_ms`。包含 `tpot` 的键还会产生其倒数 `intvty`。 | | 可选运行时元数据 | 形式必须精确为 `{name, version}` 的 `router`、`kv_p2p_transfer`，以及多节点结果中由 srt-slurm 遥测包补入的实测功耗 | -单节点 GPU 数为 `tp * pp * pcp_size`。DCP 不会增加物理 GPU 数。多节点每 GPU 指标的分母使用声明的 prefill 和 decode GPU 数。无效或缺失的必需元数据会使转换失败。只有多节点结果执行功耗聚合；单节点结果不含功耗字段或结论。功耗聚合默认尽力而为；当设置 `REQUIRE_POWER=1` 时，功耗验证失败会在保留已有结果和审计后使任务失败。 +单节点 GPU 数为 `tp * pp * pcp_size`。DCP 不会增加物理 GPU 数。多节点每 GPU 指标的分母使用声明的 prefill 和 decode GPU 数。无效或缺失的必需元数据会使转换失败。只有多节点结果执行功耗聚合；单节点结果不含功耗字段或结论。功耗聚合默认尽力而为；当设置 `REQUIRE_POWER=1` 时，或在由 launcher 自行验证功耗的 AgentX 通道上，功耗验证失败会在保留已有结果和审计后使任务失败。 InferenceX-app 将路由字段作为列或配置维度，并把数值测量存入 `benchmark_results.metrics` JSONB。映射器支持共享拓扑的 v1、拆分 prefill/decode 拓扑的 v2，以及嵌套 AgentX 指标的 v3。未知数值指标会被保留并产生警告，因此架构可以扩展，同时不会无提示地丢失数值数据。 @@ -454,7 +454,7 @@ socket 总功耗也不等于各组件供电轨之和 | `DRAM Power Socket N`（`dram_w`） | 固件提供时的 LPDDR5X 供电轨 | | 不发布；当前 NVL72 固件下为空 | | `CPU:cpuPowerUsageW` | DCGM 字段 1130，仅 CPU 供电轨 | SysIO 与 LPDDR5X | 仅在没有 ACPI 时作为 Grace 侧字段 | -在 GB300 tray 上，socket 总功耗约为每 socket 95 W，而 CPU 与 SysIO 两条供电轨之和约为 51 W；差值是 +在 GB300 tray 上，每个 socket 总功耗约为 97 W，而 CPU 与 SysIO 两条供电轨之和约为 55 W；剩余约 42 W 是 LPDDR5X 与稳压损耗，因此供电轨读数永远不会与总功耗相加或相减。CI 集群的 NVL72 固件目前只绑定 Grace、CPU 和 SysIO 三个功耗计：GB200 上的模块 ACPI 功耗计未绑定，DRAM 供电轨为空。模块读数在那里改由 NVML 提供 （`nvidia-smi --query-gpu=module.power.draw.average`），同一颗 superchip 的两颗 GPU 报出相同的模块值，空闲时 @@ -481,10 +481,15 @@ LPDDR5X 与稳压损耗，因此供电轨读数永远不会与总功耗相加或 记录 `sensor_kind`（`module`、`grace_socket` 或 `dcgm_cpu_rail`）、`source`（`acpi` 或 `dcgm`）、 预期与观测 socket 数、解析行数和原因码。`power_metric_schema_version` 保持为 `2`。 -该测量环节尽力而为，其结论与 `power_valid` 相互独立：它只从 GPU 产物包借用已绑定的正式测量窗口和 +CPU 侧结论与 `power_valid` 相互独立：它只从 GPU 产物包借用已绑定的正式测量窗口和 worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 producer 固定版本校验失败或 GPU 覆盖不足只会 使 GPU 能耗不予发布，`cpu_power_valid` 仍按 CPU 采样自身给出结论。CPU 侧的任何失败都会记录 -`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持不变。设置 `REQUIRE_POWER=1` 时，配方声明的 CPU 来源必须有效；ACPI 必须提供完整的 Grace socket 或模块总功耗，缺失 CPU 产物会生成独立的无效 CPU 审计。仅采集 GPU 的配方保留原有行为。 +`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持不变。只有在强制功耗验证且配方声明了 `telemetry.cpu_power_exporter.source` 时， +CPU 侧失败才会使任务失败。由 launcher 自行验证功耗的 AgentX 通道（[`power.py`](../infx/launch/drivers/srt/power.py) +中 `agentx=True`，包括启用 DCGM 与 ACPI 遥测的 Kimi-K3 GB200/GB300 配方）始终强制功耗验证；其他通道通过 +`REQUIRE_POWER=1` 强制功耗验证，该值来自主配置的 `require-power` 字段（如 Qwen3.5 GB200/GB300 8k1k 序列） +或工作流输入。声明 `acpi` 来源时，必须提供完整的 Grace socket 或模块总功耗；缺失 CPU 产物会生成无效的 CPU 审计，而不是不给出结论。未声明来源时， +CPU 侧结论不会改变任务的退出码。 原因码包括 `cpu_artifacts_missing`、`cpu_sensor_source_mismatch`、`cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、 `cpu_manifest_invalid`、`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、 `cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口 diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index bcf84ecfaf..87cbe107a2 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -1,4 +1,4 @@ -"""Best-effort Grace CPU-side power leg of the multinode srt-slurm package. +"""Grace CPU-side power leg of the multinode srt-slurm package. srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` and a non-authoritative ``power/cpu/cpu_manifest.json`` beside the GPU DCGM package @@ -11,11 +11,11 @@ bound formal window with the shared trapezoid and boundary interpolation and published as additive metrics next to the GPU-board numbers. The v2 ACPI CPU-only and SysIO rails are integrated the same way when every socket reports -them; a rail with gaps drops only its own metrics. The leg is -best-effort by contract: any failure records ``cpu_power_valid=0`` with -reason codes and leaves every GPU field untouched. Strict qualification also -requires a recipe-declared CPU source; GPU-only callers keep best-effort CPU -behavior. A package without ``cpu/`` emits nothing unless a source is expected. +them; a rail with gaps drops only its own metrics. Any failure records +``cpu_power_valid=0`` with reason codes and leaves every GPU field untouched. +This module only judges the leg: the caller fails the job only when power is +required and the recipe declares a CPU source (``expected_source``). A package +without ``cpu/`` emits nothing unless a source is expected. """ from __future__ import annotations diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index f79d8a9d7c..47950b82b0 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -31,9 +31,9 @@ Grace CPU-side leg: when the package also carries ``power/cpu/`` (see :mod:`.cpu_side`), each socket's Grace-side (and, when exposed, whole-module) power is integrated over the same bound formal window and published under -``cpu_power_valid``. That verdict is independent of ``power_valid``. Strict -power qualification fails when a recipe-declared CPU source is missing or invalid; -GPU-only callers retain best-effort CPU behavior. +``cpu_power_valid``. That verdict is independent of ``power_valid``. A missing +or invalid CPU leg fails the run only with ``require_power`` and an +``expected_cpu_source``; otherwise it never changes the exit code. """ from __future__ import annotations diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index 2a2d9be39b..913d99c81c 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -1019,7 +1019,7 @@ def tampered(self, request, tmp_path): ) def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, reason): pkg = tampered - # REQUIRE_POWER guards the GPU leg only; a broken CPU leg never fails the run. + # Without an expected CPU source, REQUIRE_POWER gates only the GPU leg. assert pkg.run(require_power=True) == 0 agg = pkg.agg() assert agg["cpu_power_valid"] == 0 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 38104f2334..89f731a4e6 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9503,5 +9503,5 @@ description: - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.47.2, matching the srt-slurm v2.47.2 source pin. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict, including the ACPI Grace socket total and its CPU-only and SysIO rails. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." - - "Use collect_interval_ms: 1000 in the four Kimi K3 GB300 recipes to preserve their one-second GPU sampling cadence under the pinned srt-slurm 2 schema. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." + - "Enable required DCGM telemetry with collect_interval_ms: 1000 on the Qwen3.5 GB200/GB300 8P1D recipes, and set require-power on both Qwen3.5 GB200/GB300 8k1k sequences. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From 1e1aa3c611355b87ee10c499663f52f293ef70e3 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Tue, 13 Oct 2026 15:03:02 -0700 Subject: [PATCH 19/35] chore: pin cpu-power-exporter v2.51.1 to match the srt-slurm source pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：将 cpu-power-exporter 固定到 v2.51.1，与 srt-slurm 源码 pin 保持一致 --- inferencex-e2e/configs/nvidia-master.yaml | 38 ++++++++++---------- inferencex-e2e/docs/results-and-ingestion.md | 4 +-- inferencex-e2e/perf-changelog.yaml | 2 +- 3 files changed, 22 insertions(+), 22 deletions(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 9ca4ebdbd6..45bef64041 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -4103,7 +4103,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 4 @@ -4119,7 +4119,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -4135,7 +4135,7 @@ qwen3.5-fp8-gb200-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -6581,7 +6581,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. @@ -6609,7 +6609,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" dsv4-fp4-gb300-dynamo-trt-agentx: image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24 @@ -6758,7 +6758,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c16.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } @@ -6773,7 +6773,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c32.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [48] @@ -6787,7 +6787,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c48.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [72] @@ -6801,7 +6801,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c72.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [96] @@ -6815,7 +6815,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-mooncake-c96.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" dsv4-fp4-gb200-dynamo-sglang-agentic-agg: image: lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 model: deepseek-ai/DeepSeek-V4-Pro-0813 @@ -7094,7 +7094,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 4 @@ -7110,7 +7110,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -7126,7 +7126,7 @@ qwen3.5-fp8-gb300-dynamo-sglang: dp-attn: true additional-settings: - "CONFIG_FILE=recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -8316,7 +8316,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p1d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 1, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8330,7 +8330,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p2d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 2, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8344,7 +8344,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark7-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8358,7 +8358,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/disagg-1p3d-dcp8-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: @@ -8388,7 +8388,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark7-maxseq2-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } @@ -8402,7 +8402,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/kimik3/vllm/gb300-fp4/agentx/agg-dcp8-dspark4-mooncake.yaml" - - "CPU_POWER_EXPORTER_RELEASE=v2.47.2" + - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" # Kimi-K3 MXFP4 B200 aggregated vLLM (TP8 x PP2, 2 nodes / 16 GPUs), agentic # coding. The native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does diff --git a/inferencex-e2e/docs/results-and-ingestion.md b/inferencex-e2e/docs/results-and-ingestion.md index 95c9c64363..482984cbc5 100644 --- a/inferencex-e2e/docs/results-and-ingestion.md +++ b/inferencex-e2e/docs/results-and-ingestion.md @@ -516,11 +516,11 @@ Without a declared source, the CPU verdict never changes the job's exit code. Re window binds to the result, or the window's own contract checks failed). An overflowed CPU integration reuses the GPU leg's `non_finite_power_metric`, and `aggregate_result_missing` or `aggregate_result_unwritable` appears in both audits when the aggregate itself cannot be patched. -The srt-slurm source pin (`9d65c0ae`, v2.47.2) writes the wide format and does not classify the +The srt-slurm source pin (`545233a0`, v2.51.1) writes the wide format and does not classify the Module label, so these packages yield the Grace socket total plus its CPU-only and SysIO rails; module keys require a producer that classifies that label. A rail gap drops only that rail's keys and never changes `cpu_power_valid`. The selected Qwen3.5 and Kimi-K3 CPU-telemetry recipes pass -`CPU_POWER_EXPORTER_RELEASE=v2.47.2` from their master-config `additional-settings` to +`CPU_POWER_EXPORTER_RELEASE=v2.51.1` from their master-config `additional-settings` to `make setup`. Qwen3.5 8P1D keeps its frontend, benchmark client, and infra on the reserved head so their timestamps share the collector clock; its 13-node allocation still contains 12 GPU-worker nodes. This release includes legacy ACPI hwmon discovery. Setup verifies the downloaded asset checksum, replaces a cached binary with a different release marker, and fails on a pinned download error. The pinned srt-slurm submit preflight (NVIDIA/srt-slurm#553) rejects a missing, diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index db6ea59434..fbdd2fcef3 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9683,7 +9683,7 @@ - fixed-seq-len - agentic-coding description: - - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.47.2, matching the srt-slurm v2.47.2 source pin. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." + - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.51.1, matching the srt-slurm v2.51.1 source pin. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict, including the ACPI Grace socket total and its CPU-only and SysIO rails. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." - "Enable required DCGM telemetry with collect_interval_ms: 1000 on the Qwen3.5 GB200/GB300 8P1D recipes, and set require-power on both Qwen3.5 GB200/GB300 8k1k sequences. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From 94f01bb32943344595025643a9ef7a5abb91e879 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Tue, 13 Oct 2026 23:08:12 -0700 Subject: [PATCH 20/35] fix: drop launcher-bound keys from NVL72 CPU power recipes --- .../agg-dcp16-dspark4-maxseq2-mooncake.yaml | 3 +-- .../agentx/agg-dcp16-nospec-mooncake.yaml | 3 +-- .../gb200-fp4/agentx/agg-tp8pp2-variants.yaml | 2 +- .../vllm/gb300-fp4/agentx/agg-variants.yaml | 2 +- .../gb300-fp4/agentx/disagg-variants.yaml | 2 +- .../8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 19 ----------------- .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 19 ----------------- .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 16 -------------- .../8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 21 ------------------- .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 19 ----------------- .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 17 --------------- 11 files changed, 5 insertions(+), 118 deletions(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml index a8e45010ec..bd97e71a69 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml @@ -135,11 +135,10 @@ srun_options: container-remap-root: "" telemetry: - collector_join_timeout_seconds: 12 - cpu_power_exporter: port: 9405 source: acpi + collector_join_timeout_seconds: 12 benchmark: placement: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml index 077bb7d9fb..ec557a3342 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml @@ -131,11 +131,10 @@ srun_options: container-remap-root: "" telemetry: - collector_join_timeout_seconds: 12 - cpu_power_exporter: port: 9405 source: acpi + collector_join_timeout_seconds: 12 benchmark: placement: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml index bbba8b16b7..8462044387 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml @@ -104,10 +104,10 @@ base: container-remap-root: "" telemetry: - collector_join_timeout_seconds: 12 cpu_power_exporter: port: 9405 source: acpi + collector_join_timeout_seconds: 12 benchmark: placement: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml index bb46e3be64..7798f9288f 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml @@ -119,10 +119,10 @@ base: container-remap-root: "" telemetry: - collector_join_timeout_seconds: 12 cpu_power_exporter: port: 9405 source: acpi + collector_join_timeout_seconds: 12 benchmark: env: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml index d970c00c7d..7f49f3a435 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml @@ -163,10 +163,10 @@ base: container-remap-root: "" telemetry: - collector_join_timeout_seconds: 12 cpu_power_exporter: port: 9405 source: acpi + collector_join_timeout_seconds: 12 benchmark: placement: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index 4305d00c8b..fc0b7ccdca 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -109,26 +109,7 @@ roles: stream-interval: 50 disaggregation-mode: "decode" -benchmark: - type: custom - concurrencies: "1x2x4x8x16x32x64x128" - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' - telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - port: 9401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index abeebafcb7..efd2482639 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -145,26 +145,7 @@ roles: decode-log-interval: 1 stream-interval: 50 -benchmark: - type: custom - concurrencies: "1024" - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' - telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - port: 9401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index e9e43126e4..f757410237 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -157,26 +157,10 @@ roles: stream-interval: 50 benchmark: - type: custom - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh placement: node: head - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - port: 9401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index 1f02dbe35a..42568449f9 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -109,28 +109,7 @@ roles: stream-interval: 50 disaggregation-mode: "decode" -benchmark: - type: custom - concurrencies: "1x2x4x8x16x32x64x128" - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' - telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - # 9401 is already bound by the cluster-level exporter on im-gb300 nodes; - # use a port outside that range. - port: 19401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index 81d0f95e23..8847789faf 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -144,26 +144,7 @@ roles: decode-log-interval: 1 stream-interval: 50 -benchmark: - type: custom - concurrencies: "1024" - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' - telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - port: 19401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index 8941b29cf5..f5371b71d7 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -156,27 +156,10 @@ roles: stream-interval: 50 benchmark: - type: custom - command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh placement: node: head - env: - TOKENIZER: /model - ISL: '8192' - OSL: '1024' telemetry: - enabled: true - collect_interval_ms: 1000 - storage_subdir: power - required: true - startup_timeout_seconds: 120 - request_timeout_seconds: 2 - dcgm_exporter: - container_image: dcgm-exporter - # The cluster exporter uses 9401; use a separate port. - port: 19401 - command: "dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv" cpu_power_exporter: port: 9405 source: acpi From 85dc25170a8f30a23be7d1ee5b49ccf1c0fab3bf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:17:17 -0700 Subject: [PATCH 21/35] feat(power): declare the Grace CPU exporter per cluster and select it with cpu-power MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The binder merges telemetry.cpu_power_exporter from the cluster record when a master row sets cpu-power beside power, so recipes and per-point settings no longer carry it; make setup pins the exporter release only for those jobs. 中文：Grace CPU 功耗 exporter 改由 cluster 记录声明，master 行用 cpu-power 选择；recipe 与逐点 additional-settings 不再携带，make setup 仅对这些任务固定 exporter 版本。 --- .../workflows/benchmark-multinode-tmpl.yml | 1 + .../agg-dcp16-dspark4-maxseq2-mooncake.yaml | 3 - .../agentx/agg-dcp16-nospec-mooncake.yaml | 3 - .../gb200-fp4/agentx/agg-tp8pp2-variants.yaml | 3 - .../vllm/gb300-fp4/agentx/agg-variants.yaml | 3 - .../gb300-fp4/agentx/disagg-variants.yaml | 3 - .../8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 5 -- .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 5 -- .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 5 -- .../8k1k/disagg-1p1d-tp4-tp4-stp.yaml | 5 -- .../8k1k/disagg-4p1d-dep4-dep16-stp.yaml | 5 -- .../8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 5 -- inferencex-e2e/configs/nvidia-master.yaml | 62 ++++++------------- inferencex-e2e/configs/runners.yaml | 2 + inferencex-e2e/infx/clusters/slurm.py | 10 +++ .../infx/launch/drivers/srt/__init__.py | 8 ++- .../infx/launch/drivers/srt/checkout.py | 12 +++- .../infx/launch/drivers/srt/config.py | 32 +++++++--- .../infx/launch/drivers/srt/power.py | 10 ++- .../infx/launch/drivers/srt/submit.py | 6 +- inferencex-e2e/infx/launch/request.py | 1 + inferencex-e2e/infx/matrix/fingerprint.py | 1 + inferencex-e2e/infx/matrix/generate.py | 4 ++ inferencex-e2e/infx/matrix/validation.py | 17 +++++ inferencex-e2e/infx/srt_slurm/generate.py | 24 ++++--- inferencex-e2e/infx/srt_slurm/workload.py | 23 ++++++- .../infx/tests/launch/test_launch_policy.py | 13 ++++ .../infx/tests/launch/test_srt_config.py | 28 ++++++++- .../infx/tests/launch/test_srt_driver.py | 12 +++- .../matrix/test_generate_sweep_configs.py | 16 +++++ .../infx/tests/matrix/test_validation.py | 15 +++++ .../infx/tests/srt_slurm/test_generate.py | 28 +++++++++ .../infx/tests/srt_slurm/test_workload.py | 6 ++ 33 files changed, 255 insertions(+), 121 deletions(-) diff --git a/.github/workflows/benchmark-multinode-tmpl.yml b/.github/workflows/benchmark-multinode-tmpl.yml index e147a12354..190098325e 100644 --- a/.github/workflows/benchmark-multinode-tmpl.yml +++ b/.github/workflows/benchmark-multinode-tmpl.yml @@ -146,6 +146,7 @@ env: EVAL_CONC: ${{ inputs.eval-conc }} EVAL_LIMIT: ${{ inputs.eval-limit }} POWER: ${{ fromJSON(inputs.config).power && '1' || '0' }} + CPU_POWER: ${{ fromJSON(inputs.config)['cpu-power'] && '1' || '0' }} REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }} POWER_PRODUCER_SHA: ${{ inputs.power-producer-sha }} # Agentic-coding env. Fixed-seq-len jobs leave these empty. diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml index bd97e71a69..d300a0d8d8 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-dspark4-maxseq2-mooncake.yaml @@ -135,9 +135,6 @@ srun_options: container-remap-root: "" telemetry: - cpu_power_exporter: - port: 9405 - source: acpi collector_join_timeout_seconds: 12 benchmark: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml index ec557a3342..4f48b45148 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-dcp16-nospec-mooncake.yaml @@ -131,9 +131,6 @@ srun_options: container-remap-root: "" telemetry: - cpu_power_exporter: - port: 9405 - source: acpi collector_join_timeout_seconds: 12 benchmark: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml index 8462044387..a300c96cf3 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml @@ -104,9 +104,6 @@ base: container-remap-root: "" telemetry: - cpu_power_exporter: - port: 9405 - source: acpi collector_join_timeout_seconds: 12 benchmark: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml index 7798f9288f..54750368d1 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/agg-variants.yaml @@ -119,9 +119,6 @@ base: container-remap-root: "" telemetry: - cpu_power_exporter: - port: 9405 - source: acpi collector_join_timeout_seconds: 12 benchmark: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml index 7f49f3a435..06722b41a8 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb300-fp4/agentx/disagg-variants.yaml @@ -163,9 +163,6 @@ base: container-remap-root: "" telemetry: - cpu_power_exporter: - port: 9405 - source: acpi collector_join_timeout_seconds: 12 benchmark: diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index fc0b7ccdca..1a9ae0ecff 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -108,8 +108,3 @@ roles: decode-log-interval: 1 stream-interval: 50 disaggregation-mode: "decode" - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index efd2482639..24cc0ccb49 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -144,8 +144,3 @@ roles: decode-log-interval: 1 stream-interval: 50 - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index f757410237..08759d1a7f 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -159,8 +159,3 @@ roles: benchmark: placement: node: head - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml index 42568449f9..3726af8901 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml @@ -108,8 +108,3 @@ roles: decode-log-interval: 1 stream-interval: 50 disaggregation-mode: "decode" - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml index 8847789faf..b56132b6d5 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-4p1d-dep4-dep16-stp.yaml @@ -143,8 +143,3 @@ roles: decode-log-interval: 1 stream-interval: 50 - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index f5371b71d7..1e8092407b 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -158,8 +158,3 @@ roles: benchmark: placement: node: head - -telemetry: - cpu_power_exporter: - port: 9405 - source: acpi diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 3b1f60cee2..3b75f94bfe 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -3976,13 +3976,12 @@ qwen3.5-fp8-gb200-dynamo-sglang: conc-list: [1, 2, 4, 8, 16, 32, 64, 128] srt-recipe: disagg-1p1d-tp4-tp4-stp.yaml power: true + cpu-power: true prefill: num-worker: 1 tp: 4 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 4 @@ -3993,13 +3992,12 @@ qwen3.5-fp8-gb200-dynamo-sglang: conc-list: [1024] srt-recipe: disagg-4p1d-dep4-dep16-stp.yaml power: true + cpu-power: true prefill: num-worker: 4 tp: 4 ep: 4 dp-attn: true - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -4010,13 +4008,12 @@ qwen3.5-fp8-gb200-dynamo-sglang: conc-list: [2048, 4096] srt-recipe: disagg-8p1d-dep4-dep16-stp.yaml power: true + cpu-power: true prefill: num-worker: 8 tp: 4 ep: 4 dp-attn: true - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -6373,7 +6370,6 @@ kimik3-fp4-gb200-dynamo-vllm-agentic: dp-attn: true # Kimi-K3 GB200 TP16/DCP16 profiles using Mooncake DRAM offload. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: - # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6394,17 +6390,15 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg: num-nodes: 4 srt-recipe: agg-dcp16-dspark4-maxseq2-mooncake.yaml power: true + cpu-power: true worker: num-worker: 1 tp: 16 dcp-size: 16 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: - # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-75c2eef model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6424,14 +6418,13 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg: num-nodes: 4 srt-recipe: agg-dcp16-nospec-mooncake.yaml power: true + cpu-power: true worker: num-worker: 1 tp: 16 dcp-size: 16 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" dsv4-fp4-gb300-dynamo-trt-agentx: image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24 @@ -6542,7 +6535,6 @@ dsv4-fp4-gb300-dynamo-trt-agentx: # Add GB200 TP8 x PP2/DCP8 AgentX operating points. The runtime topology uses # 16 GPUs total, allocated across four 4-GPU GB200 nodes. kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: - # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -6563,6 +6555,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: num-nodes: 4 srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c16 power: true + cpu-power: true worker: num-worker: 1 tp: 8 @@ -6570,8 +6563,6 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } @@ -6579,6 +6570,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: num-nodes: 4 srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c32 power: true + cpu-power: true worker: num-worker: 1 tp: 8 @@ -6586,14 +6578,13 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [48] num-nodes: 4 srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c48 power: true + cpu-power: true worker: num-worker: 1 tp: 8 @@ -6601,14 +6592,13 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [72] num-nodes: 4 srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c72 power: true + cpu-power: true worker: num-worker: 1 tp: 8 @@ -6616,14 +6606,13 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.11.post1" } conc-list: [96] num-nodes: 4 srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c96 power: true + cpu-power: true worker: num-worker: 1 tp: 8 @@ -6631,8 +6620,6 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2: dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" dsv4-fp4-gb200-dynamo-sglang-agentic-agg: image: lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 model: deepseek-ai/DeepSeek-V4-Pro-0813 @@ -6898,13 +6885,12 @@ qwen3.5-fp8-gb300-dynamo-sglang: conc-list: [1, 2, 4, 8, 16, 32, 64, 128] srt-recipe: disagg-1p1d-tp4-tp4-stp.yaml power: true + cpu-power: true prefill: num-worker: 1 tp: 4 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 4 @@ -6915,13 +6901,12 @@ qwen3.5-fp8-gb300-dynamo-sglang: conc-list: [1024] srt-recipe: disagg-4p1d-dep4-dep16-stp.yaml power: true + cpu-power: true prefill: num-worker: 4 tp: 4 ep: 4 dp-attn: true - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -6932,13 +6917,12 @@ qwen3.5-fp8-gb300-dynamo-sglang: conc-list: [2048, 4096] srt-recipe: disagg-8p1d-dep4-dep16-stp.yaml power: true + cpu-power: true prefill: num-worker: 8 tp: 4 ep: 4 dp-attn: true - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: num-worker: 1 tp: 16 @@ -8066,7 +8050,6 @@ qwen3.5-fp8-gb200-dynamo-sglang-mtp: # Kimi-K3 GB300 points on the measured global p90-ITL Pareto front. Each # DCP8 worker spans two 4-GPU nodes. kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: - # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -8090,14 +8073,13 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: conc-list: [48, 56] srt-recipe: disagg-variants.yaml:override_1p1d_dcp8_dcp8_dspark4 power: true + cpu-power: true prefill: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 1, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8105,14 +8087,13 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: conc-list: [48, 64] srt-recipe: disagg-variants.yaml:override_1p2d_dcp8_dcp8_dspark4 power: true + cpu-power: true prefill: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 2, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8120,14 +8101,13 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: conc-list: [1] srt-recipe: disagg-variants.yaml:override_1p3d_dcp8_dcp8_dspark7 power: true + cpu-power: true prefill: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } - spec-decoding: mtp kv-offloading: dram @@ -8135,18 +8115,16 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: conc-list: [48] srt-recipe: disagg-variants.yaml:override_1p3d_dcp8_dcp8_dspark4 power: true + cpu-power: true prefill: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: - # Recipes declare ACPI Grace CPU power; strict AgentX power fails the job on an invalid CPU leg. image: vllm/vllm-openai:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -8167,14 +8145,13 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: num-nodes: 2 srt-recipe: agg-variants.yaml:override_dcp8_dspark7_maxseq2 power: true + cpu-power: true worker: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" - spec-decoding: mtp kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } @@ -8182,14 +8159,13 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg: num-nodes: 2 srt-recipe: agg-variants.yaml:override_dcp8_dspark4 power: true + cpu-power: true worker: num-worker: 1 tp: 8 dcp-size: 8 ep: 1 dp-attn: false - additional-settings: - - "CPU_POWER_EXPORTER_RELEASE=v2.51.1" # Kimi-K3 MXFP4 B200 aggregated vLLM (TP8 x PP2, 2 nodes / 16 GPUs), agentic # coding. The native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does diff --git a/inferencex-e2e/configs/runners.yaml b/inferencex-e2e/configs/runners.yaml index a6f618953a..c3719cc1ab 100644 --- a/inferencex-e2e/configs/runners.yaml +++ b/inferencex-e2e/configs/runners.yaml @@ -744,6 +744,7 @@ clusters: aiperf-cache: /aiperf_mmap_cache hf-hub-cache: /hf_hub_cache power-exporter-port: 9401 + cpu-power-exporter: {source: acpi} gb300-nv: gpus-per-node: 4 available-cpu-dram-mib: 860_160 @@ -802,6 +803,7 @@ clusters: hf-hub-cache: /hf_hub_cache dynamo-wheels: /configs/dynamo-wheels power-exporter-port: 19401 + cpu-power-exporter: {source: acpi} mi300x-amd: gpus-per-node: 8 available-cpu-dram-mib: 1_547_820 diff --git a/inferencex-e2e/infx/clusters/slurm.py b/inferencex-e2e/infx/clusters/slurm.py index c7c670f00c..342bbe9ff2 100644 --- a/inferencex-e2e/infx/clusters/slurm.py +++ b/inferencex-e2e/infx/clusters/slurm.py @@ -173,6 +173,15 @@ def rendered(self) -> dict[str, str | None]: return self.model_dump(by_alias=True) +class CpuPowerExporter(Record): + """The Grace CPU power exporter srt-slurm scrapes on this cluster's compute trays. + + A point that selects ``cpu-power`` binds ``telemetry.cpu_power_exporter`` with this source. + """ + + source: Literal["acpi", "dcgm"] + + class SrtSlurmSettings(Record): """Cluster-owned srtslurm.yaml facts; job-specific values are added by the driver.""" @@ -203,6 +212,7 @@ class SrtSlurmSettings(Record): mounts: dict[str, str] = Field(default_factory=dict) workspace_mounts: dict[str, str] = Field(default_factory=dict, alias="workspace-mounts") power_exporter_port: int | None = Field(default=None, alias="power-exporter-port", gt=0) + cpu_power_exporter: CpuPowerExporter | None = Field(default=None, alias="cpu-power-exporter") extra: dict[str, Any] = Field(default_factory=dict) @field_validator("workspace_mounts") diff --git a/inferencex-e2e/infx/launch/drivers/srt/__init__.py b/inferencex-e2e/infx/launch/drivers/srt/__init__.py index b366ecc2ef..0b94c18513 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/__init__.py +++ b/inferencex-e2e/infx/launch/drivers/srt/__init__.py @@ -163,14 +163,16 @@ def run_multinode(launch: Launch) -> int: ) install_srtctl(run, checkout, python=system_python) config.write_lane_config(run, lane, checkout, decision, model_paths) - if rc := run_setup(run, checkout): + if rc := run_setup(run, checkout, cpu_power_exporter=decision.cpu): return rc infmax = compute_workspace(run, checkout, shared=shared) run.env["INFMAX_WORKSPACE"] = str(infmax) recipe = str(checkout.root / BOUND_RECIPE) - power_port, client_env = config.binder_inputs(run.cluster, run.srt, lane, request, decision) - if rc := submit.bind_recipe(run, checkout, staged, recipe, power_port, client_env): + power_port, cpu_source, client_env = config.binder_inputs( + run.cluster, run.srt, lane, request, decision + ) + if rc := submit.bind_recipe(run, checkout, staged, recipe, power_port, cpu_source, client_env): return rc decision = power.with_cpu_source(decision, Path(recipe)) if github_env := request.env.get("GITHUB_ENV"): diff --git a/inferencex-e2e/infx/launch/drivers/srt/checkout.py b/inferencex-e2e/infx/launch/drivers/srt/checkout.py index 822ef43e3b..f2c81c3f46 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/checkout.py +++ b/inferencex-e2e/infx/launch/drivers/srt/checkout.py @@ -18,6 +18,9 @@ from infx.launch.drivers.srt.run import SrtRun, require SUBMODULE = Path("utils/srt-slurm") +# The cpu-power-exporter release asset ``make setup`` installs for a job that scrapes Grace CPU +# power; it ships with the srt-slurm release the submodule pins, so bump both together. +CPU_POWER_EXPORTER_RELEASE = "v2.51.1" PATCHES = Path("runners/srt-slurm/patches") SETUP_LOG = "srt-setup.log" SETUP_ATTEMPTS = 5 @@ -173,12 +176,17 @@ def _discard_corrupt_archives(configs: Path) -> bool: return discarded -def run_setup(run: SrtRun, checkout: Checkout) -> int: - """Run ``make setup`` in the checkout, logging to SETUP_LOG; return its exit code.""" +def run_setup(run: SrtRun, checkout: Checkout, *, cpu_power_exporter: bool = False) -> int: + """Run ``make setup`` in the checkout, logging to SETUP_LOG; return its exit code. + + ``cpu_power_exporter`` pins the cpu-power-exporter release, which makes its download fatal. + """ log = run.workspace / SETUP_LOG for attempt in range(1, SETUP_ATTEMPTS + 1): print(f"Setting up srt-slurm, attempt {attempt} (details: {SETUP_LOG})", flush=True) argv = ["make", "setup", f"ARCH={run.cluster.arch}"] + if cpu_power_exporter: + argv.append(f"CPU_POWER_EXPORTER_RELEASE={CPU_POWER_EXPORTER_RELEASE}") rc = _run_logged(argv, log, env=run.env, cwd=checkout.root) if rc == 0: print("srt-slurm setup complete", flush=True) diff --git a/inferencex-e2e/infx/launch/drivers/srt/config.py b/inferencex-e2e/infx/launch/drivers/srt/config.py index 849acb4131..837fd25845 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/config.py +++ b/inferencex-e2e/infx/launch/drivers/srt/config.py @@ -248,21 +248,33 @@ def binder_inputs( lane: SrtLane, request: SrtRequest, decision: PowerDecision, -) -> tuple[int | None, dict[str, str]]: - """The DCGM exporter port and AgentX client env a multi-node recipe is bound with. +) -> tuple[int | None, str | None, dict[str, str]]: + """The DCGM exporter port, CPU exporter source and AgentX client env a recipe is bound with. - The client marks power windows exactly when the launcher collects the point's AgentX power, + The CPU source is the cluster's Grace exporter when the point selected cpu-power. The + client marks power windows exactly when the launcher collects the point's AgentX power, and fails invalid power when the workflow or the lane's power rule requires it. """ port = srt.power_exporter_port if decision.dcgm else None + cpu_source = None + if decision.cpu: + if srt.cpu_power_exporter is None: + raise LaunchError( + f"cluster {cluster.id!r} declares no srt-slurm.cpu-power-exporter for cpu-power" + ) + cpu_source = srt.cpu_power_exporter.source if not request.is_agentic: - return port, {} - return port, { - **agentic_client_env(cluster, srt, lane, request), - "IS_MULTINODE": str(request.is_multinode).lower(), - "ENABLE_AGENTX_POWER": "1" if decision.agentx or decision.adapter else "0", - "REQUIRE_POWER": "1" if decision.require or request.require_power else "0", - } + return port, cpu_source, {} + return ( + port, + cpu_source, + { + **agentic_client_env(cluster, srt, lane, request), + "IS_MULTINODE": str(request.is_multinode).lower(), + "ENABLE_AGENTX_POWER": "1" if decision.agentx or decision.adapter else "0", + "REQUIRE_POWER": "1" if decision.require or request.require_power else "0", + }, + ) def write_lane_config( diff --git a/inferencex-e2e/infx/launch/drivers/srt/power.py b/inferencex-e2e/infx/launch/drivers/srt/power.py index 9e9d12af76..1106834ff4 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/power.py +++ b/inferencex-e2e/infx/launch/drivers/srt/power.py @@ -141,6 +141,8 @@ class PowerDecision: agentx: bool adapter: bool = False require: bool = False + # The point selected cpu-power: the cluster's Grace CPU exporter is bound and expected. + cpu: bool = False expected_cpu_source: str | None = None @@ -152,6 +154,8 @@ def decide_power(cluster_id: str, path: LaunchPath, request: MultiNodeRequest) - Raises ``PowerPolicyError`` for a combination the lane does not allow. """ + if request.cpu_power and not request.power: + raise PowerPolicyError("cpu-power requires power on the same point") if not request.power: return NO_POWER lane = POWER_LANES.get((cluster_id, path)) @@ -160,7 +164,11 @@ def decide_power(cluster_id: str, path: LaunchPath, request: MultiNodeRequest) - for rule in lane.rules: if rule.when(request): return PowerDecision( - dcgm=True, agentx=rule.agentx, adapter=rule.adapter, require=rule.require + dcgm=True, + agentx=rule.agentx, + adapter=rule.adapter, + require=rule.require, + cpu=request.cpu_power, ) message = lane.agentic_error if request.is_agentic and lane.agentic_error else lane.error raise PowerPolicyError(message) diff --git a/inferencex-e2e/infx/launch/drivers/srt/submit.py b/inferencex-e2e/infx/launch/drivers/srt/submit.py index 0b20fe9fb9..26e574eb52 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/submit.py +++ b/inferencex-e2e/infx/launch/drivers/srt/submit.py @@ -84,12 +84,16 @@ def bind_recipe( staged: str, output: str, power_port: int | None, + cpu_power_source: str | None, client_env: Mapping[str, str], ) -> int: - """Compose and bind the staged fragment into ``output``, with telemetry if ``power_port``.""" + """Compose and bind the staged fragment into ``output``, with telemetry if ``power_port`` + and the Grace CPU exporter if ``cpu_power_source``.""" bind = [str(checkout.venv / "bin/python"), "-m", "infx.srt_slurm.workload", staged, output] if power_port is not None: bind += ["--power-port", str(power_port)] + if cpu_power_source is not None: + bind += ["--cpu-power-source", cpu_power_source] for name, value in client_env.items(): bind += ["--client-env", f"{name}={value}"] bind += _fabric(run) diff --git a/inferencex-e2e/infx/launch/request.py b/inferencex-e2e/infx/launch/request.py index 88150eaa61..ebb1b46c70 100644 --- a/inferencex-e2e/infx/launch/request.py +++ b/inferencex-e2e/infx/launch/request.py @@ -150,6 +150,7 @@ class MultiNodeRequest(SrtRequest): srt_recipe: str = Field(alias="SRT_RECIPE") eval_srt_recipe: str | None = Field(None, alias="EVAL_SRT_RECIPE") power: OneFlag = Field(False, alias="POWER") + cpu_power: OneFlag = Field(False, alias="CPU_POWER") class SingleNodeRequest(SrtRequest): diff --git a/inferencex-e2e/infx/matrix/fingerprint.py b/inferencex-e2e/infx/matrix/fingerprint.py index f29cfc6424..982916e883 100644 --- a/inferencex-e2e/infx/matrix/fingerprint.py +++ b/inferencex-e2e/infx/matrix/fingerprint.py @@ -25,6 +25,7 @@ ("benchmark", "concurrencies"), ("benchmark", "env", "CONC"), ("telemetry", "dcgm_exporter", "port"), + ("telemetry", "cpu_power_exporter"), ) EXPORTER_PORT = 0 diff --git a/inferencex-e2e/infx/matrix/generate.py b/inferencex-e2e/infx/matrix/generate.py index 1d5b7ffeda..4bea185cb6 100644 --- a/inferencex-e2e/infx/matrix/generate.py +++ b/inferencex-e2e/infx/matrix/generate.py @@ -1074,6 +1074,8 @@ def _fixed_sequence_entries( entry.update(srt_recipe_fields(config, benchmark)) if benchmark.get(Fields.POWER.value): entry[Fields.POWER.value] = True + if benchmark.get(Fields.CPU_POWER.value): + entry[Fields.CPU_POWER.value] = True entry.update( { Fields.EXP_NAME.value: f"{model_code}_{seq_len_to_str(isl, osl)}", @@ -1197,6 +1199,8 @@ def _agentic_entries( entry.update(srt_recipe_fields(config, benchmark)) if benchmark.get(Fields.POWER.value): entry[Fields.POWER.value] = True + if benchmark.get(Fields.CPU_POWER.value): + entry[Fields.CPU_POWER.value] = True entry.update( { Fields.KV_OFFLOADING.value: kv_offloading, diff --git a/inferencex-e2e/infx/matrix/validation.py b/inferencex-e2e/infx/matrix/validation.py index 1ae17ab77f..0e3d8eb2f2 100644 --- a/inferencex-e2e/infx/matrix/validation.py +++ b/inferencex-e2e/infx/matrix/validation.py @@ -83,6 +83,7 @@ class Fields(Enum): DURATION = "duration" REQUIRE_POWER = "require-power" POWER = "power" + CPU_POWER = "cpu-power" CONC = "conc" MAX_MODEL_LEN = "max-model-len" @@ -288,6 +289,7 @@ class MultiNodeMatrixEntry(BaseModel): osl: int require_power: bool = Field(default=False, alias=Fields.REQUIRE_POWER.value, strict=True) power: bool = Field(default=False, strict=True) + cpu_power: bool = Field(default=False, alias=Fields.CPU_POWER.value, strict=True) prefill: WorkerConfig decode: WorkerConfig conc: list[int] @@ -389,6 +391,7 @@ class MultiNodeAgenticMatrixEntry(BaseModel): ) node_count: int = Field(alias=Fields.NODE_COUNT.value, gt=0, strict=True) power: bool = Field(default=False, strict=True) + cpu_power: bool = Field(default=False, alias=Fields.CPU_POWER.value, strict=True) prefill: WorkerConfig decode: WorkerConfig conc: list[int] @@ -604,6 +607,13 @@ def validate_single_node_topology(self) -> Self: return _validate_tp_context_topology(self) +def _cpu_power_rides_on_power[T](entry: T) -> T: + """``cpu-power`` adds the cluster's Grace CPU exporter to a point that measures power.""" + if getattr(entry, "cpu_power", False) and not getattr(entry, "power", False): + raise ValueError(f"{Fields.CPU_POWER.value} requires {Fields.POWER.value} on the entry") + return entry + + class MultiNodeSearchSpaceEntry(BaseModel): """Multinode search space configuration.""" @@ -620,6 +630,7 @@ class MultiNodeSearchSpaceEntry(BaseModel): default=None, alias=Fields.EVAL_SRT_RECIPE.value ) power: bool = Field(default=False, strict=True) + cpu_power: bool = Field(default=False, alias=Fields.CPU_POWER.value, strict=True) num_nodes: int | None = Field(default=None, alias=Fields.NUM_NODES.value, gt=0, strict=True) router: ComponentMetadata | None = None kv_p2p_transfer: str | None = Field( @@ -629,6 +640,10 @@ class MultiNodeSearchSpaceEntry(BaseModel): conc_end: int | None = Field(default=None, alias=Fields.CONC_END.value) conc_list: list[int] | None = Field(default=None, alias=Fields.CONC_LIST.value) + @model_validator(mode="after") + def validate_cpu_power(self) -> Self: + return _cpu_power_rides_on_power(self) + @model_validator(mode="after") def validate_conc_fields(self) -> Self: return _validate_conc_fields(self) @@ -682,6 +697,7 @@ class AgenticCodingSearchSpaceEntry(BaseModel): default=None, alias=Fields.EVAL_SRT_RECIPE.value ) power: bool = Field(default=False, strict=True) + cpu_power: bool = Field(default=False, alias=Fields.CPU_POWER.value, strict=True) pp: int = Field(default=1, gt=0, strict=True) dcp_size: int = Field(default=1, alias=Fields.DCP_SIZE.value, gt=0, strict=True) pcp_size: int = Field(default=1, alias=Fields.PCP_SIZE.value, gt=0, strict=True) @@ -745,6 +761,7 @@ def validate_topology_fields(self) -> Self: f"Only multinode entries measure DCGM power; remove {Fields.POWER.value}" ) _validate_tp_context_topology(self) + _cpu_power_rides_on_power(self) if has_aggregate_worker or has_complete_multinode: explicitly_single_node_fields = { "pp", diff --git a/inferencex-e2e/infx/srt_slurm/generate.py b/inferencex-e2e/infx/srt_slurm/generate.py index 9d3c30fd79..6d9b7f400d 100644 --- a/inferencex-e2e/infx/srt_slurm/generate.py +++ b/inferencex-e2e/infx/srt_slurm/generate.py @@ -67,6 +67,7 @@ def point_environment(point: Mapping[str, Any]) -> dict[str, str]: IS_MULTINODE="true", SRT_RECIPE=point["srt-recipe"], POWER="1" if point.get("power") else "0", + CPU_POWER="1" if point.get("cpu-power") else "0", CONC_LIST=" ".join(map(str, point["conc"])), ) if agentic: @@ -98,15 +99,16 @@ def _binder_inputs( environment: Mapping[str, str], placement: tuple[str, Cluster] | None, root: Path, -) -> tuple[int | None, dict[str, str]]: - """The DCGM exporter port and AgentX client env the launcher binds a multi-node point with. +) -> tuple[int | None, str | None, dict[str, str]]: + """The DCGM exporter port, CPU exporter source and AgentX client env the launcher binds a + multi-node point with. Raises ``ValueError`` when they depend on a cluster the runner label does not single out, or the cluster's lanes refuse the point. """ agentic = environment["IS_AGENTIC"] == "1" if "prefill" not in point or not (agentic or point.get("power")): - return None, {} + return None, None, {} if placement is None: raise ValueError( f"runner {point['runner']!r} must schedule on one cluster to bind the cluster's " @@ -139,13 +141,15 @@ def bound_variant( *, expand: Callable[..., list[tuple[str | None, dict[str, Any]]]] = selected_recipes, power_port: int | None = None, + cpu_power_source: str | None = None, client_env: Mapping[str, str] | None = None, ) -> tuple[str | None, dict[str, Any]]: """The variant the launcher submits for ``point``, composed and bound. - Multi-node variants get the DCGM telemetry block on ``power_port`` and the client paths - ``client_env``, the binder inputs the launcher computes. ``'@fabric.'`` values stay - unresolved: recipe fingerprints hash them as written. + Multi-node variants get the DCGM telemetry block on ``power_port``, the Grace CPU exporter + on ``cpu_power_source``, and the client paths ``client_env``: the binder inputs the launcher + computes. ``'@fabric.'`` values stay unresolved: recipe fingerprints hash them as + written. """ if "prefill" not in point: selected, recipe = select_recipe( @@ -158,6 +162,7 @@ def bound_variant( root=root, expand=expand, power_port=power_port, + cpu_power_source=cpu_power_source, client_env=client_env, ) @@ -233,11 +238,12 @@ def generate_recipes( for point in points: environment = point_environment(point) placement = _placement(inventory, point["runner"]) - power_port, client_env = _binder_inputs(point, environment, placement, root) + power_port, cpu_source, client_env = _binder_inputs(point, environment, placement, root) fabric = _cluster_fabric(placement[1]) if placement else None variant, recipe = bound_variant( - point, environment, root, power_port=power_port, client_env=client_env - ) + point, environment, root, + power_port=power_port, cpu_power_source=cpu_source, client_env=client_env, + ) # fmt: skip if fabric is not None: recipe = resolve_fabric(recipe, fabric) check_setup_script(recipe, root / point["srt-recipe"].partition(":")[0], root) diff --git a/inferencex-e2e/infx/srt_slurm/workload.py b/inferencex-e2e/infx/srt_slurm/workload.py index 50a8372552..15f43750de 100644 --- a/inferencex-e2e/infx/srt_slurm/workload.py +++ b/inferencex-e2e/infx/srt_slurm/workload.py @@ -41,6 +41,7 @@ ("benchmark", "concurrencies"), ("telemetry", "enabled"), ("telemetry", "dcgm_exporter", "port"), + ("telemetry", "cpu_power_exporter"), ) _FIXED_ENV = ("MODEL", "ISL", "OSL", "CONC", "CONC_LIST", "RANDOM_RANGE_RATIO", "USE_CHAT_TEMPLATE") _AGENTIC_ENV = ("CONC", "CONC_LIST", "RESULT_DIR", "AGENTIC_OUTPUT_DIR", "HF_HUB_CACHE", @@ -187,11 +188,13 @@ def compose_recipe( multinode: bool, root: Path, power_port: int | None = None, + cpu_power_source: str | None = None, native_power: bool = False, ) -> dict[str, Any]: """The fragment at ``path`` over its lane's shared block, under ``base`` for bundles. - ``power_port`` adds the DCGM telemetry block with its exporter on that port. + ``power_port`` adds the DCGM telemetry block with its exporter on that port, and + ``cpu_power_source`` the cluster's Grace CPU power exporter beside it. """ raw = yaml.safe_load(path.read_text()) if not isinstance(raw, dict): @@ -201,7 +204,9 @@ def compose_recipe( if native_power: shared = merge_blocks(shared, _block(root / NATIVE_TELEMETRY_BLOCK)) if power_port is not None: - exporter = {"telemetry": {"dcgm_exporter": {"port": power_port}}} + exporter: dict[str, Any] = {"telemetry": {"dcgm_exporter": {"port": power_port}}} + if cpu_power_source is not None: + exporter["telemetry"]["cpu_power_exporter"] = {"source": cpu_power_source} shared = merge_blocks(shared, merge_blocks(_block(root / TELEMETRY_BLOCK), exporter)) if "base" in raw: return {**raw, "base": merge_blocks(shared, raw["base"])} @@ -351,13 +356,19 @@ def bind_multinode( root: Path, expand: Callable[..., list[tuple[str | None, dict[str, Any]]]] = selected_recipes, power_port: int | None = None, + cpu_power_source: str | None = None, client_env: Mapping[str, str] | None = None, ) -> tuple[str | None, dict[str, Any]]: """Bind the one variant ``recipe`` (``fragment[:selector]``) selects; return its name too.""" agentic = environment["IS_AGENTIC"] == "1" path, _, selector = recipe.partition(":") composed = compose_recipe( - Path(path), agentic=agentic, multinode=True, root=root, power_port=power_port + Path(path), + agentic=agentic, + multinode=True, + root=root, + power_port=power_port, + cpu_power_source=cpu_power_source, ) variants = expand(composed, selector or None) if len(variants) != 1: @@ -420,6 +431,11 @@ def main(argv: list[str] | None = None) -> None: parser.add_argument( "--power-port", type=int, help="merge the DCGM telemetry block, its exporter on this port" ) + parser.add_argument( + "--cpu-power-source", + choices=("acpi", "dcgm"), + help="bind the cluster's Grace CPU power exporter with this source (needs --power-port)", + ) parser.add_argument( "--client-env", type=_assignment, @@ -437,6 +453,7 @@ def main(argv: list[str] | None = None) -> None: os.environ, root=root, power_port=args.power_port, + cpu_power_source=args.cpu_power_source, client_env=dict(args.client_env), ) bound = resolve_fabric(bound, args.fabric) diff --git a/inferencex-e2e/infx/tests/launch/test_launch_policy.py b/inferencex-e2e/infx/tests/launch/test_launch_policy.py index 1f98d30d1b..6bb2182824 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_policy.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_policy.py @@ -83,6 +83,19 @@ def test_power_eligibility_of_each_cluster_lane( assert not decide_power(cluster, path, _multinode(**env, POWER="0")).dcgm +@pytest.mark.parametrize(("power", "cpu_power", "expected"), [ + ("1", "1", True), ("1", "0", False), ("0", "1", ERR), +]) # fmt: skip +def test_cpu_power_rides_on_a_power_point(power, cpu_power, expected): + request = _multinode(IS_AGENTIC="0", MODEL_PREFIX="dsr1", PRECISION="fp8", + FRAMEWORK="dynamo-sglang", POWER=power, CPU_POWER=cpu_power) # fmt: skip + if expected == ERR: + with pytest.raises(PowerPolicyError, match="cpu-power requires power"): + decide_power("gb200-nv", MULTI, request) + return + assert decide_power("gb200-nv", MULTI, request).cpu is expected + + def test_gb200_reports_agentic_misses_distinctly(): request = _multinode(IS_AGENTIC="1", MODEL_PREFIX="dsv4", PRECISION="fp4", FRAMEWORK="dynamo-sglang", CONC="4", CONC_LIST="4", POWER="1") # fmt: skip diff --git a/inferencex-e2e/infx/tests/launch/test_srt_config.py b/inferencex-e2e/infx/tests/launch/test_srt_config.py index 959a05f3b5..95ab96c798 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_config.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_config.py @@ -12,7 +12,7 @@ from infx.clusters import Cluster, load_inventory from infx.clusters.slurm import slurm_settings from infx.launch.backends.slurm import SlurmBackend -from infx.launch.context import Launch +from infx.launch.context import Launch, LaunchError from infx.launch.drivers.srt.checkout import Checkout, compute_workspace from infx.launch.drivers.srt.config import SrtJob, binder_inputs, pyxis_spelling, render, write from infx.launch.drivers.srt.lanes import SrtLane @@ -292,9 +292,10 @@ def test_the_agentx_client_measures_power_exactly_when_the_launcher_collects_it( "CONC_LIST": "4", }) # fmt: skip - port, client_env = binder_inputs( + port, cpu_source, client_env = binder_inputs( record, slurm_settings(record).srt_slurm, SrtLane(), request, decision ) + assert cpu_source is None switches = ( client_env.get("IS_MULTINODE"), @@ -302,3 +303,26 @@ def test_the_agentx_client_measures_power_exactly_when_the_launcher_collects_it( client_env.get("REQUIRE_POWER"), ) assert (port, *switches) == expected + + +@pytest.mark.parametrize("declared", [True, False]) +def test_cpu_power_binds_the_clusters_grace_exporter_or_fails_before_binding(tmp_path, declared): + srt = {"power-exporter-port": 9401} + if declared: + srt["cpu-power-exporter"] = {"source": "acpi"} + record = cluster(srt=srt) + request = MultiNodeRequest.from_env({ + "RUNNER_NAME": "c_0", "GITHUB_WORKSPACE": str(tmp_path), "IMAGE": "i", + "FRAMEWORK": "dynamo-sglang", "MODEL_PREFIX": "m", "PRECISION": "fp8", + "SPEC_DECODING": "none", "RESULT_FILENAME": "r", "IS_AGENTIC": "0", "RUN_EVAL": "false", + "EVAL_ONLY": "false", "SRT_RECIPE": "r.yaml", + }) # fmt: skip + settings = slurm_settings(record).srt_slurm + cpu = PowerDecision(dcgm=True, agentx=False, cpu=True) + if not declared: + with pytest.raises(LaunchError, match=r"no srt-slurm\.cpu-power-exporter"): + binder_inputs(record, settings, SrtLane(), request, cpu) + return + assert binder_inputs(record, settings, SrtLane(), request, cpu)[:2] == (9401, "acpi") + gpu_only = PowerDecision(dcgm=True, agentx=False) + assert binder_inputs(record, settings, SrtLane(), request, gpu_only)[:2] == (9401, None) diff --git a/inferencex-e2e/infx/tests/launch/test_srt_driver.py b/inferencex-e2e/infx/tests/launch/test_srt_driver.py index 92e9cb34c1..0632aea1fd 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_driver.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_driver.py @@ -376,6 +376,8 @@ def test_power_lane_stages_provenance_and_validates_each_concurrency( assert bound["benchmark"]["env"]["REQUIRE_POWER"] == require_power assert bound["telemetry"]["collector_join_timeout_seconds"] == 12 assert (bound["telemetry"]["enabled"], bound["telemetry"]["dcgm_exporter"]["port"]) == (True, 9401) + assert "cpu_power_exporter" not in bound["telemetry"] + assert all("CPU_POWER_EXPORTER_RELEASE" not in call for call in lines(harness.logs, "make")) runs = lines(harness.logs, "adapter") assert [run.split("--result-dir ")[1].split()[0].rsplit("/", 1)[1] for run in runs] == ["conc_4"] @@ -676,20 +678,26 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey ) github_env = harness.tmp / "github-env" - recipe = LANE_FRAGMENT + POWER_TELEMETRY + " cpu_power_exporter:\n source: acpi\n" env = lane_env( harness, "gb300-nv", - recipe, + LANE_FRAGMENT + POWER_TELEMETRY, MODEL_PREFIX="qwen3.5", PRECISION="fp8", FRAMEWORK="dynamo-sglang", MODEL="Qwen/Qwen3.5-397B-A17B-FP8", REQUIRE_POWER="1", POWER="1", + CPU_POWER="1", GITHUB_ENV=str(github_env), ) assert_ok(launch(env, harness.config, harness.workspace)) + [checkout] = harness.workspace.glob("srt-slurm-9001-*") + bound = yaml.safe_load((checkout / "recipe.yaml").read_text()) + assert bound["telemetry"]["cpu_power_exporter"] == {"source": "acpi"} + assert bound["telemetry"]["dcgm_exporter"]["port"] == 19401 + [setup] = [call for call in lines(harness.logs, "make") if call.startswith("setup ")] + assert "CPU_POWER_EXPORTER_RELEASE=v2.51.1" in setup.split() exported = dict(line.split("=", 1) for line in github_env.read_text().splitlines()) pkg = build_package(harness.tmp / "power-input") monkeypatch.chdir(harness.tmp) diff --git a/inferencex-e2e/infx/tests/matrix/test_generate_sweep_configs.py b/inferencex-e2e/infx/tests/matrix/test_generate_sweep_configs.py index 66f5871307..f32bce0f9e 100644 --- a/inferencex-e2e/infx/tests/matrix/test_generate_sweep_configs.py +++ b/inferencex-e2e/infx/tests/matrix/test_generate_sweep_configs.py @@ -3268,6 +3268,22 @@ def test_require_power_is_scoped_to_one_multinode_fixed_sequence( expand_full_sweep(config, sample_runner_config) +def test_cpu_power_reaches_the_planner_row_beside_power( + sample_multinode_config, sample_runner_config +): + from infx.matrix.generate import expand_full_sweep + + config = sample_multinode_config + point = next(iter(config.values()))["scenarios"]["fixed-seq-len"][0]["search-space"][0] + point["power"] = True + rows = expand_full_sweep(config, sample_runner_config) + assert rows + assert all(row["power"] is True and "cpu-power" not in row for row in rows) + point["cpu-power"] = True + rows = expand_full_sweep(config, sample_runner_config) + assert all(row["cpu-power"] is True for row in rows) + + def test_single_node_fixed_sequence_rejects_require_power(sample_single_node_config): from infx.matrix.validation import validate_master_config diff --git a/inferencex-e2e/infx/tests/matrix/test_validation.py b/inferencex-e2e/infx/tests/matrix/test_validation.py index 3509be0141..61dc513f98 100644 --- a/inferencex-e2e/infx/tests/matrix/test_validation.py +++ b/inferencex-e2e/infx/tests/matrix/test_validation.py @@ -639,6 +639,13 @@ def test_valid_aggregate_worker(self): assert entry.prefill is None assert entry.decode is None + def test_cpu_power_rides_on_power(self): + row = {"worker": {"tp": 8, "ep": 1, "dp-attn": False}, "num-nodes": 2, + "conc-list": [1], "cpu-power": True} # fmt: skip + with pytest.raises(ValidationError, match="cpu-power requires power"): + MultiNodeSearchSpaceEntry(**row) + assert MultiNodeSearchSpaceEntry(**{**row, "power": True}).cpu_power is True + def test_missing_conc_specification(self): with pytest.raises(ValidationError): MultiNodeSearchSpaceEntry(**{ @@ -1478,3 +1485,11 @@ def test_only_multinode_agentic_rows_measure_power(topology, accepted): else: with pytest.raises(ValidationError, match="Only multinode entries measure DCGM power"): AgenticCodingSearchSpaceEntry(**row) + + +def test_agentic_cpu_power_rides_on_power(): + worker = {"worker": {"num-worker": 1, "tp": 8, "ep": 1, "dp-attn": False}} + row = {"srt-recipe": "r.yaml", "conc-list": [4], "cpu-power": True, **worker} + with pytest.raises(ValidationError, match="cpu-power requires power"): + AgenticCodingSearchSpaceEntry(**row) + assert AgenticCodingSearchSpaceEntry(**{**row, "power": True}).cpu_power is True diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_generate.py b/inferencex-e2e/infx/tests/srt_slurm/test_generate.py index 200015fb41..0159220589 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_generate.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_generate.py @@ -198,6 +198,34 @@ def test_a_multinode_agentx_power_point_gets_its_clusters_exporter_port_and_clie assert recipe["benchmark"]["concurrencies"] == [8] +@pytest.mark.parametrize("declared", [True, False]) +def test_a_cpu_power_point_binds_the_clusters_grace_exporter_or_fails( + project, power_lane, tmp_path, capsys, declared +): + master = project / "configs/master.yaml" + config = yaml.safe_load(master.read_text()) + config["agentx-multi"]["scenarios"]["agentic-coding"][0]["search-space"][0]["cpu-power"] = True + master.write_text(yaml.safe_dump(config)) + if declared: + runners = project / "configs/runners.yaml" + inventory = yaml.safe_load(runners.read_text()) + srt = inventory["clusters"]["fixture"]["slurm"]["srt-slurm"] + srt["cpu-power-exporter"] = {"source": "acpi"} + runners.write_text(yaml.safe_dump(inventory)) + output = tmp_path / "out" + + if not declared: + with pytest.raises(SystemExit): + generate(project, output, "agentx-multi") + assert "declares no srt-slurm.cpu-power-exporter" in capsys.readouterr().err + assert not output.exists() + return + assert generate(project, output, "agentx-multi") == 0 + [(_, recipe)] = written(output) + assert recipe["telemetry"]["cpu_power_exporter"] == {"source": "acpi"} + assert recipe["telemetry"]["dcgm_exporter"]["port"] == 9555 + + def test_a_multinode_agentx_point_without_power_turns_its_clients_power_off( project, power_lane, tmp_path ): diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_workload.py b/inferencex-e2e/infx/tests/srt_slurm/test_workload.py index b2aca470eb..b01668a3b0 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_workload.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_workload.py @@ -94,6 +94,8 @@ def test_bundles_take_the_shared_block_under_base_and_keep_their_variants(projec ({"base": {}, "override_c": {"benchmark": {"concurrencies": [4]}}}, False, False, "override_c.benchmark.concurrencies (= [4])"), ({"telemetry": {"enabled": True}}, False, True, "telemetry.enabled (= True)"), + ({"base": {"telemetry": {"cpu_power_exporter": {"source": "acpi"}}}}, True, True, + "base.telemetry.cpu_power_exporter (= {'source': 'acpi'})"), ({"benchmark": {"env": {"HF_HUB_CACHE": "/elsewhere"}}}, True, True, "benchmark.env.HF_HUB_CACHE (= '/elsewhere')"), ({"base": {"benchmark": {"env": {"RESULT_DIR": "/logs"}}}}, True, False, @@ -168,6 +170,10 @@ def test_a_power_point_gets_telemetry_and_its_concurrencies_on_a_bundle_variant( "dcgm_exporter": {"container_image": "dcgm-exporter", "port": 19401}, } # fmt: skip assert bound["benchmark"]["concurrencies"] == [8] + with_cpu = compose_recipe( + path, agentic=True, multinode=True, root=project, power_port=19401, cpu_power_source="acpi" + ) + assert with_cpu["base"]["telemetry"]["cpu_power_exporter"] == {"source": "acpi"} unpowered = compose_recipe(path, agentic=True, multinode=True, root=project)["base"] assert "concurrencies" not in bind_workload(unpowered, AGENTX_ENV, agentic=True, multinode=True)[ "benchmark" From 91150455252d97651d12e73e70af1033b7799211 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:22:17 -0700 Subject: [PATCH 22/35] refactor(power): read only the wide Grace socket samples in the CPU leg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit srt-slurm v2.51.1 cpu_samples.py:110 writes only CPU_SAMPLES_HEADER and binds no module meter, so the v1 long format, the OEM-label regexes and the module and DCGM headline paths never ran. The headline comes from the CPU:cpuSidePowerUsageW rows alone and the two module keys leave CPU_METRIC_KEYS. 中文：CPU leg 仅读取宽格式 Grace socket 采样；删除从未执行的 v1 长格式、OEM 标签正则以及 module/DCGM headline 路径，并从 CPU_METRIC_KEYS 移除两个 module 指标。 --- inferencex-e2e/infx/results/power/__init__.py | 2 - inferencex-e2e/infx/results/power/audit.py | 7 +- inferencex-e2e/infx/results/power/cpu_side.py | 208 +++++++----------- .../infx/results/power/multinode.py | 4 +- .../power/test_aggregate_power_multinode.py | 171 +++----------- 5 files changed, 108 insertions(+), 284 deletions(-) diff --git a/inferencex-e2e/infx/results/power/__init__.py b/inferencex-e2e/infx/results/power/__init__.py index f0fbcc5011..fe47881bfa 100644 --- a/inferencex-e2e/infx/results/power/__init__.py +++ b/inferencex-e2e/infx/results/power/__init__.py @@ -36,8 +36,6 @@ "avg_cpu_socket_power_w", "avg_total_cpu_power_w", "total_cpu_energy_j", - "avg_total_module_power_w", - "total_module_energy_j", "avg_total_cpu_rail_power_w", "total_cpu_rail_energy_j", "avg_total_cpu_sysio_power_w", diff --git a/inferencex-e2e/infx/results/power/audit.py b/inferencex-e2e/infx/results/power/audit.py index a42134d852..79303ac2ba 100644 --- a/inferencex-e2e/infx/results/power/audit.py +++ b/inferencex-e2e/infx/results/power/audit.py @@ -7,7 +7,7 @@ from collections.abc import Mapping from typing import Any -from .cpu_side import HEADLINE_PREFERENCE +from .cpu_side import SENSOR_GRACE _REASON_CODE = re.compile(r"[a-z][a-z0-9_]{0,63}") _CPU_COUNT_FIELDS = ("expected_sockets", "observed_sockets", "sample_row_count") @@ -24,9 +24,8 @@ def _bounded_reasons(values: Any) -> list[str]: def _cpu_audit(cpu: Mapping[str, Any]) -> dict[str, Any]: """Project the CPU-side provenance: sensor kind, source, counts, reasons.""" summary: dict[str, Any] = {} - kind = cpu.get("sensor_kind") - if kind in HEADLINE_PREFERENCE: - summary["sensor_kind"] = kind + if cpu.get("sensor_kind") == SENSOR_GRACE: + summary["sensor_kind"] = SENSOR_GRACE source = cpu.get("source") if isinstance(source, str) and 0 < len(source) <= 32: summary["source"] = source diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index 87cbe107a2..23d8baf112 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -2,20 +2,18 @@ srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` and a non-authoritative ``power/cpu/cpu_manifest.json`` beside the GPU DCGM package -(srt-slurm ``src/srtctl/core/power/cpu_session.py``). Historical v2.2.1 -packages use the v1 long format, one row per sensor reading. The current -producer uses the v2 wide format, one row per (scrape, host, socket) with -the ACPI component rails as reference columns. Both are accepted here. - -Each (hostname, socket) headline series is integrated over the GPU leg's -bound formal window with the shared trapezoid and boundary interpolation and -published as additive metrics next to the GPU-board numbers. The v2 ACPI -CPU-only and SysIO rails are integrated the same way when every socket reports -them; a rail with gaps drops only its own metrics. Any failure records -``cpu_power_valid=0`` with reason codes and leaves every GPU field untouched. -This module only judges the leg: the caller fails the job only when power is -required and the recipe declares a CPU source (``expected_source``). A package -without ``cpu/`` emits nothing unless a source is expected. +(srt-slurm ``src/srtctl/core/power/cpu_session.py``). The pinned producer writes +the wide format: one row per (scrape, host, socket) whose ``sensor`` names the +Grace socket meter and whose ``cpu_rail_w`` / ``soc_w`` columns carry the ACPI +component rails of that socket. + +Each socket's Grace socket series is integrated over the GPU leg's bound formal +window with the shared trapezoid and boundary interpolation and published as +additive metrics next to the GPU-board numbers. The CPU-only and SysIO rails +are integrated the same way when every socket reports them; a rail with gaps +drops only its own metrics. Any failure records ``cpu_power_valid=0`` with +reason codes and leaves every GPU field untouched. This module only judges the +leg; the caller decides whether an invalid leg fails the job. """ from __future__ import annotations @@ -24,7 +22,6 @@ import itertools import json import math -import re from collections.abc import Iterable from dataclasses import dataclass, field from pathlib import Path @@ -36,9 +33,8 @@ CPU_SAMPLES_FILENAME = "samples.csv" CPU_MANIFEST_FILENAME = "cpu_manifest.json" -# srt-slurm contract.py: CPU_SAMPLES_HEADER_V1 (written by v2.2.1) and the -# wide CPU_SAMPLES_HEADER (written by v2.15.0, which reads both). -CPU_SAMPLES_HEADER_V1 = ( +# srt-slurm contract.CPU_SAMPLES_HEADER: the wide format every pinned producer writes. +CPU_SAMPLES_HEADER = ( "schema_version", "timestamp_unix", "hostname", @@ -46,48 +42,26 @@ "sensor", "socket_id", "power_w", - "total_power_w", -) -CPU_SAMPLES_HEADER_V2 = ( - *CPU_SAMPLES_HEADER_V1[:7], "cpu_rail_w", "soc_w", "dram_w", "total_power_w", ) -_HEADERS = {1: CPU_SAMPLES_HEADER_V1, 2: CPU_SAMPLES_HEADER_V2} +CPU_SAMPLES_SCHEMA_VERSION = 2 # One GB200/GB300 compute tray per worker host, two Grace sockets per tray. SOCKETS_PER_HOST = 2 # Shared with the GPU leg (srt-slurm contract.MAX_SAMPLE_GAP_SECONDS). MAX_SAMPLE_GAP_SECONDS = 3.0 -SENSOR_MODULE = "module" +# The headline meter: "Grace Power Socket N", the whole socket (CPU rail, SysIO, +# LPDDR5X and regulator loss). srt-slurm names its wide-format rows +# ``CPU:cpuSidePowerUsageW`` (cpu_rails.SENSOR_SUFFIXES). SENSOR_GRACE = "grace_socket" -SENSOR_DCGM = "dcgm_cpu_rail" -# Headline preference per socket. ACPI component rails ride on the Grace row -# and never select the headline or its verdict. -HEADLINE_PREFERENCE = (SENSOR_MODULE, SENSOR_GRACE, SENSOR_DCGM) -# v2 Grace-row column -> published rail family: "CPU Power Socket N" (cores, +_GRACE_SENSOR_SUFFIX = "cpuSidePowerUsageW" +# Wide-format rail column -> published rail family: "CPU Power Socket N" (cores, # no SysIO) and "SysIO Power Socket N". dram_w stays blank on NVL72 firmware. -_RAIL_COLUMNS = {"cpu_rail_w": "cpu_rail", "soc_w": "cpu_sysio"} -# Grace-side kinds, in fallback order, that feed the ``*_cpu_*`` keys. -_GRACE_SIDE_PREFERENCE = (SENSOR_GRACE, SENSOR_DCGM) - -# ``CPU:`` sensor names (srt-slurm cpu_rails.SENSOR_SUFFIXES) -# and firmware OEM labels (srt-slurm cpu_rails.ACPI_LABEL_PATTERNS for the -# Grace total). The Module label is the NVL72 tray firmware's whole-module -# sensor, which srt-slurm does not classify yet; a v2.2.1 package therefore -# yields the Grace socket total until the exporter learns the label. -_SUFFIX_KINDS = {"cpuSidePowerUsageW": SENSOR_GRACE, "cpuPowerUsageW": SENSOR_DCGM} -_LABEL_KINDS = ( - (SENSOR_MODULE, re.compile(r"\bModule\s+Power\s+Socket\s+\d+\b", re.IGNORECASE)), - (SENSOR_GRACE, re.compile(r"\bGrace\s+Power\s+Socket\s+\d+\b", re.IGNORECASE)), - ( - SENSOR_GRACE, - re.compile(r"\bTotal(?:\s+Input)?\s+Power(?:\s+in\s+uW)?\s+Socket\s+\d+\b", re.IGNORECASE), - ), -) +RAIL_COLUMNS = {"cpu_rail_w": "cpu_rail", "soc_w": "cpu_sysio"} _INTEGRATION_METHOD = "per_socket_trapezoidal_with_linear_boundary_interpolation" @@ -134,7 +108,6 @@ def to_payload(self) -> dict[str, Any]: "expected_sockets": self.expected_sockets, "observed_sockets": self.observed_sockets, "sample_row_count": self.sample_row_count, - "headline_preference": list(HEADLINE_PREFERENCE), "integration_method": _INTEGRATION_METHOD, "max_sample_gap_seconds": MAX_SAMPLE_GAP_SECONDS, "per_series_max_sample_gap_s": dict(self.per_series_max_sample_gap_s), @@ -147,16 +120,13 @@ def to_payload(self) -> dict[str, Any]: } -def classify_sensor(sensor: str) -> str | None: - """Headline kind for a ``sensor`` cell from either writer version, else None.""" - _, _, suffix = sensor.partition(":") - if suffix in _SUFFIX_KINDS: - return _SUFFIX_KINDS[suffix] - return next((kind for kind, pattern in _LABEL_KINDS if pattern.search(sensor)), None) +def is_grace_sensor(sensor: str) -> bool: + """True for the Grace socket meter's ``CPU:cpuSidePowerUsageW`` sensor name.""" + return sensor.partition(":")[2] == _GRACE_SENSOR_SUFFIX -def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: - if len(raw) != len(_HEADERS[version]): +def _parse_row(raw: list[str]) -> CpuSampleRow | None: + if len(raw) != len(CPU_SAMPLES_HEADER): return None try: schema_version = int(raw[0]) @@ -166,15 +136,15 @@ def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: except ValueError: return None hostname, source, sensor = raw[2], raw[3], raw[4] - if schema_version != version or not hostname or not source or not sensor: + if schema_version != CPU_SAMPLES_SCHEMA_VERSION or not hostname or not source or not sensor: return None if not math.isfinite(timestamp_unix) or not math.isfinite(power_w) or power_w < 0: return None if socket_id < 0: return None rails = [] - for column, family in _RAIL_COLUMNS.items(): - cell = raw[_HEADERS[version].index(column)] if column in _HEADERS[version] else "" + for column, family in RAIL_COLUMNS.items(): + cell = raw[CPU_SAMPLES_HEADER.index(column)] if not cell: continue try: @@ -196,7 +166,7 @@ def _parse_row(raw: list[str], version: int) -> CpuSampleRow | None: def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, ...]]: - """Strictly parse either sample generation; malformed rows invalidate the leg.""" + """Strictly parse the wide sample file; malformed rows invalidate the leg.""" if not path.is_file(): return (), ("cpu_samples_missing",) rows: list[CpuSampleRow] = [] @@ -204,12 +174,10 @@ def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, . try: with open(path, newline="", encoding="utf-8") as handle: reader = csv.reader(handle) - header = next(reader, None) - version = next((v for v, h in _HEADERS.items() if header == list(h)), None) - if version is None: + if next(reader, None) != list(CPU_SAMPLES_HEADER): return (), ("cpu_samples_header_mismatch",) for raw in reader: - row = _parse_row(raw, version) + row = _parse_row(raw) if row is None: _append_reason(reasons, "cpu_samples_malformed") continue @@ -227,43 +195,35 @@ def _manifest_is_object(path: Path) -> bool: SocketKey = tuple[str, int] +# (host, socket) -> family -> {timestamp: watts}; the Grace meter and its rails. +SocketSeries = dict[SocketKey, dict[str, dict[float, float]]] -def _headline_series( - rows: Iterable[CpuSampleRow], -) -> tuple[dict[SocketKey, dict[str, dict[float, float]]], dict[str, set[str]], bool]: - """Group headline rows as (host, socket) -> kind -> {timestamp: watts}.""" - series: dict[SocketKey, dict[str, dict[float, float]]] = {} - sources: dict[str, set[str]] = {} +def _grace_series(rows: Iterable[CpuSampleRow]) -> tuple[SocketSeries, set[str], bool]: + """Group the Grace socket rows per socket, with the sources seen and a duplicate flag.""" + series: SocketSeries = {} + sources: set[str] = set() duplicate = False for row in rows: - kind = classify_sensor(row.sensor) - if kind is None: + if not is_grace_sensor(row.sensor): continue - by_kind = series.setdefault((row.hostname, row.socket_id), {}) - points = by_kind.setdefault(kind, {}) + by_family = series.setdefault((row.hostname, row.socket_id), {}) + points = by_family.setdefault(SENSOR_GRACE, {}) if row.timestamp_unix in points: duplicate = True points[row.timestamp_unix] = row.power_w - sources.setdefault(kind, set()).add(row.source) - if kind == SENSOR_GRACE: - for family, watts in row.rails: - by_kind.setdefault(family, {})[row.timestamp_unix] = watts + sources.add(row.source) + for family, watts in row.rails: + by_family.setdefault(family, {})[row.timestamp_unix] = watts return series, sources, duplicate -def _select_feeds(common_kinds: set[str]) -> tuple[str | None, dict[str, str]]: - """Pick the headline kind and which kind feeds each metric family.""" - selected = next((kind for kind in HEADLINE_PREFERENCE if kind in common_kinds), None) - if selected is None: - return None, {} - if selected != SENSOR_MODULE: - return selected, {selected: "cpu"} - feeds = {SENSOR_MODULE: "module"} - grace = next((kind for kind in _GRACE_SIDE_PREFERENCE if kind in common_kinds), None) - if grace is not None: - feeds[grace] = "cpu" - return selected, feeds +def _max_sample_gap(samples: list[tuple[float, float]], start: float, end: float) -> float | None: + """The widest gap between the samples bracketing ``[start, end]``; None when not bracketed.""" + sequence = _bracketing_sequence(tuple(t for t, _ in samples), start, end) + if sequence is None: + return None + return max((later - earlier for earlier, later in itertools.pairwise(sequence)), default=0.0) def validate_cpu_leg( @@ -286,8 +246,8 @@ def validate_cpu_leg( rows, sample_reasons = read_cpu_samples(cpu_dir / CPU_SAMPLES_FILENAME) audit.sample_row_count = len(rows) audit.invalidate(*sample_reasons) - series, sources, duplicate = _headline_series(rows) - if duplicate: + series, sources, duplicate = _grace_series(rows) + if duplicate or len(sources) > 1: audit.invalidate("cpu_samples_malformed") expected_keys = { (host, socket) for host in expected_hosts for socket in range(SOCKETS_PER_HOST) @@ -298,46 +258,34 @@ def validate_cpu_leg( audit.invalidate("cpu_window_unavailable") if audit.reason_codes: return audit - if not expected_keys or set(series) != expected_keys: audit.invalidate("cpu_socket_count_mismatch") return audit - common_kinds = set.intersection(*(set(kinds) for kinds in series.values())) - selected, feeds = _select_feeds(common_kinds) - if selected is None or len(sources[selected]) != 1: - audit.invalidate("cpu_sensor_kind_mixed") - return audit - audit.sensor_kind = selected - audit.source = next(iter(sources[selected])) - if expected_source is not None and ( - audit.source != expected_source - or (expected_source == "acpi" and selected not in {SENSOR_GRACE, SENSOR_MODULE}) - ): + audit.sensor_kind = SENSOR_GRACE + audit.source = next(iter(sources)) + if expected_source is not None and audit.source != expected_source: audit.invalidate("cpu_sensor_source_mismatch") return audit - if SENSOR_GRACE in feeds: - feeds.update({rail: rail for rail in _RAIL_COLUMNS.values() if rail in common_kinds}) + # A rail is published only when every socket reports it, gap-free; the Grace + # meter must be gap-free on every socket or the whole leg is invalid. + families = [SENSOR_GRACE] + [ + rail + for rail in RAIL_COLUMNS.values() + if all(rail in by_family for by_family in series.values()) + ] start, end = window energy: dict[str, float] = {} dropped_rails: set[str] = set() - for (host, socket), by_kind in sorted(series.items()): - for kind, family in feeds.items(): - label = f"{host}/socket{socket}/{kind}" - samples = sorted(by_kind[kind].items()) - sequence = _bracketing_sequence(tuple(t for t, _ in samples), start, end) - gap = ( - None - if sequence is None - else max( - (later - earlier for earlier, later in itertools.pairwise(sequence)), - default=0.0, - ) - ) + for (host, socket), by_family in sorted(series.items()): + for family in families: + samples = sorted(by_family[family].items()) + label = f"{host}/socket{socket}/{family}" + gap = _max_sample_gap(samples, start, end) if gap is not None: audit.per_series_max_sample_gap_s[label] = gap if gap is None or gap > MAX_SAMPLE_GAP_SECONDS: - if family in _RAIL_COLUMNS.values(): + if family != SENSOR_GRACE: dropped_rails.add(family) elif gap is None: audit.invalidate("cpu_window_not_bracketed") @@ -350,19 +298,17 @@ def validate_cpu_leg( if audit.reason_codes: audit.per_series_energy_j = {} return audit - for family in dropped_rails: - energy.pop(family, None) duration = end - start - metrics: dict[str, float] = {} - if "cpu" in energy: - metrics["avg_cpu_socket_power_w"] = energy["cpu"] / duration / len(expected_keys) - metrics["avg_total_cpu_power_w"] = energy["cpu"] / duration - metrics["total_cpu_energy_j"] = energy["cpu"] - for family in ("module", *_RAIL_COLUMNS.values()): - if family in energy: - metrics[f"avg_total_{family}_power_w"] = energy[family] / duration - metrics[f"total_{family}_energy_j"] = energy[family] + metrics = { + "avg_cpu_socket_power_w": energy[SENSOR_GRACE] / duration / len(expected_keys), + "avg_total_cpu_power_w": energy[SENSOR_GRACE] / duration, + "total_cpu_energy_j": energy[SENSOR_GRACE], + } + for rail in RAIL_COLUMNS.values(): + if rail in energy and rail not in dropped_rails: + metrics[f"avg_total_{rail}_power_w"] = energy[rail] / duration + metrics[f"total_{rail}_energy_j"] = energy[rail] if any(not math.isfinite(value) for value in metrics.values()): audit.invalidate("non_finite_power_metric") return audit diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 880811fe7c..064c128491 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -29,8 +29,8 @@ and aggregate power metrics remain blocked. Grace CPU-side leg: when the package also carries ``power/cpu/`` (see -:mod:`.cpu_side`), each socket's Grace-side (and, when exposed, whole-module) -power is integrated over the same bound formal window and published under +:mod:`.cpu_side`), each socket's Grace socket power and its ACPI rails are +integrated over the same bound formal window and published under ``cpu_power_valid``. That verdict is independent of ``power_valid``. A missing or invalid CPU leg fails the run only with ``require_power`` and an ``expected_cpu_source``; otherwise it never changes the exit code. diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index fb0b8d3101..a8b658d9cd 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -755,42 +755,30 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) # --- Grace CPU-side power leg (power/cpu/) ----------------------------------- CPU_HOSTS = tuple(sorted({host for host, *_ in DEVICES})) -CPU_HEADER_V1 = [ - "schema_version", "timestamp_unix", "hostname", "source", "sensor", - "socket_id", "power_w", "total_power_w", +CPU_HEADER = [ + "schema_version", "timestamp_unix", "hostname", "source", "sensor", "socket_id", + "power_w", "cpu_rail_w", "soc_w", "dram_w", "total_power_w", ] -CPU_HEADER_V2 = CPU_HEADER_V1[:7] + ["cpu_rail_w", "soc_w", "dram_w", "total_power_w"] GRACE_W = {"node-d": 500.0, "node-p": 700.0} -MODULE_W = 1500.0 DCGM_W = 300.0 # Component rails never feed a headline metric; the values are chosen so any -# leak into a headline key is visible against the Grace and module constants. -RAILS_W = {"cpu_rail": 200.0, "soc": 50.0, "dram": 30.0} -GRACE_KINDS = {"grace": None, **RAILS_W} - -_SENSORS = { - # (source, v1 firmware OEM label, v2 collector sensor name) - "grace": ("acpi", "Grace Power Socket {s}", "CPU{s}:cpuSidePowerUsageW"), - "cpu_rail": ("acpi", "CPU Power Socket {s}", "CPU{s}:cpuRailPowerUsageW"), - "soc": ("acpi", "SysIO Power Socket {s}", "CPU{s}:socPowerUsageW"), - "dram": ("acpi", "DRAM Power Socket {s}", "CPU{s}:dramPowerUsageW"), - "module": ("acpi", "Module Power Socket {s}", "Module Power Socket {s}"), - "dcgm": ("dcgm", "CPU{s}:cpuPowerUsageW", "CPU{s}:cpuPowerUsageW"), -} +# leak into a headline key is visible against the Grace constants. +RAILS_W = {"cpu_rail": 200.0, "soc": 50.0} +# srt-slurm cpu_rails.SENSOR_SUFFIXES: the Grace socket meter and the DCGM CPU rail. +GRACE_SENSOR = "CPU{s}:cpuSidePowerUsageW" +DCGM_SENSOR = "CPU{s}:cpuPowerUsageW" -def _cpu_row(ts, host, socket, kind, watts, fmt): - source, label_v1, label_v2 = _SENSORS[kind] - sensor = (label_v1 if fmt == "v1" else label_v2).format(s=socket) - if fmt == "v1": - return [1, repr(ts), host, source, sensor, socket, repr(watts), ""] - rails = ["", "", ""] if source == "dcgm" else [repr(w) for w in RAILS_W.values()] - return [2, repr(ts), host, source, sensor, socket, repr(watts), *rails, ""] +def _cpu_row(ts, host, socket, kind, watts): + if kind == "dcgm": + return [2, repr(ts), host, "dcgm", DCGM_SENSOR.format(s=socket), socket, repr(watts), "", "", "", ""] + rails = [repr(w) for w in RAILS_W.values()] + return [2, repr(ts), host, "acpi", GRACE_SENSOR.format(s=socket), socket, repr(watts), *rails, "", ""] -def _cpu_rows(kinds_by_host=None, fmt="v1", *, hosts=CPU_HOSTS): +def _cpu_rows(kinds_by_host=None, *, hosts=CPU_HOSTS): """One row per (scrape, host, socket, kind); ``None`` watts means the Grace constant.""" - kinds_by_host = kinds_by_host or {host: GRACE_KINDS for host in hosts} + kinds_by_host = kinds_by_host or {host: {"grace": None} for host in hosts} rows = [] ts = FIRST_TS while ts <= LAST_TS: @@ -798,17 +786,17 @@ def _cpu_rows(kinds_by_host=None, fmt="v1", *, hosts=CPU_HOSTS): for socket in (0, 1): for kind, watts in kinds_by_host.get(host, {}).items(): watts = GRACE_W[host] if watts is None else watts - rows.append(_cpu_row(ts, host, socket, kind, watts, fmt)) + rows.append(_cpu_row(ts, host, socket, kind, watts)) ts += 1.0 return rows -def add_cpu_package(pkg, rows, header=CPU_HEADER_V1, manifest_text=None): +def add_cpu_package(pkg, rows, manifest_text=None): cpu_dir = pkg.power_dir / "cpu" cpu_dir.mkdir() with open(cpu_dir / "samples.csv", "w", newline="") as handle: writer = csv.writer(handle) - writer.writerow(header) + writer.writerow(CPU_HEADER) writer.writerows(rows) if manifest_text is None: manifest_text = json.dumps( @@ -852,8 +840,8 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path): from infx.results.power.audit import audit_summary pkg = build_package(tmp_path) - rows = _cpu_rows({host: {"grace": None} for host in CPU_HOSTS}, "v2") - add_cpu_package(pkg, rows, CPU_HEADER_V2) + rows = _cpu_rows() + add_cpu_package(pkg, rows) assert pkg.run(require_power=True) == 0 agg = pkg.agg() @@ -889,46 +877,17 @@ def test_grace_total_emits_cpu_keys_and_provenance(self, tmp_path): def test_rail_gap_drops_only_that_rail(self, tmp_path): pkg = build_package(tmp_path) - rows = _cpu_rows({host: {"grace": None} for host in CPU_HOSTS}, "v2") + rows = _cpu_rows() for row in rows: if row[2] == "node-d" and row[5] == 0 and 1020.0 <= float(row[1]) <= 1030.0: row[7] = "" - add_cpu_package(pkg, rows, CPU_HEADER_V2) + add_cpu_package(pkg, rows) assert pkg.run(require_power=True) == 0 agg = pkg.agg() assert_grace_keys(agg) assert "avg_total_cpu_rail_power_w" not in agg assert agg["avg_total_cpu_sysio_power_w"] == 200.0 - @pytest.mark.parametrize( - "kinds, expected_kind, expected_power, module_power", - [ - ({host: {"dcgm": DCGM_W} for host in CPU_HOSTS}, "dcgm_cpu_rail", 1200.0, None), - ( - {host: {"grace": None, "module": MODULE_W} for host in CPU_HOSTS}, - "module", 2400.0, 6000.0, - ), - ( - {"node-d": {"grace": None, "module": MODULE_W}, "node-p": {"grace": None}}, - "grace_socket", 2400.0, None, - ), - ], - ) - def test_cpu_source_selection( - self, tmp_path, kinds, expected_kind, expected_power, module_power - ): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows(kinds)) - assert pkg.run() == 0 - agg = pkg.agg() - assert agg["cpu_power_valid"] == 1 - assert agg["avg_total_cpu_power_w"] == expected_power - assert pkg.sidecar()["cpu"]["sensor_kind"] == expected_kind - if module_power is None: - assert "avg_total_module_power_w" not in agg - else: - assert agg["avg_total_module_power_w"] == module_power - @pytest.mark.parametrize( "cpu, require_power", [("missing", True), ("complete", True), ("missing", False), ("dcgm_only", True)], @@ -953,7 +912,8 @@ def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( assert set(apm.CPU_METRIC_KEYS).isdisjoint(pkg.agg()) assert pkg.sidecar()["cpu"]["reason_codes"] if cpu == "dcgm_only": - assert "cpu_sensor_source_mismatch" in pkg.sidecar()["cpu"]["reason_codes"] + # The DCGM CPU rail is not the Grace socket meter: no socket series. + assert pkg.sidecar()["cpu"]["reason_codes"] == ["cpu_socket_count_mismatch"] def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): pkg = build_package(tmp_path) @@ -982,7 +942,7 @@ def tampered(self, request, tmp_path): elif kind == "mixed": rows = _cpu_rows({"node-d": {"grace": None}, "node-p": {"dcgm": DCGM_W}}) elif kind == "malformed": - rows.append([1, repr(FIRST_TS), "node-d", "acpi", "Grace Power Socket 0", 0, "n/a", ""]) + rows.append(_cpu_row(FIRST_TS, "node-d", 0, "grace", "n/a")) add_cpu_package(pkg, rows) return pkg @@ -991,7 +951,7 @@ def tampered(self, request, tmp_path): [ ("socket", "cpu_socket_count_mismatch"), ("gap", "cpu_sample_gap_exceeded"), - ("mixed", "cpu_sensor_kind_mixed"), + ("mixed", "cpu_socket_count_mismatch"), ("malformed", "cpu_samples_malformed"), ], indirect=["tampered"], @@ -1008,82 +968,3 @@ def test_cpu_leg_failures_leave_gpu_fields_untouched(self, tmp_path, tampered, r assert cpu["cpu_power_valid"] is False assert reason in cpu["reason_codes"] assert pkg.sidecar()["power_valid"] is True - - @pytest.mark.parametrize("gpu_failure", ["coverage", "producer_pin"]) - def test_cpu_leg_survives_an_invalid_gpu_leg(self, tmp_path, gpu_failure): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows()) - if gpu_failure == "coverage": - _rewrite_samples( - pkg, - lambda body: [ - row for row in body - if not (row[3] == "node-p" and row[4] == "0" and 20 <= int(row[2]) <= 24) - ], - ) - assert pkg.run(sha="b" * 40 if gpu_failure == "producer_pin" else PRODUCER_SHA) == 0 - agg = pkg.agg() - assert agg["power_valid"] == 0 - assert "total_gpu_energy_j" not in agg - assert_grace_keys(agg) - sidecar = pkg.sidecar() - gpu_reason = ( - "producer_commit_mismatch" if gpu_failure == "producer_pin" - else "package_recompute_invalid" - ) - assert gpu_reason in sidecar["reasons"] - assert sidecar["cpu"]["reason_codes"] == [] - - def test_result_without_window_invalidates_cpu(self, tmp_path): - pkg = build_package(tmp_path) - add_cpu_package(pkg, _cpu_rows()) - result = dict(BENCH_FIELDS, max_concurrency=8) - pkg.original_result.write_text(json.dumps(result)) - pkg.bench_result.write_text(json.dumps(result)) - assert pkg.run() == 0 - assert pkg.agg()["cpu_power_valid"] == 0 - assert "cpu_window_unavailable" in pkg.sidecar()["cpu"]["reason_codes"] - - -def test_host_power_preserves_heterogeneous_hosts_inside_one_role(tmp_path): - pkg = build_package(tmp_path) - manifest_path = pkg.power_dir / "manifest.json" - manifest = json.loads(manifest_path.read_text()) - for device in manifest["expected_devices"]: - for assignment in device["assignments"]: - assignment["worker_role"] = "agg" - assignment["het_group"] = None - manifest_path.write_text(json.dumps(manifest)) - assert pkg.run(prefill_gpus=0, decode_gpus=0, aggregate_gpus=4) == 0 - assert pkg.agg()["workers"] == [ - {"role": "agg", "worker_idx": 0, "hosts": ["node-d"], - "num_gpus": 2, "avg_power_w": 300.0}, - {"role": "agg", "worker_idx": 1, "hosts": ["node-p"], - "num_gpus": 2, "avg_power_w": 400.0}, - ] - - -def test_host_power_uses_timestamp_window_when_reported_duration_is_skewed(tmp_path): - pkg = build_package(tmp_path, bench_extra={"duration": 60.1}) - window_path = pkg.windows_dir / f"{RESULT_STEM}.json" - window = json.loads(window_path.read_text()) - window["duration"] = 60.1 - window_path.write_text(json.dumps(window)) - - assert pkg.run() == 0 - workers = pkg.agg()["workers"] - assert [(worker["role"], worker["avg_power_w"]) for worker in workers] == [ - ("decode", 300.0), ("prefill", 400.0) - ] - - -@pytest.mark.parametrize("temperature", ["nan", "inf", "bad", "-300", "200.5", "9223372036854775794"]) -def test_v3_samples_reject_invalid_temperature(tmp_path, temperature): - path = tmp_path / "samples.csv" - path.write_text( - "schema_version,timestamp_unix,scrape_seq,hostname,gpu_index,gpu_uuid,power_w,gpu_util_pct,sm_active,temperature_c\n" - f"3,1000,0,node-a,0,GPU-a,400,,,{temperature}\n" - ) - rows, reasons = apm.read_samples(path) - assert rows == () - assert reasons == ("samples_csv_malformed",) From 847227bdd6d1070cc7176527ca32042ef2d952cf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:24:59 -0700 Subject: [PATCH 23/35] refactor(power): expect the CPU leg from cpu-power instead of re-reading the bound recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The point's cpu-power switch already decides whether the exporter is bound (PowerDecision.cpu), so the launcher no longer re-reads the recipe or writes POWER_EXPECTED_CPU_SOURCE to GITHUB_ENV, and the processors take --expected-cpu-power or the workflow's CPU_POWER. The source check goes with it: the cluster record requires a source (clusters/slurm.py CpuPowerExporter), the binder always writes it, and the Grace leg accepts only its own sensor. 中文：CPU leg 是否必需改由 cpu-power 开关决定；launcher 不再回读 recipe 或写 GITHUB_ENV，处理器改用 --expected-cpu-power 或 workflow 的 CPU_POWER。cluster 记录必须声明 source 且 binder 总会写入，故删除多余的 source 核对。 --- .../workflows/benchmark-multinode-tmpl.yml | 1 - inferencex-e2e/infx/launch/artifacts.py | 11 +++++----- .../infx/launch/drivers/srt/__init__.py | 4 ---- .../infx/launch/drivers/srt/collect.py | 4 ++-- .../infx/launch/drivers/srt/power.py | 17 +------------- .../infx/results/agentic/power_adapter.py | 8 +++---- inferencex-e2e/infx/results/fixed_sequence.py | 2 +- inferencex-e2e/infx/results/power/cpu_side.py | 13 ++++++----- .../infx/results/power/multinode.py | 22 ++++++++----------- .../tests/launch/test_launch_artifacts.py | 2 +- .../infx/tests/launch/test_launch_policy.py | 21 +----------------- .../infx/tests/launch/test_srt_driver.py | 6 ++--- .../power/test_aggregate_power_multinode.py | 6 ++--- 13 files changed, 37 insertions(+), 80 deletions(-) diff --git a/.github/workflows/benchmark-multinode-tmpl.yml b/.github/workflows/benchmark-multinode-tmpl.yml index 190098325e..bd50b498e9 100644 --- a/.github/workflows/benchmark-multinode-tmpl.yml +++ b/.github/workflows/benchmark-multinode-tmpl.yml @@ -334,7 +334,6 @@ jobs: fi export EVAL_CONC export IS_MULTINODE=true - echo "POWER_EXPECTED_CPU_SOURCE=" >> "$GITHUB_ENV" "$INFERENCEX_LAUNCH_PYTHON" -m infx.launch run if [ "${EVAL_ONLY}" = "true" ]; then echo "Eval-only mode: skipping benchmark result file check" diff --git a/inferencex-e2e/infx/launch/artifacts.py b/inferencex-e2e/infx/launch/artifacts.py index ec0d774e5f..08f2bd3729 100644 --- a/inferencex-e2e/infx/launch/artifacts.py +++ b/inferencex-e2e/infx/launch/artifacts.py @@ -153,7 +153,7 @@ def collect_agentic_power_results( concurrencies: Sequence[int], *, results_python: str | None, - expected_cpu_source: str | None = None, + expected_cpu: bool = False, ) -> int: """Stage the AgentX power audit inputs and validate each concurrency's power window. @@ -176,7 +176,7 @@ def collect_agentic_power_results( if step := validate_agentic_power( logs_dir, workspace, result_filename, producer_sha, concurrencies, - results_python=results_python, require_power=True, expected_cpu_source=expected_cpu_source, + results_python=results_python, require_power=True, expected_cpu=expected_cpu, ): # fmt: skip rc = step return rc @@ -191,11 +191,12 @@ def validate_agentic_power( *, results_python: str | None, require_power: bool, - expected_cpu_source: str | None = None, + expected_cpu: bool = False, ) -> int: """Run the AgentX power adapter for every concurrency; return the last failing code. - With ``require_power`` a concurrency without a valid power window fails. + With ``require_power`` a concurrency without a valid power window fails, and with + ``expected_cpu`` one without a valid Grace CPU leg fails too. """ logs_dir = logs_dir.resolve() workspace = workspace.resolve() @@ -211,7 +212,7 @@ def validate_agentic_power( workspace / f"{result_filename}_conc{concurrency}.json", producer_sha, "--audit-source", f"LOGS/agentic/conc_{concurrency}/power_validation.json", - *(["--expected-cpu-source", expected_cpu_source] if expected_cpu_source else []), + *(["--expected-cpu-power"] if expected_cpu else []), require_power=require_power, ): # fmt: skip rc = step diff --git a/inferencex-e2e/infx/launch/drivers/srt/__init__.py b/inferencex-e2e/infx/launch/drivers/srt/__init__.py index 0b94c18513..9ece0ff26c 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/__init__.py +++ b/inferencex-e2e/infx/launch/drivers/srt/__init__.py @@ -174,10 +174,6 @@ def run_multinode(launch: Launch) -> int: ) if rc := submit.bind_recipe(run, checkout, staged, recipe, power_port, cpu_source, client_env): return rc - decision = power.with_cpu_source(decision, Path(recipe)) - if github_env := request.env.get("GITHUB_ENV"): - with Path(github_env).open("a") as handle: - handle.write(f"POWER_EXPECTED_CPU_SOURCE={decision.expected_cpu_source or ''}\n") prepare_recipe(Path(recipe), srtctl_job_name(request.runner_name), run.srt.dist_timeout_s) arguments = submit.multinode_arguments(run, lane, recipe, overrides, preflight=preflight) manifest = run.workspace / submit.MULTINODE_SUBMISSION diff --git a/inferencex-e2e/infx/launch/drivers/srt/collect.py b/inferencex-e2e/infx/launch/drivers/srt/collect.py index 9742f33c55..cc2b96993e 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/collect.py +++ b/inferencex-e2e/infx/launch/drivers/srt/collect.py @@ -187,7 +187,7 @@ def stage_logs_on_exit() -> None: infmax, *audit, results_python=python, - expected_cpu_source=power.expected_cpu_source, + expected_cpu=power.cpu, ) else: power_rc = validate_agentic_power( @@ -195,7 +195,7 @@ def stage_logs_on_exit() -> None: *audit, results_python=python, require_power=request.require_power, - expected_cpu_source=power.expected_cpu_source, + expected_cpu=power.cpu, ) if power_rc: print( diff --git a/inferencex-e2e/infx/launch/drivers/srt/power.py b/inferencex-e2e/infx/launch/drivers/srt/power.py index 1106834ff4..c0cda8c7a7 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/power.py +++ b/inferencex-e2e/infx/launch/drivers/srt/power.py @@ -6,12 +6,9 @@ from __future__ import annotations -from dataclasses import dataclass, replace -from pathlib import Path +from dataclasses import dataclass from typing import TYPE_CHECKING -import yaml - from infx.launch.context import LaunchError from infx.launch.policy import LaunchPath, Match, any_of @@ -143,7 +140,6 @@ class PowerDecision: require: bool = False # The point selected cpu-power: the cluster's Grace CPU exporter is bound and expected. cpu: bool = False - expected_cpu_source: str | None = None NO_POWER = PowerDecision(dcgm=False, agentx=False) @@ -172,14 +168,3 @@ def decide_power(cluster_id: str, path: LaunchPath, request: MultiNodeRequest) - ) message = lane.agentic_error if request.is_agentic and lane.agentic_error else lane.error raise PowerPolicyError(message) - - -def with_cpu_source(decision: PowerDecision, recipe: Path) -> PowerDecision: - """Read the CPU source from the fully bound recipe used for this job.""" - if not decision.dcgm: - return decision - bound = yaml.safe_load(recipe.read_text()) - telemetry = bound.get("telemetry", {}) - cpu = telemetry.get("cpu_power_exporter", {}) - source = cpu.get("source") if isinstance(cpu, dict) else None - return replace(decision, expected_cpu_source=source if source in {"acpi", "dcgm"} else None) diff --git a/inferencex-e2e/infx/results/agentic/power_adapter.py b/inferencex-e2e/infx/results/agentic/power_adapter.py index 941e4c151e..0cb4cca17f 100644 --- a/inferencex-e2e/infx/results/agentic/power_adapter.py +++ b/inferencex-e2e/infx/results/agentic/power_adapter.py @@ -355,7 +355,7 @@ def run_native_agentic_power( logs_root: Path, expected_producer_sha: str, require_power: bool = False, - expected_cpu_source: str | None = None, + expected_cpu: bool = False, audit_source: str | None = None, validation_result: Path | None = None, ) -> int: @@ -416,7 +416,7 @@ def run_native_agentic_power( logs_root=logs_root, validation_result=validation_result, require_power=require_power, - expected_cpu_source=expected_cpu_source, + expected_cpu=expected_cpu, ) if audit_source is not None: try: @@ -449,7 +449,7 @@ def main() -> int: action="store_true", default=os.environ.get("REQUIRE_POWER", "").lower() in {"1", "true", "yes"}, ) - parser.add_argument("--expected-cpu-source", choices=("acpi", "dcgm")) + parser.add_argument("--expected-cpu-power", action="store_true") args = parser.parse_args() if args.multinode_contract_missing: if args.agg_result is None: @@ -502,7 +502,7 @@ def main() -> int: logs_root=args.logs_root, expected_producer_sha=args.expected_producer_sha, require_power=args.require_power, - expected_cpu_source=args.expected_cpu_source, + expected_cpu=args.expected_cpu_power, audit_source=args.audit_source, validation_result=args.validation_result, ) diff --git a/inferencex-e2e/infx/results/fixed_sequence.py b/inferencex-e2e/infx/results/fixed_sequence.py index 733e4f6abe..53ac1ce670 100644 --- a/inferencex-e2e/infx/results/fixed_sequence.py +++ b/inferencex-e2e/infx/results/fixed_sequence.py @@ -279,7 +279,7 @@ def aggregate_power_result( aggregate_gpus=aggregate_gpus, expected_producer_sha=env.get("POWER_PRODUCER_SHA") or None, logs_root=Path(env.get("POWER_RESULT_ROOT", "LOGS")), - expected_cpu_source=env.get("POWER_EXPECTED_CPU_SOURCE") or None, + expected_cpu=env.get("CPU_POWER") == "1", validation_result=validation_path, require_power=require_power, ) diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index 23d8baf112..ce91d75955 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -231,11 +231,15 @@ def validate_cpu_leg( *, window: tuple[float, float] | None, expected_hosts: Iterable[str], - expected_source: str | None = None, + expected: bool = False, ) -> CpuPowerAudit | None: - """Validate and integrate the CPU sub-package; None when the run carried none.""" + """Validate and integrate the CPU sub-package; None when the run carried none. + + ``expected`` is the point's ``cpu-power`` switch: a missing sub-package is + then a verdict rather than silence. + """ if not cpu_dir.is_dir(): - if expected_source is None: + if not expected: return None audit = CpuPowerAudit() audit.invalidate("cpu_artifacts_missing") @@ -263,9 +267,6 @@ def validate_cpu_leg( return audit audit.sensor_kind = SENSOR_GRACE audit.source = next(iter(sources)) - if expected_source is not None and audit.source != expected_source: - audit.invalidate("cpu_sensor_source_mismatch") - return audit # A rail is published only when every socket reports it, gap-free; the Grace # meter must be gap-free on every socket or the whole leg is invalid. diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 064c128491..02ee33e6de 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -32,8 +32,8 @@ :mod:`.cpu_side`), each socket's Grace socket power and its ACPI rails are integrated over the same bound formal window and published under ``cpu_power_valid``. That verdict is independent of ``power_valid``. A missing -or invalid CPU leg fails the run only with ``require_power`` and an -``expected_cpu_source``; otherwise it never changes the exit code. +or invalid CPU leg fails the run only with ``require_power`` and +``expected_cpu``; otherwise it never changes the exit code. """ from __future__ import annotations @@ -922,7 +922,7 @@ def validate_and_integrate( decode_gpus: int, aggregate_gpus: int, expected_producer_sha: str | None, - expected_cpu_source: str | None = None, + expected_cpu: bool = False, ) -> MultinodePowerAudit: """Recompute package validity, cross-check verdicts, and integrate energy. @@ -948,7 +948,7 @@ def validate_and_integrate( power_dir / CPU_DIRNAME, window=window, expected_hosts=audit.expected_worker_hosts, - expected_source=expected_cpu_source, + expected=expected_cpu, ) return audit @@ -1424,7 +1424,7 @@ def run( aggregate_gpus: int = 0, validation_result: Path | None = None, require_power: bool = False, - expected_cpu_source: str | None = None, + expected_cpu: bool = False, ) -> int: """Validate multinode power artifacts, patch metrics, and audit the verdict.""" validation_result = validation_result or bench_result.with_name( @@ -1442,7 +1442,7 @@ def run( decode_gpus=decode_gpus, aggregate_gpus=aggregate_gpus, expected_producer_sha=expected_producer_sha, - expected_cpu_source=expected_cpu_source, + expected_cpu=expected_cpu, ) for reason in benchmark_reasons: _add_reason(audit, reason) @@ -1491,11 +1491,7 @@ def run( f"reasons={','.join(cpu.reason_codes) or '-'}" ) - if ( - require_power - and expected_cpu_source is not None - and (audit.cpu is None or not audit.cpu.valid) - ): + if require_power and expected_cpu and (audit.cpu is None or not audit.cpu.valid): print("[aggregate_power_multinode] Required CPU power is invalid", file=sys.stderr) return 1 @@ -1579,7 +1575,7 @@ def main() -> int: default=os.environ.get("REQUIRE_POWER", "").lower() in {"1", "true", "yes"}, help="Fail invalid GPU power and any expected CPU source (also REQUIRE_POWER=1)", ) - parser.add_argument("--expected-cpu-source", choices=("acpi", "dcgm")) + parser.add_argument("--expected-cpu-power", action="store_true") args = parser.parse_args() return run( args.power_dir, @@ -1592,7 +1588,7 @@ def main() -> int: logs_root=args.logs_root, validation_result=args.validation_result, require_power=args.require_power, - expected_cpu_source=args.expected_cpu_source, + expected_cpu=args.expected_cpu_power, ) diff --git a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py index d18d4cba38..b0e34fbef6 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_artifacts.py @@ -156,7 +156,7 @@ def test_agentic_collection_requires_declared_cpu_and_keeps_gpu_valid(tmp_path): PRODUCER_SHA, [4], results_python=sys.executable, - expected_cpu_source="acpi", + expected_cpu=True, ) == 1 result = json.loads((workspace / "point_conc4.json").read_text()) assert result["power_valid"] == 1 diff --git a/inferencex-e2e/infx/tests/launch/test_launch_policy.py b/inferencex-e2e/infx/tests/launch/test_launch_policy.py index 6bb2182824..9158afbee8 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_policy.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_policy.py @@ -10,12 +10,7 @@ from infx.launch.context import LaunchError from infx.launch.drivers import check_tables from infx.launch.drivers.srt import models, power -from infx.launch.drivers.srt.power import ( - PowerDecision, - PowerPolicyError, - decide_power, - with_cpu_source, -) +from infx.launch.drivers.srt.power import PowerPolicyError, decide_power from infx.launch.lifecycle import Lifecycle from infx.launch.policy import LaunchPath, Match, salloc_time_limit from infx.launch.request import LaunchRequest, MultiNodeRequest @@ -103,20 +98,6 @@ def test_gb200_reports_agentic_misses_distinctly(): decide_power("gb200-nv", MULTI, request) -@pytest.mark.parametrize( - ("source", "expected"), - [("acpi", "acpi"), ("dcgm", "dcgm"), (None, None)], -) -def test_power_cpu_source_comes_from_bound_recipe(tmp_path, source, expected): - recipe = tmp_path / "bound.yaml" - cpu = f" cpu_power_exporter:\n source: {source}\n" if source else "" - recipe.write_text(f"telemetry:\n enabled: true\n{cpu}") - decision = PowerDecision(dcgm=True, agentx=True, require=True) - resolved = with_cpu_source(decision, recipe) - assert resolved.expected_cpu_source == expected - assert resolved.require is True - - def test_a_power_lane_needs_its_clusters_exporter_port(monkeypatch): labels = {"cluster:c": ["c_0"]} srt = {"network-interface": ""} diff --git a/inferencex-e2e/infx/tests/launch/test_srt_driver.py b/inferencex-e2e/infx/tests/launch/test_srt_driver.py index 0632aea1fd..54feef3268 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_driver.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_driver.py @@ -677,7 +677,6 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey build_package, ) - github_env = harness.tmp / "github-env" env = lane_env( harness, "gb300-nv", @@ -689,7 +688,6 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey REQUIRE_POWER="1", POWER="1", CPU_POWER="1", - GITHUB_ENV=str(github_env), ) assert_ok(launch(env, harness.config, harness.workspace)) [checkout] = harness.workspace.glob("srt-slurm-9001-*") @@ -698,11 +696,11 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey assert bound["telemetry"]["dcgm_exporter"]["port"] == 19401 [setup] = [call for call in lines(harness.logs, "make") if call.startswith("setup ")] assert "CPU_POWER_EXPORTER_RELEASE=v2.51.1" in setup.split() - exported = dict(line.split("=", 1) for line in github_env.read_text().splitlines()) pkg = build_package(harness.tmp / "power-input") monkeypatch.chdir(harness.tmp) + # The workflow exports CPU_POWER once; the launcher and the processor both read it. result_env = { - **exported, + "CPU_POWER": "1", "REQUIRE_POWER": "1", "IS_MULTINODE": "true", "RESULT_FILENAME": "fixture", diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index a8b658d9cd..0f2d9ff258 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -61,7 +61,7 @@ def run( aggregate_gpus=0, sha=PRODUCER_SHA, require_power=False, - expected_cpu_source=None, + expected_cpu=False, ): return apm.run( self.power_dir, @@ -74,7 +74,7 @@ def run( logs_root=self.logs_root, validation_result=self.validation_result, require_power=require_power, - expected_cpu_source=expected_cpu_source, + expected_cpu=expected_cpu, ) def agg(self): @@ -900,7 +900,7 @@ def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( add_cpu_package(pkg, _cpu_rows()) elif cpu == "dcgm_only": add_cpu_package(pkg, _cpu_rows({host: {"dcgm": DCGM_W} for host in CPU_HOSTS})) - assert pkg.run(require_power=require_power, expected_cpu_source="acpi") == int( + assert pkg.run(require_power=require_power, expected_cpu=True) == int( require_power and cpu != "complete" ) assert _gpu_fields(pkg.agg()) == _reference_agg(tmp_path) From b7754322d7564cd1e46c83c95b87bfb2ad793558 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:28:54 -0700 Subject: [PATCH 24/35] refactor(power): share the sample-gap limit and drop the dead window-trust field MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CPU leg now asks _window_contract_holds directly on the serialized window instead of carrying formal_window_trusted across the audit, MAX_SAMPLE_GAP_SECONDS has one definition in common.py, and patch_power_metrics goes because _patch_agg became its only caller's replacement on this branch. 中文：CPU leg 直接用序列化窗口调用 _window_contract_holds，不再在 audit 上携带 formal_window_trusted；MAX_SAMPLE_GAP_SECONDS 只在 common.py 定义一处；删除在本分支上已无调用方的 patch_power_metrics。 --- inferencex-e2e/infx/results/power/common.py | 32 ++++--------------- inferencex-e2e/infx/results/power/cpu_side.py | 10 ++++-- .../infx/results/power/multinode.py | 16 ++++------ 3 files changed, 19 insertions(+), 39 deletions(-) diff --git a/inferencex-e2e/infx/results/power/common.py b/inferencex-e2e/infx/results/power/common.py index 9b15add50e..aaf191fa27 100644 --- a/inferencex-e2e/infx/results/power/common.py +++ b/inferencex-e2e/infx/results/power/common.py @@ -7,12 +7,10 @@ import json import math import sys -from collections.abc import Iterable, Mapping +from collections.abc import Mapping from dataclasses import dataclass from pathlib import Path -from . import POWER_METRIC_SCHEMA_VERSION, with_power_metrics - @dataclass(frozen=True) class BenchmarkData: @@ -30,6 +28,11 @@ def integration_duration_s(self) -> float: return self.end_unix - self.start_unix +# Widest gap between consecutive samples inside the integration window that +# still counts as continuous coverage (srt-slurm contract.MAX_SAMPLE_GAP_SECONDS). +MAX_SAMPLE_GAP_SECONDS = 3.0 + + def _append_reason(reasons: list[str], reason: str) -> None: """Append a validation reason once while preserving discovery order.""" if reason not in reasons: @@ -263,26 +266,3 @@ def audit_metrics(metrics: Mapping[str, float | None]) -> dict[str, float]: for key, value in metrics.items() if value is not None and math.isfinite(value) } - - -def patch_power_metrics( - path: Path, - *, - metric_keys: Iterable[str], - power_valid: bool, - metrics: Mapping[str, float], - cpu_power_valid: bool | None = None, - cpu_metrics: Mapping[str, float] | None = None, -) -> None: - """Validate replacement metrics before atomically updating an aggregate.""" - data = json.loads(path.read_text(encoding="utf-8")) - data = with_power_metrics( - data, - metric_keys=metric_keys, - schema_version=POWER_METRIC_SCHEMA_VERSION, - power_valid=power_valid, - metrics=metrics, - cpu_power_valid=cpu_power_valid, - cpu_metrics=cpu_metrics, - ) - _write_json_atomic(path, data) diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index ce91d75955..78df481111 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -27,7 +27,13 @@ from pathlib import Path from typing import Any -from .common import _append_reason, _bracketing_sequence, _integrate_device, audit_metrics +from .common import ( + MAX_SAMPLE_GAP_SECONDS, + _append_reason, + _bracketing_sequence, + _integrate_device, + audit_metrics, +) CPU_DIRNAME = "cpu" CPU_SAMPLES_FILENAME = "samples.csv" @@ -51,8 +57,6 @@ # One GB200/GB300 compute tray per worker host, two Grace sockets per tray. SOCKETS_PER_HOST = 2 -# Shared with the GPU leg (srt-slurm contract.MAX_SAMPLE_GAP_SECONDS). -MAX_SAMPLE_GAP_SECONDS = 3.0 # The headline meter: "Grace Power Socket N", the whole socket (CPU rail, SysIO, # LPDDR5X and regulator loss). srt-slurm names its wide-format rows diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 02ee33e6de..4584d30a4c 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -59,6 +59,7 @@ with_power_metrics, ) from .common import ( + MAX_SAMPLE_GAP_SECONDS, BenchmarkData, _append_reason, _bracketing_sequence, @@ -94,7 +95,6 @@ SAMPLES_HEADER_V3 = (*SAMPLES_HEADER_V2, "temperature_c") SAMPLES_HEADERS = {1: SAMPLES_HEADER, 2: SAMPLES_HEADER_V2, 3: SAMPLES_HEADER_V3} -MAX_SAMPLE_GAP_SECONDS = 3.0 MAX_TEMPERATURE_C = 200.0 WORKER_ROLES = ("prefill", "decode", "agg") @@ -893,9 +893,8 @@ class MultinodePowerAudit: observed_gpu_count: int = 0 expected_gpu_count: int = 0 metrics: dict[str, float] = field(default_factory=dict) - # Inputs the CPU-side leg borrows from the GPU package; not serialized. + # Input the CPU-side leg borrows from the GPU package; not serialized. expected_worker_hosts: tuple[str, ...] = () - formal_window_trusted: bool = False # None when the package carried no cpu/ sub-package. cpu: CpuPowerAudit | None = None @@ -942,7 +941,7 @@ def validate_and_integrate( expected_producer_sha=expected_producer_sha, ) window = None - if audit.window is not None and audit.formal_window_trusted: + if audit.window is not None and _window_contract_holds(audit.window, audit.window_validations): window = (audit.window["start_time_unix"], audit.window["end_time_unix"]) audit.cpu = validate_cpu_leg( power_dir / CPU_DIRNAME, @@ -1118,9 +1117,6 @@ def _validate_gpu_leg( window = _select_window_for_result( audit, parsed_windows, logs_root, bench_result_path, benchmark ) - if window is not None: - audit.formal_window_trusted = _window_contract_holds(window, validations) - if ( not audit.package_integrity_valid or not topology_valid @@ -1295,12 +1291,12 @@ def _select_window_for_result( ) -def _window_contract_holds(window: ParsedWindow, validations: list[dict]) -> bool: +def _window_contract_holds(window: dict, validations: list[dict]) -> bool: """True when the bound window passed every check that is not GPU coverage.""" for validation in validations: if (validation["benchmark_type"], validation["concurrency"]) == ( - window.benchmark_type, - window.concurrency, + window["benchmark_type"], + window["concurrency"], ): return all(reason in _GPU_COVERAGE_REASONS for reason in validation["reason_codes"]) return False From c0a9c79cdc1a61885fd9539dd35dba50b84a935c Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:29:49 -0700 Subject: [PATCH 25/35] docs(power): describe the cpu-power switch and Grace sensor semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENT_OPERATIONS.md carries the Grace sensor table Oren asked for (whole-socket meter, CPU and SysIO rails, no module series) in place of the link to the deleted results-and-ingestion.md, and the #3296 changelog entry describes the final diff. 中文：AGENT_OPERATIONS.md 补充 Oren 要求的 Grace 传感器说明（整 socket 读数、CPU/SysIO rail、无 module 序列），替换指向已删除文档的链接；#3296 的 changelog 条目改为描述最终 diff。 --- .github/AGENT_OPERATIONS.md | 6 +++++- inferencex-e2e/perf-changelog.yaml | 4 ++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 1b0e2d356d..7f55db4908 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -72,7 +72,11 @@ Multinode srt-slurm results, and single-node fixed-sequence and AgentX results w Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count. -NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j`. They add `avg_total_module_power_w` and `total_module_energy_j` when the module sensor is exposed on every socket. They add `avg_total_cpu_rail_power_w` and `total_cpu_rail_energy_j` (ACPI CPU rail), and `avg_total_cpu_sysio_power_w` and `total_cpu_sysio_energy_j` (ACPI SysIO rail), each pair only when every Grace socket reports that rail without gaps. The rails are a non-additive breakdown inside the Grace socket total: never add them to it or subtract them from it. The CPU verdict is independent of `power_valid`. An invalid CPU leg fails the job only when power is required and the recipe declares `telemetry.cpu_power_exporter.source`; AgentX lanes whose launcher validates power always require it, and other lanes require it through `REQUIRE_POWER=1`; see [`inferencex-e2e/docs/results-and-ingestion.md`](../inferencex-e2e/docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72). +A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for those jobs only, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor. + +The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware, and the superchip module meter is unbound in ACPI; NVML exposes it (`nvidia-smi --query-gpu=module.power.draw.average`, srt-slurm #591 pending) and no package publishes it. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. + +The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`. Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 8ea6c89478..cc8ce2c3dc 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9731,7 +9731,7 @@ - fixed-seq-len - agentic-coding description: - - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes with the pinned cpu-power-exporter v2.51.1, matching the srt-slurm v2.51.1 source pin. Verify the release asset checksum and submit-time exporter compatibility; CPU-enabled jobs fail before sbatch if the exporter is missing, non-executable, or built for the wrong architecture." - - "Integrate CPU power over the formal GPU measurement window and emit cpu_power_valid and CPU-side metrics independently of the GPU power verdict, including the ACPI Grace socket total and its CPU-only and SysIO rails. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. During strict power qualification, require the recipe's declared CPU source and reject missing or invalid CPU totals; recipes without a declared source retain best-effort CPU behavior." + - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes. The master rows set cpu-power beside power; the binder merges the cluster's cpu-power-exporter record (ACPI on gb200-nv and gb300-nv) into the bound recipe, and make setup pins cpu-power-exporter v2.51.1 for those jobs, matching the srt-slurm v2.51.1 source pin." + - "Integrate each Grace socket total over the formal GPU measurement window and emit cpu_power_valid, avg_cpu_socket_power_w, avg_total_cpu_power_w, and total_cpu_energy_j independently of the GPU power verdict, with the ACPI CPU-only and SysIO rails as separate keys. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. Under required power a cpu-power row fails on a missing or invalid CPU leg; other rows keep the best-effort CPU verdict." - "Enable required DCGM telemetry with collect_interval_ms: 1000 on the Qwen3.5 GB200/GB300 8P1D recipes, and set require-power on both Qwen3.5 GB200/GB300 8k1k sequences. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296 From 0005144095b70bf78f277cc6625b792f8818e8d4 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 01:42:32 -0700 Subject: [PATCH 26/35] fix(power): count sample_loss_exceeded as GPU coverage for the CPU window MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit srt-slurm power/windows.py _check_coverage has emitted sample_loss_exceeded beside the three other coverage codes since v2.39.3 (#537), so a window that only lost GPU samples keeps binding the CPU leg instead of reporting cpu_window_unavailable. 中文：srt-slurm 自 v2.39.3 起在同一覆盖检查中输出 sample_loss_exceeded；仅丢失 GPU 采样的窗口仍可绑定 CPU leg，而不再报告 cpu_window_unavailable。 --- inferencex-e2e/infx/results/power/multinode.py | 10 ++++++++-- .../power/test_aggregate_power_multinode.py | 17 +++++++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 4584d30a4c..8da35fcf6f 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -1285,9 +1285,15 @@ def _select_window_for_result( # Per-window audit codes that describe GPU sample coverage rather than the -# window's own contract; the CPU-side leg re-checks its own coverage. +# window's own contract (srt-slurm power/windows.py _check_coverage); the +# CPU-side leg re-checks its own coverage. _GPU_COVERAGE_REASONS = frozenset( - {"measurement_window_not_bracketed", "sample_gap_exceeded", "gpu_uuid_changed"} + { + "measurement_window_not_bracketed", + "sample_gap_exceeded", + "gpu_uuid_changed", + "sample_loss_exceeded", + } ) diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index 0f2d9ff258..bb3f905ebc 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -915,6 +915,23 @@ def test_expected_acpi_cpu_controls_strict_exit_without_changing_gpu( # The DCGM CPU rail is not the Grace socket meter: no socket series. assert pkg.sidecar()["cpu"]["reason_codes"] == ["cpu_socket_count_mismatch"] + @pytest.mark.parametrize( + "reason, window_trusted", + [ + # srt-slurm power/windows.py _check_coverage: GPU coverage only. + ("measurement_window_not_bracketed", True), + ("sample_gap_exceeded", True), + ("gpu_uuid_changed", True), + ("sample_loss_exceeded", True), + # The window's own contract: the CPU leg has no window either. + ("measurement_window_clock_mismatch", False), + ], + ) + def test_only_gpu_coverage_failures_leave_the_cpu_window_bound(self, reason, window_trusted): + window = {"benchmark_type": "sa-bench", "concurrency": 4} + validations = [{**window, "reason_codes": [reason], "power_coverage_valid": False}] + assert apm._window_contract_holds(window, validations) is window_trusted + def test_package_without_cpu_dir_emits_no_cpu_fields(self, tmp_path): pkg = build_package(tmp_path) assert pkg.run() == 0 From 6ce28a20507d42a9a3be491f5fc0dafee6ae80e2 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:30:56 -0700 Subject: [PATCH 27/35] test(power): restore the v3 temperature rejection test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 911504552 deleted this GPU sample test with the CPU parser cleanup; read_samples still enforces MAX_TEMPERATURE_C. 中文：恢复被 911504552 随 CPU 解析清理误删的 v3 温度校验测试；read_samples 仍执行 MAX_TEMPERATURE_C 检查。 --- .../results/power/test_aggregate_power_multinode.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index bb3f905ebc..bbbb6fd564 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -752,6 +752,18 @@ def test_v2_samples_reject_mixed_versions_and_invalid_utilization(tmp_path, row) assert reasons == ("samples_csv_malformed",) +@pytest.mark.parametrize("temperature", ["nan", "inf", "bad", "-300", "200.5", "9223372036854775794"]) +def test_v3_samples_reject_invalid_temperature(tmp_path, temperature): + path = tmp_path / "samples.csv" + path.write_text( + "schema_version,timestamp_unix,scrape_seq,hostname,gpu_index,gpu_uuid,power_w,gpu_util_pct,sm_active,temperature_c\n" + f"3,1000,0,node-a,0,GPU-a,400,,,{temperature}\n" + ) + rows, reasons = apm.read_samples(path) + assert rows == () + assert reasons == ("samples_csv_malformed",) + + # --- Grace CPU-side power leg (power/cpu/) ----------------------------------- CPU_HOSTS = tuple(sorted({host for host, *_ in DEVICES})) From 728d4bf7eeb6a23ba4645b90bb4c415e1f2ae584 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:32:30 -0700 Subject: [PATCH 28/35] refactor(power): drop the launcher cpu-power guard the matrix validator owns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit validation.py _cpu_power_rides_on_power rejects cpu-power without power on every search-space entry, and CPU_POWER reaches the launcher only from that validated matrix row. 中文：删除启动器中重复的 cpu-power 检查；validation.py 已在所有 search-space 条目上拒绝缺少 power 的 cpu-power，启动器的 CPU_POWER 只来自已校验的矩阵行。 --- inferencex-e2e/infx/launch/drivers/srt/power.py | 2 -- .../infx/tests/launch/test_launch_policy.py | 13 ------------- 2 files changed, 15 deletions(-) diff --git a/inferencex-e2e/infx/launch/drivers/srt/power.py b/inferencex-e2e/infx/launch/drivers/srt/power.py index c0cda8c7a7..54079d0767 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/power.py +++ b/inferencex-e2e/infx/launch/drivers/srt/power.py @@ -150,8 +150,6 @@ def decide_power(cluster_id: str, path: LaunchPath, request: MultiNodeRequest) - Raises ``PowerPolicyError`` for a combination the lane does not allow. """ - if request.cpu_power and not request.power: - raise PowerPolicyError("cpu-power requires power on the same point") if not request.power: return NO_POWER lane = POWER_LANES.get((cluster_id, path)) diff --git a/inferencex-e2e/infx/tests/launch/test_launch_policy.py b/inferencex-e2e/infx/tests/launch/test_launch_policy.py index 9158afbee8..f92abb1075 100644 --- a/inferencex-e2e/infx/tests/launch/test_launch_policy.py +++ b/inferencex-e2e/infx/tests/launch/test_launch_policy.py @@ -78,19 +78,6 @@ def test_power_eligibility_of_each_cluster_lane( assert not decide_power(cluster, path, _multinode(**env, POWER="0")).dcgm -@pytest.mark.parametrize(("power", "cpu_power", "expected"), [ - ("1", "1", True), ("1", "0", False), ("0", "1", ERR), -]) # fmt: skip -def test_cpu_power_rides_on_a_power_point(power, cpu_power, expected): - request = _multinode(IS_AGENTIC="0", MODEL_PREFIX="dsr1", PRECISION="fp8", - FRAMEWORK="dynamo-sglang", POWER=power, CPU_POWER=cpu_power) # fmt: skip - if expected == ERR: - with pytest.raises(PowerPolicyError, match="cpu-power requires power"): - decide_power("gb200-nv", MULTI, request) - return - assert decide_power("gb200-nv", MULTI, request).cpu is expected - - def test_gb200_reports_agentic_misses_distinctly(): request = _multinode(IS_AGENTIC="1", MODEL_PREFIX="dsv4", PRECISION="fp4", FRAMEWORK="dynamo-sglang", CONC="4", CONC_LIST="4", POWER="1") # fmt: skip From e9e50d3e059a4fd46fb3511b4bbf2685f107beba Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:33:54 -0700 Subject: [PATCH 29/35] refactor(power): accept only the ACPI Grace CPU exporter source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cpu_side.py integrates only CPU:cpuSidePowerUsageW rows, which srt-slurm emits from ACPI; a dcgm source always ended in cpu_socket_count_mismatch. 中文：CPU exporter 来源只接受 ACPI；cpu_side.py 只积分 ACPI 输出的 CPU:cpuSidePowerUsageW 行，dcgm 来源必然以 cpu_socket_count_mismatch 失败。 --- inferencex-e2e/infx/clusters/slurm.py | 3 ++- inferencex-e2e/infx/srt_slurm/workload.py | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/inferencex-e2e/infx/clusters/slurm.py b/inferencex-e2e/infx/clusters/slurm.py index 342bbe9ff2..4731be5171 100644 --- a/inferencex-e2e/infx/clusters/slurm.py +++ b/inferencex-e2e/infx/clusters/slurm.py @@ -177,9 +177,10 @@ class CpuPowerExporter(Record): """The Grace CPU power exporter srt-slurm scrapes on this cluster's compute trays. A point that selects ``cpu-power`` binds ``telemetry.cpu_power_exporter`` with this source. + Only ACPI serves the Grace socket meter the CPU leg integrates. """ - source: Literal["acpi", "dcgm"] + source: Literal["acpi"] class SrtSlurmSettings(Record): diff --git a/inferencex-e2e/infx/srt_slurm/workload.py b/inferencex-e2e/infx/srt_slurm/workload.py index 15f43750de..95e4ac57c9 100644 --- a/inferencex-e2e/infx/srt_slurm/workload.py +++ b/inferencex-e2e/infx/srt_slurm/workload.py @@ -433,7 +433,7 @@ def main(argv: list[str] | None = None) -> None: ) parser.add_argument( "--cpu-power-source", - choices=("acpi", "dcgm"), + choices=("acpi",), help="bind the cluster's Grace CPU power exporter with this source (needs --power-port)", ) parser.add_argument( From fc3faaf071a7b372994bbd40c17356701c5e094a Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:36:00 -0700 Subject: [PATCH 30/35] refactor(power): stop gating the CPU leg on cpu_manifest.json MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit srt-slurm documents cpu_manifest.json as non-authoritative debugging metadata and only logs a failed teardown write (cpu_session.py:218), so a broken manifest no longer invalidates a leg whose samples cover the window. 中文：CPU leg 不再依据 cpu_manifest.json 判定；srt-slurm 将其定义为非权威调试元数据且写入失败只记日志，样本已覆盖窗口时不再因 manifest 损坏而判为无效。 --- .github/AGENT_OPERATIONS.md | 2 +- inferencex-e2e/infx/results/power/cpu_side.py | 19 ++++--------------- .../power/test_aggregate_power_multinode.py | 19 +------------------ 3 files changed, 6 insertions(+), 34 deletions(-) diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 7f55db4908..990e8ff7ca 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -76,7 +76,7 @@ A multinode master row with `cpu-power: true` beside `power: true` adds srt-slur The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware, and the superchip module meter is unbound in ACPI; NVML exposes it (`nvidia-smi --query-gpu=module.power.draw.average`, srt-slurm #591 pending) and no package publishes it. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. -The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_manifest_invalid`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`. +The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`. Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index 78df481111..cb2e5c082e 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -1,9 +1,9 @@ """Grace CPU-side power leg of the multinode srt-slurm package. -srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` and a -non-authoritative ``power/cpu/cpu_manifest.json`` beside the GPU DCGM package -(srt-slurm ``src/srtctl/core/power/cpu_session.py``). The pinned producer writes -the wide format: one row per (scrape, host, socket) whose ``sensor`` names the +srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` beside +the GPU DCGM package (srt-slurm ``src/srtctl/core/power/cpu_session.py``); its +``cpu_manifest.json`` is debugging metadata and is not read. The pinned producer +writes the wide format: one row per (scrape, host, socket) whose ``sensor`` names the Grace socket meter and whose ``cpu_rail_w`` / ``soc_w`` columns carry the ACPI component rails of that socket. @@ -20,7 +20,6 @@ import csv import itertools -import json import math from collections.abc import Iterable from dataclasses import dataclass, field @@ -37,7 +36,6 @@ CPU_DIRNAME = "cpu" CPU_SAMPLES_FILENAME = "samples.csv" -CPU_MANIFEST_FILENAME = "cpu_manifest.json" # srt-slurm contract.CPU_SAMPLES_HEADER: the wide format every pinned producer writes. CPU_SAMPLES_HEADER = ( @@ -191,13 +189,6 @@ def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, . return tuple(rows), tuple(reasons) -def _manifest_is_object(path: Path) -> bool: - try: - return isinstance(json.loads(path.read_text(encoding="utf-8")), dict) - except (OSError, UnicodeDecodeError, ValueError): - return False - - SocketKey = tuple[str, int] # (host, socket) -> family -> {timestamp: watts}; the Grace meter and its rails. SocketSeries = dict[SocketKey, dict[str, dict[float, float]]] @@ -249,8 +240,6 @@ def validate_cpu_leg( audit.invalidate("cpu_artifacts_missing") return audit audit = CpuPowerAudit() - if not _manifest_is_object(cpu_dir / CPU_MANIFEST_FILENAME): - audit.invalidate("cpu_manifest_invalid") rows, sample_reasons = read_cpu_samples(cpu_dir / CPU_SAMPLES_FILENAME) audit.sample_row_count = len(rows) audit.invalidate(*sample_reasons) diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index bbbb6fd564..9e5a8902f4 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -803,30 +803,13 @@ def _cpu_rows(kinds_by_host=None, *, hosts=CPU_HOSTS): return rows -def add_cpu_package(pkg, rows, manifest_text=None): +def add_cpu_package(pkg, rows): cpu_dir = pkg.power_dir / "cpu" cpu_dir.mkdir() with open(cpu_dir / "samples.csv", "w", newline="") as handle: writer = csv.writer(handle) writer.writerow(CPU_HEADER) writer.writerows(rows) - if manifest_text is None: - manifest_text = json.dumps( - { - "schema_version": 1, - "producer": "srt-slurm.cpu-power", - "producer_git_commit": PRODUCER_SHA, - "started_at_unix": 990.0, - "stopped_at_unix": 1070.0, - "nodes": { - host: {"resolved_mode": "acpi", "scrape_count": 65, "error_count": 0} - for host in CPU_HOSTS - }, - }, - indent=2, - ) - (cpu_dir / "cpu_manifest.json").write_text(manifest_text) - return cpu_dir def _gpu_fields(agg): From dd130b7a703ddbd61f5eb6caa257112415bc4e7e Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:37:42 -0700 Subject: [PATCH 31/35] chore(recipes): drop the Qwen3.5 8P1D benchmark placement that restates the head default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit srt-slurm schema.py:1162 already places the benchmark client on head, the dedicated frontend node. 中文：删除 Qwen3.5 8P1D recipe 中重复默认值的 benchmark placement；srt-slurm schema.py:1162 默认即在 head（即 dedicated frontend 节点）运行 benchmark client。 --- .../sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 4 ---- .../sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml | 4 ---- 2 files changed, 8 deletions(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index 08759d1a7f..d0526b16b3 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -155,7 +155,3 @@ roles: decode-log-interval: 1 stream-interval: 50 - -benchmark: - placement: - node: head diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml index 1e8092407b..3506fab0cf 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml @@ -154,7 +154,3 @@ roles: decode-log-interval: 1 stream-interval: 50 - -benchmark: - placement: - node: head From eabb245af321a65ee948d00e9f657c65d86aee24 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:38:25 -0700 Subject: [PATCH 32/35] refactor(power): drop the multinode CLI flag no caller passes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fixed_sequence.py and the AgentX adapter call run() with expected_cpu directly; nothing invokes multinode.py as a command. 中文：删除 multinode CLI 中无人传入的 --expected-cpu-power；fixed_sequence.py 与 AgentX adapter 直接以 expected_cpu 调用 run()，没有任何地方以命令行方式调用 multinode.py。 --- inferencex-e2e/infx/results/power/multinode.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index 8da35fcf6f..bfce921ac2 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -1575,9 +1575,8 @@ def main() -> int: "--require-power", action="store_true", default=os.environ.get("REQUIRE_POWER", "").lower() in {"1", "true", "yes"}, - help="Fail invalid GPU power and any expected CPU source (also REQUIRE_POWER=1)", + help="Fail when power telemetry is invalid (also enabled by REQUIRE_POWER=1)", ) - parser.add_argument("--expected-cpu-power", action="store_true") args = parser.parse_args() return run( args.power_dir, @@ -1590,7 +1589,6 @@ def main() -> int: logs_root=args.logs_root, validation_result=args.validation_result, require_power=args.require_power, - expected_cpu=args.expected_cpu_power, ) From faffae20d2bcc974ce4fa6551e95b986145b8aea Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:40:32 -0700 Subject: [PATCH 33/35] refactor(matrix): read cpu-power fields directly like the sibling validators MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both schemas declare cpu_power and power, so the getattr defaults implied fields that could be absent. 中文：cpu-power 校验与同文件其他校验器一致，直接读取字段；两个 schema 都声明 cpu_power 与 power，getattr 默认值暗示字段可能缺失。 --- inferencex-e2e/infx/matrix/validation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/inferencex-e2e/infx/matrix/validation.py b/inferencex-e2e/infx/matrix/validation.py index 0e3d8eb2f2..01c118d0bb 100644 --- a/inferencex-e2e/infx/matrix/validation.py +++ b/inferencex-e2e/infx/matrix/validation.py @@ -607,11 +607,11 @@ def validate_single_node_topology(self) -> Self: return _validate_tp_context_topology(self) -def _cpu_power_rides_on_power[T](entry: T) -> T: +def _cpu_power_rides_on_power(self: Any) -> Any: """``cpu-power`` adds the cluster's Grace CPU exporter to a point that measures power.""" - if getattr(entry, "cpu_power", False) and not getattr(entry, "power", False): + if self.cpu_power and not self.power: raise ValueError(f"{Fields.CPU_POWER.value} requires {Fields.POWER.value} on the entry") - return entry + return self class MultiNodeSearchSpaceEntry(BaseModel): From 73971a0657c5c520572123058f2895ba2220f1cf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 14 Oct 2026 15:40:32 -0700 Subject: [PATCH 34/35] test(power): assert the setup pin through CPU_POWER_EXPORTER_RELEASE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The release bump then edits one constant instead of a test literal. 中文：测试改用 CPU_POWER_EXPORTER_RELEASE 常量断言 setup 版本，升级版本时只需修改一处常量。 --- inferencex-e2e/infx/tests/launch/test_srt_driver.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/infx/tests/launch/test_srt_driver.py b/inferencex-e2e/infx/tests/launch/test_srt_driver.py index 54feef3268..8dfff525f2 100644 --- a/inferencex-e2e/infx/tests/launch/test_srt_driver.py +++ b/inferencex-e2e/infx/tests/launch/test_srt_driver.py @@ -19,6 +19,7 @@ from infx.launch.__main__ import main from infx.launch.drivers.srt import lanes, models +from infx.launch.drivers.srt.checkout import CPU_POWER_EXPORTER_RELEASE from infx.launch.drivers.srt.lanes import LaneMount, SrtLane from infx.launch.drivers.srt.models import Override from infx.launch.policy import LaunchPath, Match @@ -695,7 +696,7 @@ def test_launch_cpu_expectation_reaches_fixed_sequence_processor(harness, monkey assert bound["telemetry"]["cpu_power_exporter"] == {"source": "acpi"} assert bound["telemetry"]["dcgm_exporter"]["port"] == 19401 [setup] = [call for call in lines(harness.logs, "make") if call.startswith("setup ")] - assert "CPU_POWER_EXPORTER_RELEASE=v2.51.1" in setup.split() + assert f"CPU_POWER_EXPORTER_RELEASE={CPU_POWER_EXPORTER_RELEASE}" in setup.split() pkg = build_package(harness.tmp / "power-input") monkeypatch.chdir(harness.tmp) # The workflow exports CPU_POWER once; the launcher and the processor both read it. From 323371beb81da3ed39299b106d5540cb14a45f00 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Thu, 15 Oct 2026 18:21:28 -0700 Subject: [PATCH 35/35] feat(power): publish NVML module power from CPU samples v3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v2 files are still accepted, and module power is a reference rail that is never summed into the CPU totals. 中文：从 CPU 采样 v3 中发布 NVML module 功耗。仍兼容 v2 文件；module 功耗仅作为参考 rail，不计入 CPU 总功耗。 --- .github/AGENT_OPERATIONS.md | 2 +- inferencex-e2e/infx/results/power/__init__.py | 7 ++- inferencex-e2e/infx/results/power/cpu_side.py | 60 +++++++++++-------- .../infx/results/power/multinode.py | 2 +- .../power/test_aggregate_power_multinode.py | 52 +++++++++++++++- inferencex-e2e/perf-changelog.yaml | 2 +- 6 files changed, 92 insertions(+), 33 deletions(-) diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 990e8ff7ca..8710529af6 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -74,7 +74,7 @@ Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for those jobs only, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor. -The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware, and the superchip module meter is unbound in ACPI; NVML exposes it (`nvidia-smi --query-gpu=module.power.draw.average`, srt-slurm #591 pending) and no package publishes it. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. +The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`. diff --git a/inferencex-e2e/infx/results/power/__init__.py b/inferencex-e2e/infx/results/power/__init__.py index fe47881bfa..a765c894e4 100644 --- a/inferencex-e2e/infx/results/power/__init__.py +++ b/inferencex-e2e/infx/results/power/__init__.py @@ -30,8 +30,9 @@ "decode_joules_per_output_token", ) ALL_POWER_METRIC_KEYS = WHOLE_METRIC_KEYS + ROLE_METRIC_KEYS -# Additive Grace CPU-side metrics from the multinode cpu/ sub-package. They -# share schema version 2 and are governed by cpu_power_valid, not power_valid. +# Additive metrics from the multinode cpu/ sub-package: the Grace socket, its +# ACPI rails and the NVML module rail. They share power metric schema version 2 +# and are governed by cpu_power_valid, not power_valid. CPU_METRIC_KEYS = ( "avg_cpu_socket_power_w", "avg_total_cpu_power_w", @@ -40,6 +41,8 @@ "total_cpu_rail_energy_j", "avg_total_cpu_sysio_power_w", "total_cpu_sysio_energy_j", + "avg_total_module_power_w", + "total_module_energy_j", ) diff --git a/inferencex-e2e/infx/results/power/cpu_side.py b/inferencex-e2e/infx/results/power/cpu_side.py index cb2e5c082e..e2fb4f5484 100644 --- a/inferencex-e2e/infx/results/power/cpu_side.py +++ b/inferencex-e2e/infx/results/power/cpu_side.py @@ -2,16 +2,18 @@ srt-slurm's ``cpu_power_exporter`` leg writes ``power/cpu/samples.csv`` beside the GPU DCGM package (srt-slurm ``src/srtctl/core/power/cpu_session.py``); its -``cpu_manifest.json`` is debugging metadata and is not read. The pinned producer -writes the wide format: one row per (scrape, host, socket) whose ``sensor`` names the +``cpu_manifest.json`` is debugging metadata and is not read. Both accepted +schemas are the wide format: one row per (scrape, host, socket) whose ``sensor`` names the Grace socket meter and whose ``cpu_rail_w`` / ``soc_w`` columns carry the ACPI -component rails of that socket. +component rails of that socket. Schema v3 (srt-slurm v2.59.0+) adds ``module_w``, +NVML superchip module power; the module contains both GPUs, so it is published +as its own rail and never added to the CPU or GPU totals. Each socket's Grace socket series is integrated over the GPU leg's bound formal window with the shared trapezoid and boundary interpolation and published as -additive metrics next to the GPU-board numbers. The CPU-only and SysIO rails -are integrated the same way when every socket reports them; a rail with gaps -drops only its own metrics. Any failure records ``cpu_power_valid=0`` with +additive metrics next to the GPU-board numbers. The CPU-only, SysIO and module +rails are integrated the same way when every socket reports them; a rail with +gaps drops only its own metrics. Any failure records ``cpu_power_valid=0`` with reason codes and leaves every GPU field untouched. This module only judges the leg; the caller decides whether an invalid leg fails the job. """ @@ -37,8 +39,7 @@ CPU_DIRNAME = "cpu" CPU_SAMPLES_FILENAME = "samples.csv" -# srt-slurm contract.CPU_SAMPLES_HEADER: the wide format every pinned producer writes. -CPU_SAMPLES_HEADER = ( +_CPU_SAMPLES_HEADER_V2 = ( "schema_version", "timestamp_unix", "hostname", @@ -51,7 +52,12 @@ "dram_w", "total_power_w", ) -CPU_SAMPLES_SCHEMA_VERSION = 2 +# srt-slurm cpu_samples._SCHEMA_VERSION_BY_HEADER: every row carries its header's +# version. v2.51.1 writes v2; v2.59.0+ writes v3, module_w before total_power_w. +_SCHEMA_VERSION_BY_HEADER = { + _CPU_SAMPLES_HEADER_V2: 2, + (*_CPU_SAMPLES_HEADER_V2[:-1], "module_w", "total_power_w"): 3, +} # One GB200/GB300 compute tray per worker host, two Grace sockets per tray. SOCKETS_PER_HOST = 2 @@ -62,8 +68,9 @@ SENSOR_GRACE = "grace_socket" _GRACE_SENSOR_SUFFIX = "cpuSidePowerUsageW" # Wide-format rail column -> published rail family: "CPU Power Socket N" (cores, -# no SysIO) and "SysIO Power Socket N". dram_w stays blank on NVL72 firmware. -RAIL_COLUMNS = {"cpu_rail_w": "cpu_rail", "soc_w": "cpu_sysio"} +# no SysIO), "SysIO Power Socket N" and the v3 NVML superchip module. dram_w +# stays blank on NVL72 firmware; module_w is blank when NVML is unavailable. +RAIL_COLUMNS = {"cpu_rail_w": "cpu_rail", "soc_w": "cpu_sysio", "module_w": "module"} _INTEGRATION_METHOD = "per_socket_trapezoidal_with_linear_boundary_interpolation" @@ -127,18 +134,19 @@ def is_grace_sensor(sensor: str) -> bool: return sensor.partition(":")[2] == _GRACE_SENSOR_SUFFIX -def _parse_row(raw: list[str]) -> CpuSampleRow | None: - if len(raw) != len(CPU_SAMPLES_HEADER): +def _parse_row(raw: list[str], header: tuple[str, ...], version: int) -> CpuSampleRow | None: + if len(raw) != len(header): return None + cell = dict(zip(header, raw, strict=True)) try: - schema_version = int(raw[0]) - timestamp_unix = float(raw[1]) - socket_id = int(raw[5]) - power_w = float(raw[6]) + schema_version = int(cell["schema_version"]) + timestamp_unix = float(cell["timestamp_unix"]) + socket_id = int(cell["socket_id"]) + power_w = float(cell["power_w"]) except ValueError: return None - hostname, source, sensor = raw[2], raw[3], raw[4] - if schema_version != CPU_SAMPLES_SCHEMA_VERSION or not hostname or not source or not sensor: + hostname, source, sensor = cell["hostname"], cell["source"], cell["sensor"] + if schema_version != version or not hostname or not source or not sensor: return None if not math.isfinite(timestamp_unix) or not math.isfinite(power_w) or power_w < 0: return None @@ -146,11 +154,11 @@ def _parse_row(raw: list[str]) -> CpuSampleRow | None: return None rails = [] for column, family in RAIL_COLUMNS.items(): - cell = raw[CPU_SAMPLES_HEADER.index(column)] - if not cell: + value = cell.get(column, "") + if not value: continue try: - watts = float(cell) + watts = float(value) except ValueError: return None if not math.isfinite(watts) or watts < 0: @@ -168,7 +176,7 @@ def _parse_row(raw: list[str]) -> CpuSampleRow | None: def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, ...]]: - """Strictly parse the wide sample file; malformed rows invalidate the leg.""" + """Strictly parse a v2 or v3 wide sample file; malformed rows invalidate the leg.""" if not path.is_file(): return (), ("cpu_samples_missing",) rows: list[CpuSampleRow] = [] @@ -176,10 +184,12 @@ def read_cpu_samples(path: Path) -> tuple[tuple[CpuSampleRow, ...], tuple[str, . try: with open(path, newline="", encoding="utf-8") as handle: reader = csv.reader(handle) - if next(reader, None) != list(CPU_SAMPLES_HEADER): + header = tuple(next(reader, ())) + version = _SCHEMA_VERSION_BY_HEADER.get(header) + if version is None: return (), ("cpu_samples_header_mismatch",) for raw in reader: - row = _parse_row(raw) + row = _parse_row(raw, header, version) if row is None: _append_reason(reasons, "cpu_samples_malformed") continue diff --git a/inferencex-e2e/infx/results/power/multinode.py b/inferencex-e2e/infx/results/power/multinode.py index bfce921ac2..29877b1283 100644 --- a/inferencex-e2e/infx/results/power/multinode.py +++ b/inferencex-e2e/infx/results/power/multinode.py @@ -29,7 +29,7 @@ and aggregate power metrics remain blocked. Grace CPU-side leg: when the package also carries ``power/cpu/`` (see -:mod:`.cpu_side`), each socket's Grace socket power and its ACPI rails are +:mod:`.cpu_side`), each socket's Grace socket power and its reference rails are integrated over the same bound formal window and published under ``cpu_power_valid``. That verdict is independent of ``power_valid``. A missing or invalid CPU leg fails the run only with ``require_power`` and diff --git a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py index 9e5a8902f4..c81e554847 100644 --- a/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py +++ b/inferencex-e2e/infx/tests/results/power/test_aggregate_power_multinode.py @@ -771,11 +771,15 @@ def test_v3_samples_reject_invalid_temperature(tmp_path, temperature): "schema_version", "timestamp_unix", "hostname", "source", "sensor", "socket_id", "power_w", "cpu_rail_w", "soc_w", "dram_w", "total_power_w", ] +CPU_HEADER_V3 = [*CPU_HEADER[:-1], "module_w", "total_power_w"] GRACE_W = {"node-d": 500.0, "node-p": 700.0} DCGM_W = 300.0 # Component rails never feed a headline metric; the values are chosen so any # leak into a headline key is visible against the Grace constants. RAILS_W = {"cpu_rail": 200.0, "soc": 50.0} +# NVML superchip module power per socket, distinct from the Grace and rail +# constants so a leak into a CPU key is visible. +MODULE_W = 1500.0 # srt-slurm cpu_rails.SENSOR_SUFFIXES: the Grace socket meter and the DCGM CPU rail. GRACE_SENSOR = "CPU{s}:cpuSidePowerUsageW" DCGM_SENSOR = "CPU{s}:cpuPowerUsageW" @@ -803,12 +807,18 @@ def _cpu_rows(kinds_by_host=None, *, hosts=CPU_HOSTS): return rows -def add_cpu_package(pkg, rows): +def _v3(rows, module_w=MODULE_W): + """The same scrapes in srt-slurm v2.59.0+'s schema 3; ``None`` leaves module_w blank.""" + module = "" if module_w is None else repr(module_w) + return [[3, *row[1:-1], module, row[-1]] for row in rows] + + +def add_cpu_package(pkg, rows, header=CPU_HEADER): cpu_dir = pkg.power_dir / "cpu" cpu_dir.mkdir() with open(cpu_dir / "samples.csv", "w", newline="") as handle: writer = csv.writer(handle) - writer.writerow(CPU_HEADER) + writer.writerow(header) writer.writerows(rows) @@ -883,6 +893,37 @@ def test_rail_gap_drops_only_that_rail(self, tmp_path): assert "avg_total_cpu_rail_power_w" not in agg assert agg["avg_total_cpu_sysio_power_w"] == 200.0 + def test_v3_module_rail_is_published_apart_from_cpu_keys(self, tmp_path): + pkg = build_package(tmp_path) + add_cpu_package(pkg, _v3(_cpu_rows()), CPU_HEADER_V3) + assert pkg.run(require_power=True) == 0 + agg = pkg.agg() + assert_grace_keys(agg) + assert agg["avg_total_module_power_w"] == 6000.0 + assert agg["total_module_energy_j"] == 360000.0 + assert _gpu_fields(agg) == _reference_agg(tmp_path) + module_series = {f"{host}/socket{socket}/module" for host in CPU_HOSTS for socket in (0, 1)} + assert module_series <= set(pkg.sidecar()["cpu"]["per_series_energy_j"]) + + @pytest.mark.parametrize("module", ["blank", "gap"]) + def test_v3_incomplete_module_drops_only_module_keys(self, tmp_path, module): + pkg = build_package(tmp_path) + if module == "blank": + # The musl exporter build cannot load NVML and leaves module_w empty. + rows = _v3(_cpu_rows(), module_w=None) + else: + rows = _v3(_cpu_rows()) + for row in rows: + if row[2] == "node-d" and row[5] == 0 and 1020.0 <= float(row[1]) <= 1030.0: + row[10] = "" + add_cpu_package(pkg, rows, CPU_HEADER_V3) + assert pkg.run(require_power=True) == 0 + agg = pkg.agg() + assert_grace_keys(agg) + assert agg["avg_total_cpu_rail_power_w"] == 800.0 + assert agg["avg_total_cpu_sysio_power_w"] == 200.0 + assert {"avg_total_module_power_w", "total_module_energy_j"}.isdisjoint(agg) + @pytest.mark.parametrize( "cpu, require_power", [("missing", True), ("complete", True), ("missing", False), ("dcgm_only", True)], @@ -947,6 +988,7 @@ def tampered(self, request, tmp_path): kind = request.param pkg = build_package(tmp_path) rows = _cpu_rows() + header = CPU_HEADER if kind == "socket": rows = self._drop(rows, "node-p", 1, lambda ts: True) elif kind == "gap": @@ -955,7 +997,10 @@ def tampered(self, request, tmp_path): rows = _cpu_rows({"node-d": {"grace": None}, "node-p": {"dcgm": DCGM_W}}) elif kind == "malformed": rows.append(_cpu_row(FIRST_TS, "node-d", 0, "grace", "n/a")) - add_cpu_package(pkg, rows) + elif kind == "version": + rows, header = _v3(rows), CPU_HEADER_V3 + rows[0][0] = 2 + add_cpu_package(pkg, rows, header) return pkg @pytest.mark.parametrize( @@ -965,6 +1010,7 @@ def tampered(self, request, tmp_path): ("gap", "cpu_sample_gap_exceeded"), ("mixed", "cpu_socket_count_mismatch"), ("malformed", "cpu_samples_malformed"), + ("version", "cpu_samples_malformed"), ], indirect=["tampered"], ) diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index f44ec0984d..85376ccbbe 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -9906,6 +9906,6 @@ - agentic-coding description: - "Collect ACPI Grace CPU power on the Qwen3.5 GB200/GB300 1P1D, 4P1D, and 8P1D NVL72 recipes and the Kimi K3 GB200/GB300 AgentX recipes. The master rows set cpu-power beside power; the binder merges the cluster's cpu-power-exporter record (ACPI on gb200-nv and gb300-nv) into the bound recipe, and make setup pins cpu-power-exporter v2.51.1 for those jobs, matching the srt-slurm v2.51.1 source pin." - - "Integrate each Grace socket total over the formal GPU measurement window and emit cpu_power_valid, avg_cpu_socket_power_w, avg_total_cpu_power_w, and total_cpu_energy_j independently of the GPU power verdict, with the ACPI CPU-only and SysIO rails as separate keys. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. Under required power a cpu-power row fails on a missing or invalid CPU leg; other rows keep the best-effort CPU verdict." + - "Integrate each Grace socket total over the formal GPU measurement window and emit cpu_power_valid, avg_cpu_socket_power_w, avg_total_cpu_power_w, and total_cpu_energy_j independently of the GPU power verdict, with the ACPI CPU-only and SysIO rails as separate keys. When the producer writes module_w (CPU samples schema v3), also emit avg_total_module_power_w and total_module_energy_j as a separate NVML superchip module reference rail; the module includes both GPUs and is never added to the CPU totals. Publish validated per-host GPU power so cross-host workers can be modeled as physical trays. Under required power a cpu-power row fails on a missing or invalid CPU leg; other rows keep the best-effort CPU verdict." - "Enable required DCGM telemetry with collect_interval_ms: 1000 on the Qwen3.5 GB200/GB300 8P1D recipes, and set require-power on both Qwen3.5 GB200/GB300 8k1k sequences. Keep the Qwen3.5 8P1D frontend, benchmark client, and infra on the reserved head so power and benchmark timestamps share its clock, while retaining 13 allocated nodes and 12 GPU workers." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3296