[PowerX] collect NVL72 CPU and per-host GPU power / [PowerX] 采集 NVL72 CPU 与逐主机 GPU 功耗

.github/AGENT_OPERATIONS.md

@@ -68,10 +68,16 @@ Eval selection, the `--no-evals` / `--evals-only` / `--all-evals` flags, and cha
 
 ## Power telemetry
 
-Multinode srt-slurm results, and single-node fixed-sequence and AgentX results with a retained native telemetry package, may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`.
+Multinode srt-slurm results, and single-node fixed-sequence and AgentX results with a retained native telemetry package, may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power.
 
 Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count.
 
+A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for those jobs only, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor.
+
+The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU<n>:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes.
+
+The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
+
 Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
 
 A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment; native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.

.github/workflows/benchmark-multinode-tmpl.yml

@@ -146,6 +146,7 @@ env:
   EVAL_CONC: ${{ inputs.eval-conc }}
   EVAL_LIMIT: ${{ inputs.eval-limit }}
   POWER: ${{ fromJSON(inputs.config).power && '1' || '0' }}
+  CPU_POWER: ${{ fromJSON(inputs.config)['cpu-power'] && '1' || '0' }}
   REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
   POWER_PRODUCER_SHA: ${{ inputs.power-producer-sha }}
   # Agentic-coding env. Fixed-seq-len jobs leave these empty.

inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml

@@ -10,18 +10,20 @@ services:
   - name: etcd
     type: etcd
     placement:
-      node: dedicated
+      node: head
   - name: nats
     type: nats
     placement:
-      node: dedicated
+      node: head
 dynamo:
   install: true
 
   source:
     wheel: "1.5.0.dev20260917"
 frontend:
   type: dynamo
+  placement:
+    node: dedicated
   enable_multiple_frontends: true
   num_additional_frontends: 4
   nginx_container: nginx

inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-8p1d-dep4-dep16-stp.yaml

@@ -9,18 +9,20 @@ services:
   - name: etcd
     type: etcd
     placement:
-      node: dedicated
+      node: head
   - name: nats
     type: nats
     placement:
-      node: dedicated
+      node: head
 dynamo:
   install: true
 
   source:
     wheel: "1.5.0.dev20260917"
 frontend:
   type: dynamo
+  placement:
+    node: dedicated
   enable_multiple_frontends: true
   num_additional_frontends: 4
   nginx_container: nginx

inferencex-e2e/configs/nvidia-master.yaml

@@ -3726,12 +3726,14 @@ qwen3.5-fp8-gb200-dynamo-sglang:
     fixed-seq-len:
     - isl: 8192
       osl: 1024
+      require-power: true
       search-space:
       # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1).
       - spec-decoding: "none"
         conc-list: [1, 2, 4, 8, 16, 32, 64, 128]
         srt-recipe: disagg-1p1d-tp4-tp4-stp.yaml
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 4
@@ -3747,6 +3749,7 @@ qwen3.5-fp8-gb200-dynamo-sglang:
         conc-list: [1024]
         srt-recipe: disagg-4p1d-dep4-dep16-stp.yaml
         power: true
+        cpu-power: true
         prefill:
           num-worker: 4
           tp: 4
@@ -3761,6 +3764,8 @@ qwen3.5-fp8-gb200-dynamo-sglang:
       - spec-decoding: "none"
         conc-list: [2048, 4096]
         srt-recipe: disagg-8p1d-dep4-dep16-stp.yaml
+        power: true
+        cpu-power: true
         prefill:
           num-worker: 8
           tp: 4
@@ -6198,6 +6203,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg:
         num-nodes: 4
         srt-recipe: agg-dcp16-dspark4-maxseq2-mooncake.yaml
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 16
@@ -6225,6 +6231,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg:
         num-nodes: 4
         srt-recipe: agg-dcp16-nospec-mooncake.yaml
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 16
@@ -6361,6 +6368,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2:
         num-nodes: 4
         srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c16
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -6375,6 +6383,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2:
         num-nodes: 4
         srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c32
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -6388,6 +6397,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2:
         num-nodes: 4
         srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c48
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -6401,6 +6411,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2:
         num-nodes: 4
         srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c72
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -6414,6 +6425,7 @@ kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2:
         num-nodes: 4
         srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c96
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -6679,12 +6691,14 @@ qwen3.5-fp8-gb300-dynamo-sglang:
     fixed-seq-len:
     - isl: 8192
       osl: 1024
+      require-power: true
       search-space:
       # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1).
       - spec-decoding: "none"
         conc-list: [1, 2, 4, 8, 16, 32, 64, 128]
         srt-recipe: disagg-1p1d-tp4-tp4-stp.yaml
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 4
@@ -6700,6 +6714,7 @@ qwen3.5-fp8-gb300-dynamo-sglang:
         conc-list: [1024]
         srt-recipe: disagg-4p1d-dep4-dep16-stp.yaml
         power: true
+        cpu-power: true
         prefill:
           num-worker: 4
           tp: 4
@@ -6714,6 +6729,8 @@ qwen3.5-fp8-gb300-dynamo-sglang:
       - spec-decoding: "none"
         conc-list: [2048, 4096]
         srt-recipe: disagg-8p1d-dep4-dep16-stp.yaml
+        power: true
+        cpu-power: true
         prefill:
           num-worker: 8
           tp: 4
@@ -7869,6 +7886,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg:
         conc-list: [48, 56]
         srt-recipe: disagg-variants.yaml:override_1p1d_dcp8_dcp8_dspark4
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 8
@@ -7882,6 +7900,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg:
         conc-list: [48, 64]
         srt-recipe: disagg-variants.yaml:override_1p2d_dcp8_dcp8_dspark4
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 8
@@ -7895,6 +7914,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg:
         conc-list: [1]
         srt-recipe: disagg-variants.yaml:override_1p3d_dcp8_dcp8_dspark7
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 8
@@ -7908,6 +7928,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg:
         conc-list: [48]
         srt-recipe: disagg-variants.yaml:override_1p3d_dcp8_dcp8_dspark4
         power: true
+        cpu-power: true
         prefill:
           num-worker: 1
           tp: 8
@@ -7937,6 +7958,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg:
         num-nodes: 2
         srt-recipe: agg-variants.yaml:override_dcp8_dspark7_maxseq2
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8
@@ -7950,6 +7972,7 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg:
         num-nodes: 2
         srt-recipe: agg-variants.yaml:override_dcp8_dspark4
         power: true
+        cpu-power: true
         worker:
           num-worker: 1
           tp: 8

inferencex-e2e/configs/runners.yaml

@@ -622,6 +622,7 @@ clusters:
           aiperf-cache: /aiperf_mmap_cache
           hf-hub-cache: /hf_hub_cache
         power-exporter-port: 9401
+        cpu-power-exporter: {source: acpi}
   gb300-nv:
     gpus-per-node: 4
     available-cpu-dram-mib: 920_000
@@ -680,6 +681,7 @@ clusters:
           hf-hub-cache: /hf_hub_cache
           dynamo-wheels: /configs/dynamo-wheels
         power-exporter-port: 19401
+        cpu-power-exporter: {source: acpi}
   mi300x-amd:
     gpus-per-node: 8
     available-cpu-dram-mib: 1_547_820

inferencex-e2e/infx/clusters/slurm.py

@@ -173,6 +173,16 @@ def rendered(self) -> dict[str, str | None]:
         return self.model_dump(by_alias=True)
 
 
+class CpuPowerExporter(Record):
+    """The Grace CPU power exporter srt-slurm scrapes on this cluster's compute trays.
+
+    A point that selects ``cpu-power`` binds ``telemetry.cpu_power_exporter`` with this source.
+    Only ACPI serves the Grace socket meter the CPU leg integrates.
+    """
+
+    source: Literal["acpi"]
+
+
 class SrtSlurmSettings(Record):
     """Cluster-owned srtslurm.yaml facts; job-specific values are added by the driver."""
 
@@ -203,6 +213,7 @@ class SrtSlurmSettings(Record):
     mounts: dict[str, str] = Field(default_factory=dict)
     workspace_mounts: dict[str, str] = Field(default_factory=dict, alias="workspace-mounts")
     power_exporter_port: int | None = Field(default=None, alias="power-exporter-port", gt=0)
+    cpu_power_exporter: CpuPowerExporter | None = Field(default=None, alias="cpu-power-exporter")
     extra: dict[str, Any] = Field(default_factory=dict)
 
     @field_validator("workspace_mounts")

inferencex-e2e/infx/launch/artifacts.py

@@ -153,6 +153,7 @@ def collect_agentic_power_results(
     concurrencies: Sequence[int],
     *,
     results_python: str | None,
+    expected_cpu: bool = False,
 ) -> int:
     """Stage the AgentX power audit inputs and validate each concurrency's power window.
 
@@ -175,7 +176,7 @@ def collect_agentic_power_results(
 
     if step := validate_agentic_power(
         logs_dir, workspace, result_filename, producer_sha, concurrencies,
-        results_python=results_python, require_power=True,
+        results_python=results_python, require_power=True, expected_cpu=expected_cpu,
     ):  # fmt: skip
         rc = step
     return rc
@@ -190,10 +191,12 @@ def validate_agentic_power(
     *,
     results_python: str | None,
     require_power: bool,
+    expected_cpu: bool = False,
 ) -> int:
     """Run the AgentX power adapter for every concurrency; return the last failing code.
 
-    With ``require_power`` a concurrency without a valid power window fails.
+    With ``require_power`` a concurrency without a valid power window fails, and with
+    ``expected_cpu`` one without a valid Grace CPU leg fails too.
     """
     logs_dir = logs_dir.resolve()
     workspace = workspace.resolve()
@@ -207,7 +210,10 @@ def validate_agentic_power(
             results_python, workspace, logs_dir,
             logs_dir / "agentic" / f"conc_{concurrency}",
             workspace / f"{result_filename}_conc{concurrency}.json",
-            producer_sha, require_power=require_power,
+            producer_sha,
+            "--audit-source", f"LOGS/agentic/conc_{concurrency}/power_validation.json",
+            *(["--expected-cpu-power"] if expected_cpu else []),
+            require_power=require_power,
         ):  # fmt: skip
             rc = step
     return rc

inferencex-e2e/infx/launch/drivers/srt/__init__.py

@@ -163,14 +163,16 @@ def run_multinode(launch: Launch) -> int:
     )
     install_srtctl(run, checkout, python=system_python)
     config.write_lane_config(run, lane, checkout, decision, model_paths)
-    if rc := run_setup(run, checkout):
+    if rc := run_setup(run, checkout, cpu_power_exporter=decision.cpu):
         return rc
     infmax = compute_workspace(run, checkout, shared=shared)
     run.env["INFMAX_WORKSPACE"] = str(infmax)
 
     recipe = str(checkout.root / BOUND_RECIPE)
-    power_port, client_env = config.binder_inputs(run.cluster, run.srt, lane, request, decision)
-    if rc := submit.bind_recipe(run, checkout, staged, recipe, power_port, client_env):
+    power_port, cpu_source, client_env = config.binder_inputs(
+        run.cluster, run.srt, lane, request, decision
+    )
+    if rc := submit.bind_recipe(run, checkout, staged, recipe, power_port, cpu_source, client_env):
         return rc
     prepare_recipe(Path(recipe), srtctl_job_name(request.runner_name), run.srt.dist_timeout_s)
     arguments = submit.multinode_arguments(run, lane, recipe, overrides, preflight=preflight)

inferencex-e2e/infx/launch/drivers/srt/checkout.py

@@ -18,6 +18,9 @@
 from infx.launch.drivers.srt.run import SrtRun, require
 
 SUBMODULE = Path("utils/srt-slurm")
+# The cpu-power-exporter release asset ``make setup`` installs for a job that scrapes Grace CPU
+# power; it ships with the srt-slurm release the submodule pins, so bump both together.
+CPU_POWER_EXPORTER_RELEASE = "v2.51.1"
 PATCHES = Path("runners/srt-slurm/patches")
 SETUP_LOG = "srt-setup.log"
 SETUP_ATTEMPTS = 5
@@ -173,12 +176,17 @@ def _discard_corrupt_archives(configs: Path) -> bool:
     return discarded
 
 
-def run_setup(run: SrtRun, checkout: Checkout) -> int:
-    """Run ``make setup`` in the checkout, logging to SETUP_LOG; return its exit code."""
+def run_setup(run: SrtRun, checkout: Checkout, *, cpu_power_exporter: bool = False) -> int:
+    """Run ``make setup`` in the checkout, logging to SETUP_LOG; return its exit code.
+
+    ``cpu_power_exporter`` pins the cpu-power-exporter release, which makes its download fatal.
+    """
     log = run.workspace / SETUP_LOG
     for attempt in range(1, SETUP_ATTEMPTS + 1):
         print(f"Setting up srt-slurm, attempt {attempt} (details: {SETUP_LOG})", flush=True)
         argv = ["make", "setup", f"ARCH={run.cluster.arch}"]
+        if cpu_power_exporter:
+            argv.append(f"CPU_POWER_EXPORTER_RELEASE={CPU_POWER_EXPORTER_RELEASE}")
         rc = _run_logged(argv, log, env=run.env, cwd=checkout.root)
         if rc == 0:
             print("srt-slurm setup complete", flush=True)

inferencex-e2e/infx/launch/drivers/srt/collect.py

@@ -181,11 +181,21 @@ def stage_logs_on_exit() -> None:
         python = request.inferencex_results_python
         if power.agentx:
             power_rc = collect_agentic_power_results(
-                status, job.id, logs, infmax, *audit, results_python=python
+                status,
+                job.id,
+                logs,
+                infmax,
+                *audit,
+                results_python=python,
+                expected_cpu=power.cpu,
             )
         else:
             power_rc = validate_agentic_power(
-                logs, *audit, results_python=python, require_power=request.require_power
+                logs,
+                *audit,
+                results_python=python,
+                require_power=request.require_power,
+                expected_cpu=power.cpu,
             )
         if power_rc:
             print(