DRAFT: feat(power): vendor-agnostic power/energy monitoring

[viraatc]

What

First-class, opt-in, vendor-neutral power/energy monitoring. When settings.power.source is set, a best-effort sidecar streams telemetry during the performance phase; at finalization the trace is windowed, integrated into energy, and written to a sibling power.json (+ a report.txt section). Report / result_summary.json stay snapshot-pure — mirroring the profiling precedent.

Design: general framework + pluggable sources

The core PowerConfig is vendor-neutral — it knows nothing about GPUs/Prometheus. A source is a tiny registered builder (@power_source("name")) that turns config into a sidecar argv + parse spec. nvidia_smi is just the first reference source; all NVIDIA-specific bits live in its builder, not the schema.

from inference_endpoint.power import power_source, ResolvedSource

@power_source("redfish")                     # users add their own, no core edits
def _build(cfg):
    return ResolvedSource(argv=["redfish-power", cfg.options["bmc"]], fmt="jsonl",
                          value_kind="power_w", ts_field="ts", value_field="value",
                          label_field="label", csv_header=False)

• Minimal exposed config: core = source, interval_s, options (+ 3 hidden advanced). Per-source settings live in options (nvidia_smi: {gpu_indices}, prometheus: {url, query}, command: {argv}); each builder owns + validates its own keys.
• Built-ins: nvidia_smi, prometheus (DCGM exporter for remote GPUs), and command (any program emitting canonical JSONL — the process boundary is the plugin API for non-Python users).
• Disabled = zero footprint: source=None serializes to {}.

Robustness / correctness

• Sidecar via plain subprocess.Popen (not ServiceLauncher, which aborts the run on no-ready-signal); own process group, SIGTERM→SIGKILL, PR_SET_PDEATHSIG backstop, stdbuf line-buffering. Every path swallows errors — a broken collector never fails or perturbs the benchmark.
• Windowed to the performance phase via the existing monotonic→wall anchor; trapezoid integration (power_w) or counter-delta (energy_j).
• energy_per_output_token_j emitted only when tokens and energy share the same window, else suppressed with a note (no mixed-denominator metric).

Files

New src/inference_endpoint/power/ (sources.py registry, collector.py, parse.py, window.py, render.py, prom_poll.py); settings.power in config/schema.py; integration in commands/benchmark/execute.py. Docs: docs/POWER_MONITORING.md; example: examples/11_Power_Monitoring/.

Test plan

tests/unit/power/ — config/serialization, plugin registration, unknown-source error, nvidia/JSONL parsing (+ malformed drops), trapezoid + counter integration, J/token consistency guard, status precedence, end-to-end sidecar. uv run pytest tests/unit/{power,config,commands} → 231 passed; pre-commit run --all-files green.

Follow-ups (v2)

Multi-source totals + scope-aware double-count guards; entry-point auto-discovery (intentionally omitted — the in-process registry + command cover today's needs). Tracked alongside the execute.py refactor in #384.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces vendor-agnostic power and energy monitoring telemetry to benchmark runs. It launches a sidecar process during the performance phase to stream power samples from built-in sources (nvidia_smi, prometheus) or custom commands, subsequently writing the integrated energy metrics to a sibling power.json file and report.txt. The review feedback highlights several robustness and compatibility issues: potential uncaught exceptions in parse.py when handling malformed JSONL or non-integer CSV fields; a design bug in templates.py where headerless CSV commands fail with a KeyError; an incomplete counter-reset check in window.py that can miss intermediate resets; a Windows compatibility issue in collector.py due to platform-specific os functions; and a potential TypeError in prom_poll.py if a Prometheus series contains a null metric.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[viraatc]

Note: this PR threads power start/stop hooks + a payload builder into the already-large commands/benchmark/execute.py (~1270 lines; _run_benchmark_async ~370). That growth-per-sidecar smell is tracked separately in #384 (refactor execute.py behind a uniform run-hook seam). Keeping this PR additive and behavior-preserving; the decomposition is intentionally deferred to #384.