diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 93ff5ce300..86fadeb6c3 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -48,7 +48,7 @@ gh api -X POST \ -f 'inputs[duration-override]=' ``` -The top-level `ref` selects the workflow definition and is normally `main`. `inputs[ref]` selects the repository revision under test. Direct config dispatches set `inputs[generate-cli-command]` with paths relative to the selected checkout's `inferencex-e2e/` directory. Trusted changelog-driven dispatches instead set both `inputs[changelog-base-ref]` and `inputs[changelog-head-ref]`. `duration-override` replaces per-config seconds, and `require-power` makes invalid multi-node power telemetry fatal. +The top-level `ref` selects the workflow definition and is normally `main`. `inputs[ref]` selects the repository revision under test. Direct config dispatches set `inputs[generate-cli-command]` with paths relative to the selected checkout's `inferencex-e2e/` directory. Trusted changelog-driven dispatches instead set both `inputs[changelog-base-ref]` and `inputs[changelog-head-ref]`. `duration-override` replaces per-config seconds, and `require-power` makes invalid power telemetry fatal for multi-node jobs and native single-node throughput and AgentX jobs; single-node matrix rows cannot set it. For AgentX preflight, add `-F 'inputs[agentx-fast]=true'`. Official runs use 10 warmup requests per lane and a one-hour profile. @@ -68,15 +68,15 @@ Eval selection, the `--no-evals` / `--evals-only` / `--all-evals` flags, and cha ## Power telemetry -Multinode srt-slurm results may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`. Single-node results carry no power fields until srt-slurm telemetry covers those lanes. +Multinode srt-slurm results, and single-node fixed-sequence and AgentX results with a retained native telemetry package, may include `power_valid`, `avg_power_w`, `avg_total_gpu_power_w`, `total_gpu_energy_j`, and joules per query/input/output/total token. Invalid telemetry records `power_valid: 0` without energy metrics and fails only with `REQUIRE_POWER=1`. Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count. Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. -A multinode master row with `power: true` enables official energy collection: the launcher merges `inferencex-e2e/configs/srt-recipes/telemetry-dcgm.yaml` into its srt-slurm recipe. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated. +A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment; native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated. -Power audit artifacts are named `power_audit_` and contain the multinode `power_validation__*.json` sidecars. They are uploaded even when validation fails. +Power audit artifacts are named `power_audit_` and contain the `power_validation*.json` sidecars plus, for native telemetry jobs, `LOGS/power/` (`samples.csv` v3 with `temperature_c`, `manifest.json`, `windows/`). They are uploaded even when validation fails. ## Result artifacts and metrics diff --git a/.github/workflows/benchmark-tmpl.yml b/.github/workflows/benchmark-tmpl.yml index 0a3e0c100b..b9b5eed3ee 100644 --- a/.github/workflows/benchmark-tmpl.yml +++ b/.github/workflows/benchmark-tmpl.yml @@ -75,6 +75,11 @@ on: required: false type: string default: "" + require-power: + description: "Fail result processing when GPU power telemetry is invalid" + type: boolean + required: false + default: false env: PYTHONPATH: ${{ github.workspace }}/inferencex-e2e INFERENCEX_E2E_ROOT: ${{ github.workspace }} @@ -86,6 +91,7 @@ env: SALLOC_TIME_LIMIT: '480' KEEP_LOGS: '0' IS_MULTINODE: 'false' + REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }} RANDOM_RANGE_RATIO: 0.8 HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }} HF_HUB_CACHE: '/mnt/hf_hub_cache/' @@ -319,6 +325,10 @@ jobs: echo "no raw result to process: $RESULT_FILENAME.json" >&2 exit 1 fi + # A native power job's launcher stamp names the telemetry producer + if [ -f power-producer-sha.txt ]; then + export POWER_PRODUCER_SHA="$(cat power-producer-sha.txt)" + fi "$INFERENCEX_RESULTS_PYTHON" -P -m infx.results.fixed_sequence - name: Upload result @@ -454,6 +464,23 @@ jobs: ${{ env.INFERENCEX_E2E_ROOT }}/srt-setup.log if-no-files-found: ignore + - name: Upload power audit bundle + if: ${{ always() && !inputs.eval-only }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: power_audit_${{ env.RESULT_FILENAME }} + path: | + ${{ env.INFERENCEX_E2E_ROOT }}/${{ env.RESULT_FILENAME }}.json + ${{ env.INFERENCEX_E2E_ROOT }}/agg_${{ env.RESULT_FILENAME }}.json + ${{ env.INFERENCEX_E2E_ROOT }}/power_validation_${{ env.RESULT_FILENAME }}.json + ${{ env.INFERENCEX_E2E_ROOT }}/results/agentic_power_timezone_offset.txt + ${{ env.INFERENCEX_E2E_ROOT }}/LOGS/power/** + ${{ env.INFERENCEX_E2E_ROOT }}/LOGS/agentic/**/agentic_power_concurrency_*.json + ${{ env.INFERENCEX_E2E_ROOT }}/LOGS/${{ env.RESULT_FILENAME }}.json + ${{ env.INFERENCEX_E2E_ROOT }}/power-producer-sha.txt + ${{ env.INFERENCEX_E2E_ROOT }}/exporter-image.sha256 + if-no-files-found: ignore + - name: Upload eval results (if any) id: upload-eval if: ${{ always() && (env.RUN_EVAL == 'true' || inputs.eval-only) }} diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml index 1ad5121caa..556d856b39 100644 --- a/.github/workflows/e2e-tests.yml +++ b/.github/workflows/e2e-tests.yml @@ -395,6 +395,7 @@ jobs: secrets: INFERENCEX_OFFICIAL_RO_HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }} with: + require-power: ${{ inputs.require-power }} config: ${{ toJSON(matrix.config) }} klaud-run: ${{ inputs.klaud-run }} runner: ${{ matrix.config.runner }} @@ -505,6 +506,7 @@ jobs: secrets: INFERENCEX_OFFICIAL_RO_HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }} with: + require-power: ${{ inputs.require-power }} config: ${{ toJSON(matrix.config) }} klaud-run: ${{ inputs.klaud-run }} runner: ${{ matrix.config.runner }} diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md index 91acdc5732..ca6e82049e 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md @@ -40,8 +40,8 @@ Recipes use `schema: 2`, `engine`, and `roles`. Each worker role owns its node c | `roles.prefill.args.tp-size` (SGLang) | `prefill.tp` | | `roles.prefill.args.ep-size` (SGLang) | `prefill.ep` | | `roles.prefill.args.enable-dp-attention` | `prefill.dp-attn` | -| `benchmark.concurrencies` | `conc-list` (bound for `power: true` rows) | -| `telemetry:` | search-space `power: true` | +| `benchmark.concurrencies` | `conc-list` (bound for multi-node `power: true` rows) or the native single-node point's concurrency | +| `telemetry:` | multi-node search-space `power: true`; shared native telemetry for single-node benchmarks | | Recipe directory, relative to this tree | entry-level `srt-recipe-dir` | | Recipe file, optionally with a `base`, `override_`, or `zip_override_[]` selector | search-space `srt-recipe`; `eval-srt-recipe` for eval-only real verification | @@ -53,7 +53,7 @@ All referenced recipes must be checked in: srt-slurm 2 ships curated examples in Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound: -1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. +1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches. 2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning. 3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:`, `model.container: `, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require` (GB200 and GB300 Kimi-K3 AgentX), so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments. 4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/