Repository navigation
[PowerX] use the configurable AMD exporter and native power telemetry - #3781
Merged
Merged
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
Contributor
|
Claude finished @edwingao28's task in 1m 49s —— View job PR #3781 review: one blocking changelog issue
🔴 Blocking: The launch and power code had no blocking issues:
One non-blocking note: the multinode eval upload no longer includes |
cquil11
reviewed
Oct 7, 2026
|
|
||
| - config-keys: | ||
| - "*" | ||
| description: |
Collaborator
There was a problem hiding this comment.
one perf changelog entry as discussed earlier
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Collaborator
|
moving srt slurm submodule bump to separate pr and rebasing this one |
cquil11
force-pushed
the
feat/powerx-amd-upstream-572
branch
from
October 7, 2026 17:54
4a64d00 to
4172b89
Compare
cquil11
requested review from
Ankur-singh,
chunfangamd and
hbarclay
as code owners
October 7, 2026 17:54
cquil11
force-pushed
the
feat/powerx-amd-upstream-572
branch
4 times, most recently
from
October 7, 2026 17:54
4a64d00 to
4172b89
Compare
cquil11
changed the title
Contributor
There was a problem hiding this comment.
Beyond the inline finding, I also checked the dropped matrix.config['require-power'] fallback for test-sweep-single-node (e2e-tests.yml:509): single-node fixed-sequence recipes are validated to reject a per-recipe require-power field (infx/matrix/generate.py rejects it outside the one scoped multinode fixed-sequence), so only the global inputs.require-power flag can ever apply there — the missing fallback is not a behavior change.
Extended reasoning...
The PR re-pins srt-slurm to an upstream producer, reworks single-node power staging/validation in collect.py, and touches several CI workflow wiring points for require-power propagation; no auth/crypto surface is involved. A confirmed finding (collect.py:76) about a new recursive copytree that can turn log-staging I/O errors into benchmark-run failures is being posted inline, so approval is not warranted regardless. I independently verified one of the ruled-out candidates (the single-node require-power fallback removal) against infx/matrix/generate.py's validation, which rejects a per-recipe require-power flag for single-node fixed-sequence configs, confirming it is a non-issue.
4 of 11 tasks
cquil11
reviewed
Oct 7, 2026
functionstackx
added this pull request to stack #3844
October 8, 2026 22:11
Contributor
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
edwingao28
pushed a commit
that referenced
this pull request
Oct 12, 2026
cquil11
force-pushed
the
feat/powerx-amd-upstream-572
branch
from
October 14, 2026 04:56
a88e001 to
6dc48b1
Compare
中文：将主分支合并到 AMD 功耗遥测 PR。
The launcher no longer re-reads the bound recipe, patches its request, or writes $GITHUB_ENV. REQUIRE_POWER alone sets telemetry.required, so a single-node fragment may not set it, and result processing takes the producer SHA from the launcher stamp as the multi-node workflow does.
select_recipe already omits the native telemetry block for eval-only points, so the binder no longer rewrites telemetry.enabled.
Single-node and multi-node AgentX run the adapter through one helper, and the single-node adapter writes its sidecar into the workspace instead of the launcher copying it there.
The aggregate already records num_gpus from the same environment as GPU_COUNT, so the adapter drops --expected-num-gpus.
cquil11
approved these changes
Oct 14, 2026
adibarra
added a commit
that referenced
this pull request
Oct 14, 2026
Main's native single-node power telemetry (#3781) adds power_invalid_reasons and power_audit to single-node AgentX rows, which the AgentX row model rejected as unknown extras, so every such row would be quarantined. The AgentX models now accept both optional fields as the fixed-sequence models do, and the committed JSON Schema is regenerated.
adibarra
added a commit
that referenced
this pull request
Oct 14, 2026
Main's native single-node power telemetry (#3781) stages the cluster's GPU exporter for every non-eval single-node point and stops preparation when the cluster record has none, so the DRAM-cap driver test's ad hoc cluster now needs default_gpu_exporter to reach submission.
adibarra
added a commit
that referenced
this pull request
Oct 14, 2026
Main's native single-node power telemetry (#3781) added two ways a point fails on power: a required power package that cannot be staged, and a failed AgentX power adapter run under REQUIRE_POWER. Both now record a typed job-event error (ArtifactError, PowerValidationFailed) in the collect stage instead of ending as a bare ExitStatus or CleanupFailed. Console output is unchanged.
adibarra
added a commit
that referenced
this pull request
Oct 16, 2026
…it host pools to measured DRAM (#3804) * fix(launch): cap single-node srun steps at their GPUs' share of node DRAM Port of #3615 onto the current srt driver. Every single-node srun step gets --mem = available-cpu-dram-mib / gpus-per-node * GPU_COUNT through SRT_SRUN_OPTIONS, so a point using half the node can no longer take more than half its host DRAM. The sbatch job requests the whole available-cpu-dram-mib only when slurm.srt-slurm.single-node-exclusive is true; on shared nodes it requests the same GPU share, so it leaves the rest of the node to other jobs. Clusters without available-cpu-dram-mib keep their previous behavior. Co-authored-by: Cam Quilici <cjquilici@gmail.com> * fix(agentx): keep single-node DRAM pools 32 GiB under the srun step cap Three single-node AgentX DRAM arms pinned host KV pools at or within a few GiB of their TP4 srun step's new --mem share of node DRAM: - minimaxm3-fp4-b200-vllm-agentic-mtp TP4 SimpleCPU: dram-utilization 1.0 -> 0.96, cpu_bytes_to_use and TOTAL_CPU_DRAM_GB 1082 -> 1038 GB (headroom 0.08 -> 41.1 GiB under 1,031,960 MiB). - minimaxm3-fp4-b200-trtllm-agentic-mtp TP4: per-rank host_cache_size 250 -> 240 GiB (headroom 7.8 -> 47.8 GiB under 1,031,960 MiB). - qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 conc 64-80: per-rank hicache-size 400 -> 395 GB (headroom 21.5 -> 40.1 GiB under 1,547,890 MiB). Also trims the MiniMax-M3 B200 vLLM master-config comment to its rationale. * fix(srt): request --mem=0 for exclusive single-node jobs Slurm's RealMemory on b200-nscale and b300-dsxe is below available-cpu-dram-mib, so sbatch rejected the whole-node --mem. Exclusive jobs now take all the memory Slurm has, and a step that spans the whole node uses the job's memory instead of a share cap. * fix(runners): set available-cpu-dram-mib to what Slurm can allocate Measured on each cluster's GPU nodes as the lower of RealMemory - MemSpecLimit and MemTotal less 16 GiB: h100-dgxc and b300-dsxe reserve 128 GiB through MemSpecLimit, b200-nscale configures RealMemory 15.5 GiB under MemTotal, and the AMD clusters set RealMemory to MemTotal. * fix(recipes): match TOTAL_CPU_DRAM_GB to measured node DRAM The lower available-cpu-dram-mib on h100-dgxc, b200-nscale and mi300x-amd lowers the matrix total-cpu-dram-gb of their single-node DRAM offload points, and the single-node binder rejected every recipe variant still naming the old TOTAL_CPU_DRAM_GB. Each variant now names the new budget, and host pools sized from it follow their recipe's rule: - DSv4 B200 vLLM: SimpleCPU bytes per rank = budget / 8, 243.375 -> 241.5 GB. - MiniMax-M3 B200 vLLM: cpu_bytes_to_use = budget, 1038 -> 1030 GB. - MiniMax-M3 H100 vLLM: Mooncake segment = (budget GiB - 414) / 8 - 30, 119 -> 105 GB. - MiniMax-M3 MI300X vLLM: LMCache L1 = 8 x budget / 8, 1296 -> 1280 GB. - Qwen3.5 B200 SGLang: hicache-size = (budget - TP) * 15 / TP / 31, 104 -> 103. - Qwen3.5 H100 SGLang: hicache-size = budget / TP8 / 2 pools, 108 -> 100. GLM-5.2 B200's 169 GB/rank pool (~1,662 GB with the DSA indexer) and the MiniMax-M3 B200 TRT-LLM per-rank pools, which are not derived from the budget, still fit. Every pool stays 32+ GiB under its srun step's share of node DRAM. * fix(recipes): keep single-node DRAM pools within the 3 TB policy share Validation OOM-killed the Qwen3.5 MI355X FP8 TP4 CONC 64 HiCache point inside its srun step cap: 4 x 395 GB per-rank pools also exceeded the 3 TB policy share of four GPUs (1500 GB). The MiniMax-M3 B300 TRT-LLM TP2/TP4 host caches (388.6 GB per rank) exceeded theirs too (750/1500 GB). - Qwen3.5 MI355X FP8: dram-utilization 0.80 -> 1.0, hicache-size at CONC 64-80 395 -> 374 GB per rank (budget 1199 -> 1499 GB). - MiniMax-M3 B300 TRT-LLM: dram-utilization 0.8 -> 1.0, TP2/TP4 host_cache_size 388554555392 -> 374500000000/374750000000 bytes; TOTAL_CPU_DRAM_GB 599/1199/2399 -> 749/1499/2999. The budgets of two formula-sized pools now name the pool itself, as the refactor stack's '@dram' references expect: MiniMax-M3 MI300X 0.80 -> 0.7972 (TOTAL_CPU_DRAM_GB 1284 -> 1280, the LMCache L1) and MiniMax-M3 H100 0.80 -> 0.449 (1607 -> 902, eight 105 GiB Mooncake segments). * fix(srt): cap single-node steps only on shared nodes A step capped at its GPUs' share of node DRAM leaves no room for the engine and page cache once its host pools fill the share: qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 HiCache failed inside the step cgroup ("Requesting 150.27 GB but only have 15.04 GB free") with its pools at the DRAM policy limit. Exclusive single-node jobs keep requesting --mem=0 and no longer cap their srun steps, so pools must fit the node. On shared nodes the job and its step still get the serving GPUs' share, which pools must fit. * fix(runners): restore h100-dgxc DRAM; earlier value was measured on another cluster 8c9da40 took h100-dgxc's 1,915,904 MiB from login-h100, which fronts the DSXE H100 cluster (partitions batch_1/batch_all, /data), not the DGX Cloud cluster runners.yaml calls h100-dgxc (partition hpc-gpu-1, /mnt/nfs). The 128 GiB MemSpecLimit that commit's message attributes to h100-dgxc is the DSXE cluster's; h100-dgxc was not measured. Restore 2,063,837 MiB and undo the H100 values derived from the wrong one: - Qwen3.5 H100 SGLang: hicache-size 100 -> 108 GB, TOTAL_CPU_DRAM_GB 1607 -> 1731. - MiniMax-M3 H100 vLLM: dram-utilization 0.449 -> 0.80, Mooncake segment 105 -> 119 GB, TOTAL_CPU_DRAM_GB 902 -> 1731. Every h100-dgxc and h100 matrix row and H100 recipe again matches main. CONFIGS.md gives the Slurm rule as how to measure the field, not as what every recorded value is. runners.yaml h100-dgxc has no registered runners; the DSXE H100 runners (cluster:h100-dsxe) have no runners.yaml entry. * refactor(srt): tighten single-node DRAM test, comments and docs Condense the driver's DRAM cap comment, the available-cpu-dram-mib doc and the MiniMax-M3 B200 dram-utilization comment; drop the test comment that repeated the driver's and name the test for all three of its cases; fix the MI300X LMCache recipe comment, whose L1 is now the whole 1280 GB budget rather than shares of 1284 GB. 精简驱动中的 DRAM 上限注释、available-cpu-dram-mib 文档和 MiniMax-M3 B200 的 dram-utilization 注释；删除重复驱动注释的测试注释，并按三种情形重命名该 测试；修正 MI300X LMCache 配方注释：其 L1 现为完整的 1280 GB 预算，而非 1284 GB 的分摊份额。 * chore: drop comments that break the code comment rules Main now keeps infx free of explanatory comments and bans header comments in configs. Remove the driver's DRAM cap comment, which CONFIGS.md already documents, and the MiniMax-M3 B200 vLLM master-config header. * test(srt): give the DRAM-cap test cluster a GPU exporter Main's native single-node power telemetry (#3781) stages the cluster's GPU exporter for every non-eval single-node point and stops preparation when the cluster record has none, so the DRAM-cap driver test's ad hoc cluster now needs default_gpu_exporter to reach submission. * fix(recipes): size minimaxm3 b200 host pools to their whole-node DRAM budget b200-nscale runs single-node jobs whole-node (--mem=0), so the TP4 pools no longer need headroom under a per-GPU step cap. The vLLM SimpleCPU pool and the TRT-LLM per-rank host cache now take the full TP4 budget (dram-utilization 1.0, 1073 GB), and the TRT TP4 rows report the budget their pool actually uses instead of a nominal 0.8 one. --------- Co-authored-by: Cam Quilici <cjquilici@gmail.com>
This was referenced Oct 16, 2026
adibarra
added a commit
that referenced
this pull request
Oct 17, 2026
* refactor(launch): share bench's masking echo and send warnings to stderr The launcher and the container-side bench commands each had an echo() helper, and only the launcher's masked *TOKEN* and *SECRET* values. The masking echo now lives in infx.bench.proc (stdlib-only, Python 3.10), which the launcher already imports; bench call() and tee() also mask the child environment's secrets. Two launcher warnings that went to stdout now go to stderr with the other diagnostics. * feat(launch): record one job_event.json per benchmark job Debugging a failed benchmark job meant reading a multi-hour Actions log plus the server logs. Each `python -m infx.launch run` now writes one record, job_event.json, into the workspace when it exits on any path: the run, job and point identity, image, recipe, cluster, Slurm job and nodes; the launch decisions (path, DCGM power, golden acceptance length, eval flags, KV offloading); seconds per stage (prepare, submit, queue_wait, run, collect); the outcome; and for a failure the first error's stage, type, message, exit code, retriable hint and the log to read first. On GitHub Actions a failure also emits one ::error annotation titled with its stage. The benchmark templates upload the record as job_event_<RESULT_FILENAME>, and collect-results joins the records it can download into events.jsonl in the job_events artifact (python -m infx.results.collect_events). A batch-wrapped launch continues the record of the launch it re-runs. The golden acceptance planner now reports the length it simulates, and the Slurm accounting read includes the job's node list. * fix(ci): collect job events after every benchmark and eval job Collect-results waits only for throughput jobs, so eval-only jobs were missing from events.jsonl and evals-only sweeps produced no job_events artifact. A dedicated collect-job-events template now runs after every sweep job in run-sweep.yml and e2e-tests.yml. Add the Chinese troubleshooting section. * refactor(launch): trim job event duplication and echo-only test assertions finish() reuses exited() for its exit-status fallback, the single-use PREPARE constant is inlined, and a cleanup's name is looked up only when its nonzero result is reported. The srt driver tests stop asserting environment values copied verbatim into the record, a non-negative stage duration check, and an annotation-absence check that could not fail without GITHUB_ACTIONS. * fix(launch): record the power rule's require flag in job events The stack's PowerDecision gained require (the client fails a point whose AgentX power is invalid), and the job event records the decision with asdict, so its Power record must accept the field. * docs: read job event records first when root-causing sweep failures Main removed docs/troubleshooting.md, where this branch described job_event.json. Keep the essentials in the debug-runs skill's root-cause step: where the records are uploaded and collected, what a failure's error fields mean, and one jq query. * feat(launch): record native single-node power failures in the job event Main's native single-node power telemetry (#3781) added two ways a point fails on power: a required power package that cannot be staged, and a failed AgentX power adapter run under REQUIRE_POWER. Both now record a typed job-event error (ArtifactError, PowerValidationFailed) in the collect stage instead of ending as a bare ExitStatus or CleanupFailed. Console output is unchanged. * feat(launch): make the job event answer what ran, where, how long and why it failed Schema v2 of job_event.json: - Slurm accounting times (slurm_submitted_at/started_at/ended_at, queue_s, batch_queue_s) from the one sacct reading; the misleading log-wait queue_wait stage folds into run. - error.detail: the evidence log's likeliest cause line plus its last 20 lines, also named in the ::error annotation. - versions, topology, per-point metrics and GitHub links (job.check_run_id, PR number, test name, ref under test, passed from run-sweep.yml and e2e-tests.yml). - isl/osl are null for AgentX points. - The generator CLI fingerprints every matrix row as changelog plans do. - collect-job-events adds each job's conclusion and failed step and synthesizes a NoLauncherRecord record for benchmark jobs that wrote none. * fix(launch): record a rejected recipe binding with the binder's error The srt-slurm recipe binders (single-node bind_point and multi-node bind_recipe) now stream their stderr to the job log and keep it in srt-bind.log. A failure is recorded as RecipeBindFailed whose message is the binder's last line, with srt-bind.log as evidence and its cause line and tail as detail, instead of a bare ExitStatus. The templates upload srt-bind.log with the server logs.
8 of 10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable native srt-slurm GPU power collection for single-node fixed-sequence and AgentX jobs, including AMD's digest-pinned high-resolution exporter. Publish the native power package and result-side validation report, honor required-power failures, accept temperature samples, and reject packages whose producer recorded failed clock synchronization.
Rebased onto main after Alec's recipe-refactor stack merged. Native telemetry is composed through a shared fragment and bound after variant selection; exporter settings and workspace mounts come from cluster records. Recipe generation validates against each eligible cluster's exporter defaults. Eval-only jobs disable telemetry, and existing multi-node power policies remain in place.
Retains NVIDIA/srt-slurm#573 as a local patch to restrict worker-node samples to the job's participating GPUs. Includes the cleanup previously merged through #3798.
Replayed five saved hardware points through the refactored consumers: MI355X fixed-sequence and AgentX, plus B200 TP2/TP4 fixed-sequence and AgentX. All passed strict validation with every published power metric identical to the saved hardware results and original PR implementation.
Smoke runs on
d357d8b3b4:qwen3.5-fp8-mi355x-sglangqwen3.5-fp8-mi355x-sglang-agentic-mtpqwen3.5-fp4-b200-sglangminimaxm3-fp4-b200-trtllm-agentic-mtp