Repository navigation
[AgentX] add HiCache offload to Muse Spark Open H100 / [AgentX] 为 Muse Spark Open H100 添加 HiCache 卸载 - #4181
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Add a HiCache host-DRAM KV offload arm at concurrency 2, 4 and 8 with 160 GB per rank, alongside the GPU-resident sweep; the few-GB GPU KV pool left by 67 GB of weights per H100 evicts agentic prefixes between turns." | ||
| - "Keep the batch-8 cap on running requests and decode graphs from the GPU-resident sweep." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX |
There was a problem hiding this comment.
🔴 The perf-changelog check will fail on this PR, and merge-time link filling will refuse this entry. The new block's pr-link ends in pull/XXXX, but PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:22 accepts only XXX or .../pull/XXX. Fix: use the real PR link https://github.com/SemiAnalysisAI/InferenceX/pull/<N>, or the exact placeholder https://github.com/SemiAnalysisAI/InferenceX/pull/XXX so prepare_perf_changelog_merge.py can fill it in.
Why this was flagged
compare_entries calls validate_added_pr_link (inferencex-e2e/infx/workflows/validate_perf_changelog.py:139-150) for each appended entry. On a PR run it raises ChangelogValidationError("new PR entry must use ... or an XXX placeholder; found ...") unless the link is in PR_LINK_PLACEHOLDERS ({"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}) or equals the PR's own URL. pull/XXXX, with four X's, is neither, so the changelog validation job fails. prepare_perf_changelog_merge.py:108 also raises "appended entry ... has unexpected pr-link" for the same link, so the merge flow does not swap it for the real URL. As a result the entry that triggers result collection for musespark-fp4-h100-sglang-agentic-dflash cannot land as written.
Verification: Trigger: prepare_perf_changelog_merge runs at merge time. The appended entry's pr-link ends in pull/XXXX, which has four X's. PR_LINK_PLACEHOLDERS (validate_perf_changelog.py:22-24) contains only "XXX" and ".../pull/XXX". prepare_perf_changelog_merge.py:108 raises "appended entry ... has unexpected pr-link", so automatic link filling at merge refuses this entry.
…增 Muse Spark Open H100 并发 1–64 扫描 (#4149) * feat: add Muse Spark Open H100 capacity check * chore: link H100 Muse Spark sweep to PR 4149 * fix: reserve H100 memory for Muse Spark KV cache * fix: fit Muse Spark prefill and long-context KV on H100 * fix: cap H100 Muse Spark decode graphs at 8 for vision headroom c48 started with 0.29 GB free per GPU and OOMed in the vision encoder's all-gather on the first image prefill. * fix: keep H100 DFlash sampling valid above graph capacity 启用 eager draft sampler，保留解码图批量上限 8 和完整并发扫描。 * fix: cap H100 Muse Spark at 8 running requests, sweep c1-c16 Batches above the batch-8 decode graphs run eager and OOM in fused MoE with ~1 GB free per GPU (c48, run 38870885984). Drop c24-c64. * [AgentX] add HiCache offload to Muse Spark Open H100 / [AgentX] 为 Muse Spark Open H100 添加 HiCache 卸载 (#4181) * feat: add HiCache offload arm to H100 Muse Spark at c2-c8 * chore: link H100 HiCache entry to PR 4181 --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
1 participant
Stacked on #4149; retarget to main once it merges.
Adds a host-DRAM KV offload arm (HiCache, 160 GB per rank, dram-utilization 0.80) at concurrency 2, 4 and 8, alongside #4149's GPU-resident c1–c16 sweep.
Why: at TP8 each H100 holds ~67 GB of target weights, leaving a GPU KV pool of only a few GB, so agentic prefixes are likely evicted between turns and recomputed. HiCache keeps them in host DRAM (h100-dsxe has ~1.83 TiB). The batch-8 cap on running requests and decode graphs from #4149 stays, since eager batches above it OOM in the MoE.
Risk: HiCache's GPU staging buffers have to fit in the ~1 GB/GPU headroom at mem-fraction-static 0.97; the sweep will show whether they do.