Skip to content

[None][perf] Remove spurious sync in sparse fmha forward#16771

Merged
pcicotti merged 1 commit into
NVIDIA:feat/m3_with_msafrom
brb-nv:user/brb/remove-spurious-fmha-sync
Jul 23, 2026
Merged

[None][perf] Remove spurious sync in sparse fmha forward#16771
pcicotti merged 1 commit into
NVIDIA:feat/m3_with_msafrom
brb-nv:user/brb/remove-spurious-fmha-sync

Conversation

@brb-nv

@brb-nv brb-nv commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Description

This MR removes spurious syncs in sparse fmha forward
Before: Blocking D2H copy due to kv_segment_lens living on GPU
After: kv_segment_lens lives on CPU as passed by TRTLLM.

No changes to MSA repo.

For ISL=8192 and OSL=1 @ c=1, request latency in ms:

Before:

[Latency] P50    : 257.6939
[Latency] P90    : 261.0175
[Latency] P95    : 261.7910
[Latency] P99    : 262.7895
[Latency] MINIMUM: 256.4940
[Latency] MAXIMUM: 262.7895
[Latency] AVERAGE: 258.1691

After:

[Latency] P50    : 249.3527
[Latency] P90    : 250.5275
[Latency] P95    : 250.7692
[Latency] P99    : 252.6669
[Latency] MINIMUM: 248.4402
[Latency] MAXIMUM: 252.6669
[Latency] AVERAGE: 249.5198

Test Coverage

$ pytest tests/integration/defs/accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=True] -s -v

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
@brb-nv
brb-nv requested a review from a team as a code owner July 23, 2026 03:10
@brb-nv
brb-nv requested review from PerkzZheng and kris1025 July 23, 2026 03:11
@brb-nv
brb-nv requested review from pcicotti, peihu-nv and zheyuf July 23, 2026 03:11
@pcicotti
pcicotti merged commit 3cd0e3d into NVIDIA:feat/m3_with_msa Jul 23, 2026
8 checks passed
brb-nv added a commit to brb-nv/TensorRT-LLM that referenced this pull request Jul 23, 2026
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants