[https://nvbugs/6198785][fix] Unify phase-1 CUDA graph cleanup

[Mgluhovskoi]

Description

Fix a CUDA unspecified launch failure during DeepSeek-V3-Lite initialization when piecewise and outer CUDA graphs are captured during KV-cache memory profiling.

Root cause

The failing snapshot released whole-model CUDA graphs during Phase-1 executor shutdown, but piecewise graph runners were not connected to that lifecycle. They could retain captures, outputs, address metadata, and private pool handles from the profiling allocation epoch, then replay stale state after final KV-cache allocation.

Current main connects piecewise runners to the synchronized PyExecutor.shutdown() path, which releases graphs before resource managers. The creator still retained a legacy second, conditionally gated graph release after manager teardown. Besides obscuring ownership, that duplicate release could rotate already-reset private pool handles a second time.

Changes

• Keep synchronized PyExecutor.shutdown() as the single owner of Phase-1 graph destruction and remove the creator's legacy second release.
• Continue discarding Phase-1 attention metadata before final KV-cache construction.
• Add focused coverage for idempotent runner cleanup, backend-wide cleanup, private-pool rotation, the no-runner path, and graph-before-resource-manager shutdown ordering.
• Remove the two NVBug 6198785 integration-test waivers so both reported configurations return to CI coverage.

Validation

• Focused unit tests on the revised code: 4 passed in 6 consecutive executions (24 total test passes).
• Original 4x H200 TP4 compile/overlap reproducer after adding piecewise cleanup: 4 consecutive passes.
• 4x H200 compile regression with overlap disabled: passed.
• Reported 4x H200 non-compile outer-graph configuration: 2 consecutive final-patch passes.
• Python compilation, Ruff checks/formatting, and git diff --check: passed.
• All eight H200 volatile corrected and uncorrected ECC counters remained zero.

Impact

CUDA graphs captured during memory profiling are invalidated by one synchronized lifecycle owner and safely recaptured after final KV-cache allocation. Redundant private-pool rotation is avoided. There are no user-facing API changes.

Dev Engineer Review

• Fixed DeepSeek-V3-Lite CUDA unspecified launch failure during KV-cache initialization by adjusting CUDA-graph/pool cleanup ownership: PyExecutor.shutdown() is the single owner of Phase-1 CUDA graph destruction, removing the creator-side duplicate release that could leave stale piecewise/outer graph state.
• Preserved Phase-1 attention metadata cleanup before final KV-cache construction: after kv_cache_creator.configure_kv_cache_capacity(py_executor), the code no longer conditionally releases Phase-1 CUDA graph pools and instead clears remaining eng.attn_metadata for model_engine and draft_model_engine.
• Removed two obsolete nvbugs/6198785 waivers for TestDeepSeekV3Lite::test_bfloat16_4gpus.
• No public API changes; validation reported included focused unit tests, H200 reproducer/regression tests, Python compilation, Ruff, formatting, and diff checks.

QA Engineer Review

• Test-list-only change: modified tests/integration/test_lists/waives.txt.
• Removed SKIP entries for accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus:
  • tp4, mtp_nextn=0, attention_dp=False, cuda_graph=True
  • With overlap_scheduler=False/torch_compile=False and overlap_scheduler=True/torch_compile=True (two waivers total), both referencing nvbugs/6198785.
• No test-db/ or qa/ files were modified.
• Verdict: needs follow-up because CBTS coverage data is unavailable.

[Mgluhovskoi]

/bot run --disable-fail-fast

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8ebbcc20-e508-47f9-aac3-4ef0538d7b85

📥 Commits

Reviewing files that changed from the base of the PR and between 6e0ef83 and 0b8fdca.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (2)

• tests/integration/test_lists/waives.txt
• tensorrt_llm/_torch/pyexecutor/py_executor_creator.py

---

Walkthrough

The estimating-KV-cache cleanup path now relies on configure_kv_cache_capacity for CUDA graph release and clears profiling metadata afterward. Two DeepSeek V3 Lite integration test waivers were removed.

Changes

KV cache cleanup and validation

Layer / File(s)	Summary
Executor cleanup and integration validation tensorrt_llm/_torch/pyexecutor/py_executor_creator.py	The estimating-KV-cache path removes explicit Phase-1 CUDA graph release and clears profiling attention metadata.
Integration waiver updates tests/integration/test_lists/waives.txt	Two DeepSeek V3 Lite test skip waivers referencing nvbugs/6198785 are removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: niukuo, mzweilz, qijune, schetlur-nv, yihuilu512

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific, concise, and matches the main fix: phase-1 CUDA graph cleanup for nvbugs/6198785.
Description check	✅ Passed	The description follows the template well and includes clear problem, root cause, changes, validation, and impact sections.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #61123 [ run ] triggered by Bot. Commit: 26eada4 Link to invocation

[tensorrt-cicd]

PR_Github #61123 [ run ] completed with state SUCCESS. Commit: 26eada4
/LLM/main/L0_MergeRequest_PR pipeline #49377 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61331 [ run ] triggered by Bot. Commit: 6e0ef83 Link to invocation

[tensorrt-cicd]

PR_Github #61331 [ run ] completed with state FAILURE. Commit: 6e0ef83
/LLM/main/L0_MergeRequest_PR pipeline #49560 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61369 [ run ] triggered by Bot. Commit: 6e0ef83 Link to invocation

[tensorrt-cicd]

PR_Github #61369 [ run ] completed with state FAILURE. Commit: 6e0ef83
/LLM/main/L0_MergeRequest_PR pipeline #49597 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61392 [ run ] triggered by Bot. Commit: 0b8fdca Link to invocation

[tensorrt-cicd]

PR_Github #61392 [ run ] completed with state SUCCESS. Commit: 0b8fdca
/LLM/main/L0_MergeRequest_PR pipeline #49619 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61518 [ run ] triggered by Bot. Commit: 0b8fdca Link to invocation

[tensorrt-cicd]

PR_Github #61518 [ run ] completed with state SUCCESS. Commit: 0b8fdca
/LLM/main/L0_MergeRequest_PR pipeline #49734 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61596 [ run ] triggered by Bot. Commit: 0b8fdca Link to invocation

[tensorrt-cicd]

PR_Github #61596 [ run ] completed with state SUCCESS. Commit: 0b8fdca
/LLM/main/L0_MergeRequest_PR pipeline #49805 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Mgluhovskoi]

/bot run

[tensorrt-cicd]

PR_Github #61634 [ run ] triggered by Bot. Commit: 0b8fdca Link to invocation

[tensorrt-cicd]

PR_Github #61634 [ run ] completed with state SUCCESS. Commit: 0b8fdca
/LLM/main/L0_MergeRequest_PR pipeline #49841 completed with status: 'SUCCESS'

CI Report

Link to invocation