[TRTLLMINF-133][infra] Move check testlist stage earlier in CI by tburt-nv · Pull Request #17513 · NVIDIA/TensorRT-LLM

tburt-nv · 2026-08-11T19:04:35Z

Dev Engineer Review

Adds parallel Check Test List execution with Release Check.
Replaces full-wheel installation with pinned trt-test-db and binding stubs.
Centralizes TRT_TEST_DB_VERSION.
Defers MPI and NVLS imports to reduce collection failures.
Adds binding-stub coverage checks through pre-commit.
Verify stub parity, waiver handling, model resolution, and pipeline failure propagation.

QA Engineer Review

Updates AutoDeploy accuracy setup for Llama, Eagle3, Nemotron, Qwen, MiniMax, and Gemma models.
Adds pure-Python collection support through stubify_bindings.py.
Documents CI collection, waiver validation, parity checks, and stub coverage.
Test-list coverage for the changed accuracy tests requires follow-up.
Coverage verdict: needs follow-up.

Per-File QA Perspective

jenkins/L0_MergeRequest.groovy: Verify parallel execution, fail-fast behavior, waiver validation, and duplicate-waiver detection.
jenkins/L0_Test.groovy: Verify removal of GPU-based validation and use of TRT_TEST_DB_VERSION.
jenkins/ci_versions.properties: Verify consistent package-version usage.
scripts/check_test_list.py: Verify stubbed collection, parity checks, and waiver handling.
tests/integration/defs/accuracy/test_llm_api_autodeploy.py: Verify runtime model resolution for all affected variants and confirm test-list coverage.
tests/integration/defs/conftest.py: Verify lazy MPI imports and NVLS fallback behavior.
tests/integration/defs/stubify_bindings.py: Verify binding coverage, nested imports, Torch stubs, and preservation of unrelated errors.
tests/integration/test_lists/test-db/README.md: Verify documented commands match CI behavior.
tests/unittest/utils/util.py: Verify CUDA initialization errors return math.inf and no-device handling remains -1.
.pre-commit-config.yaml: Verify binding-stub validation triggers for the intended files.
scripts/check_binding_stubs.py: Verify extension discovery, coverage reporting, and exit statuses.

Description

This PR refactors the check testlist script to use a "stub" wheel, so that it doesn't depend on a full build. Since the script has no build dependency, this PR moves it in parallel with the release check stage, so that it can run more quickly and without delaying test stages.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

tburt-nv · 2026-08-11T19:04:51Z

/bot skip --comment "test skip workflow"

tensorrt-cicd · 2026-08-11T19:10:42Z

PR_Github #65375 [ skip ] triggered by Bot. Commit: c341004 Link to invocation

tensorrt-cicd · 2026-08-11T19:22:29Z

PR_Github #65375 [ skip ] completed with state SUCCESS. Commit: c341004
Release Check Pipeline #4753 failed

Link to invocation

tburt-nv · 2026-08-11T19:48:49Z

/bot skip --comment "test skip workflow"

tensorrt-cicd · 2026-08-11T20:12:21Z

PR_Github #65378 [ skip ] triggered by Bot. Commit: 57e1475 Link to invocation

tensorrt-cicd · 2026-08-11T20:27:31Z

PR_Github #65378 [ skip ] completed with state SUCCESS. Commit: 57e1475
Skipping testing for commit 57e1475

Link to invocation

tburt-nv · 2026-08-12T16:55:55Z

/bot skip --comment "test skip workflow"

tburt-nv · 2026-08-12T17:43:52Z

/bot skip --comment "test skip workflow"

tensorrt-cicd · 2026-08-12T17:49:45Z

PR_Github #65630 [ skip ] triggered by Bot. Commit: 03b9668 Link to invocation

tensorrt-cicd · 2026-08-12T18:02:52Z

PR_Github #65630 [ skip ] completed with state SUCCESS. Commit: 03b9668
Release Check Pipeline #4773 failed

Link to invocation

tburt-nv · 2026-08-12T18:37:09Z

/bot run

tensorrt-cicd · 2026-08-12T18:43:39Z

PR_Github #65645 [ run ] triggered by Bot. Commit: 6220a4a Link to invocation

tensorrt-cicd · 2026-08-12T19:02:47Z

PR_Github #65645 [ run ] completed with state FAILURE. Commit: 6220a4a
/LLM/main/L0_MergeRequest_PR pipeline #53367 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-08-12T19:35:05Z

The latest pipeline caught a real issue in the main branch, which perfectly demonstrates the need for this PR: #17579

tburt-nv · 2026-08-12T19:35:34Z

/bot run

tensorrt-cicd · 2026-08-12T19:42:09Z

PR_Github #65657 [ run ] triggered by Bot. Commit: a887462 Link to invocation

tensorrt-cicd · 2026-08-12T20:25:25Z

PR_Github #65657 [ run ] completed with state SUCCESS. Commit: a887462
/LLM/main/L0_MergeRequest_PR pipeline #53379 completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

tburt-nv · 2026-08-12T21:58:24Z

/bot run

tensorrt-cicd · 2026-09-09T20:35:56Z

PR_Github #72511 [ run ] completed with state FAILURE. Commit: 54f1594
/LLM/main/L0_MergeRequest_PR pipeline #59523 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-09T21:20:28Z

/bot run

tensorrt-cicd · 2026-09-09T21:27:56Z

PR_Github #72532 [ run ] triggered by Bot. Commit: 20eb829 Link to invocation

tensorrt-cicd · 2026-09-09T23:25:11Z

PR_Github #72532 [ run ] completed with state SUCCESS. Commit: 20eb829
/LLM/main/L0_MergeRequest_PR pipeline #59541 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-10T15:02:49Z

/bot run

tensorrt-cicd · 2026-09-10T15:09:50Z

PR_Github #72735 [ run ] triggered by Bot. Commit: c542acc Link to invocation

tensorrt-cicd · 2026-09-10T15:58:32Z

PR_Github #72735 [ run ] completed with state SUCCESS. Commit: c542acc
/LLM/main/L0_MergeRequest_PR pipeline #59723 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-10T16:49:13Z

/bot run

tensorrt-cicd · 2026-09-10T16:55:27Z

PR_Github #72755 [ run ] triggered by Bot. Commit: c542acc Link to invocation

tensorrt-cicd · 2026-09-10T19:02:09Z

PR_Github #72755 [ run ] completed with state SUCCESS. Commit: c542acc
/LLM/main/L0_MergeRequest_PR pipeline #59743 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-10T19:16:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-10T19:23:24Z

PR_Github #72777 [ run ] triggered by Bot. Commit: c542acc Link to invocation

tensorrt-cicd · 2026-09-10T21:34:41Z

PR_Github #72777 [ run ] completed with state SUCCESS. Commit: c542acc
/LLM/main/L0_MergeRequest_PR pipeline #59762 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-10T21:51:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-10T21:58:47Z

PR_Github #72798 [ run ] triggered by Bot. Commit: c542acc Link to invocation

tensorrt-cicd · 2026-09-11T12:33:32Z

PR_Github #72798 [ run ] completed with state FAILURE. Commit: c542acc
/LLM/main/L0_MergeRequest_PR pipeline #59782 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

tburt-nv · 2026-09-16T23:16:25Z

/bot run

tensorrt-cicd · 2026-09-16T23:22:39Z

PR_Github #73930 [ run ] triggered by Bot. Commit: fb1b48d Link to invocation

tensorrt-cicd · 2026-09-17T01:18:30Z

PR_Github #73930 [ run ] completed with state FAILURE. Commit: fb1b48d
/LLM/main/L0_MergeRequest_PR pipeline #60787 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-09-17T22:37:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-17T22:45:34Z

PR_Github #74225 [ run ] triggered by Bot. Commit: fb1b48d Link to invocation

tensorrt-cicd · 2026-09-18T02:09:42Z

PR_Github #74225 [ run ] completed with state FAILURE. Commit: fb1b48d
/LLM/main/L0_MergeRequest_PR pipeline #61052 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned tburt-nv Aug 11, 2026

tburt-nv force-pushed the user/tburt/shift_testlist_check branch from ae32480 to 885d8c7 Compare August 12, 2026 16:55

tburt-nv force-pushed the user/tburt/shift_testlist_check branch from 885d8c7 to 03b9668 Compare August 12, 2026 17:43

tburt-nv force-pushed the user/tburt/shift_testlist_check branch from 03b9668 to 6220a4a Compare August 12, 2026 18:36

tburt-nv force-pushed the user/tburt/shift_testlist_check branch from 6220a4a to a887462 Compare August 12, 2026 19:35

tburt-nv marked this pull request as ready for review August 12, 2026 21:58

tburt-nv requested review from a team as code owners August 12, 2026 21:58

tburt-nv requested a review from hnover-nv August 12, 2026 21:58

tburt-nv added the ci: full pre-merge approved label Sep 10, 2026

tburt-nv added 6 commits September 16, 2026 16:11

move test list check to top-level pipeline

91418f8

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

lazily evaluate model paths, and remove TRTLLM_BINDINGS_STUB

8025659

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

Use binding stubs for test-list collection

fada9c6

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

add pre-commit check for stub alignment

43c0e0e

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

avoid model mount

fe61f0f

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

remove inaccurate test

fb1b48d

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

tburt-nv force-pushed the user/tburt/shift_testlist_check branch from c542acc to fb1b48d Compare September 16, 2026 23:15

Conversation

tburt-nv commented Aug 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

tburt-nv commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tburt-nv commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tburt-nv commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Sep 9, 2026

Uh oh!

tburt-nv commented Sep 9, 2026

Uh oh!

tensorrt-cicd commented Sep 9, 2026

Uh oh!

tensorrt-cicd commented Sep 9, 2026

Uh oh!

tburt-nv commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tburt-nv commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tburt-nv commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tburt-nv commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 10, 2026

Uh oh!

tensorrt-cicd commented Sep 11, 2026

Uh oh!

tburt-nv commented Sep 16, 2026

tburt-nv commented Aug 11, 2026 •

edited by coderabbitai Bot

Loading