[None][test] Prune Llama-3.1 8b model from tests by xinhe-nv · Pull Request #18654 · NVIDIA/TensorRT-LLM

xinhe-nv · 2026-09-03T09:40:49Z

Retires Llama-3.1-8B / Llama-3.1-8B-Instruct from the test suite.

Dev Engineer Review

The change removes Llama 3.1 8B mappings, selectors, fixtures, waivers, and test cases. It redirects selected coverage to Qwen3, Qwen3.5, GPT-OSS, and Nemotron.

The change also removes multi-node OpenAI, Eagle3, benchmark, quickstart, NGRAM, mixed-precision, MIG, and Llama FP8 LoRA coverage. Verify that these removals are within scope and that equivalent coverage remains. Verify the new Qwen3 model paths, speculative-decoding layers, baselines, and Open MPI 5 skip.

A Llama Eagle3 reference remains in tests/unittest/llmapi/test_session_reuse.py. Confirm that this reference is intentional and outside the retirement scope.

QA Engineer Review

The change modifies integration tests, unit tests, CI lists, QA lists, and waivers. It adds or redirects Qwen3, Qwen3.5, GPT-OSS, and Nemotron coverage.

The updated Qwen3 selectors appear in QA lists and waivers. The change removes Llama, Eagle3, benchmark, quickstart, and multi-node OpenAI coverage. CI results include failures and instability, and the latest run requires ci: full pre-merge approved before another multi-GPU run. Coverage is needs follow-up.

Per-File QA Perspective

tests/integration/defs/conftest.py: Removes Llama model-path fixtures. Verify remaining model mappings.
tests/integration/defs/disaggregated/test_ad_disagg.py: Replaces Llama Eagle3 with Qwen3 Eagle3. Verify paths, capture layers, baselines, and selectors.
tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_qwen3_8b.yaml: Selects Qwen3-8B. Verify model resolution and parallelism.
tests/integration/defs/disaggregated/test_disaggregated.py: Replaces the Llama logprob case with Qwen3. Verify logprob behavior and list selectors.
tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py: Adds an Open MPI 5 skip and uses Qwen3. Verify skip conditions and coverage.
tests/integration/defs/examples/test_ad_speculative_decoding.py: Removes Llama Eagle3 acceptance-rate and wrapper tests. Verify no stale selectors remain.
tests/integration/defs/llmapi/test_llm_examples.py: Removes the Llama Medusa mapping. Verify remaining mappings.
tests/integration/defs/test_e2e.py: Removes Llama, Eagle3, benchmark, and quickstart tests. Verify remaining GPT-OSS selectors and skips.
tests/integration/test_lists/qa/llm_function_core.txt: Updates disaggregated and speculative-decoding IDs and removes a Llama entry. Verify selectors.
tests/integration/test_lists/qa/llm_spark_func.yml: Removes Llama entries and retains Qwen3 and GPT-OSS coverage.
tests/integration/test_lists/test-db/l0_a10.yml: Adds Nemotron and BART tests and removes a benchmark test. Verify A10 selection.
tests/integration/test_lists/test-db/l0_b200.yml: Removes legacy Llama, Eagle3, NGRAM, mixed-precision, and benchmark entries.
tests/integration/test_lists/test-db/l0_dgx_h200.yml: Removes the H200 benchmark entry.
tests/integration/test_lists/test-db/l0_h100.yml: Changes benchmark, guided-decoding, disaggregated, and Eagle3 assignments and adds Nemotron coverage.
tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml: Removes Llama BF16, FP8, and NVFP4 entries.
tests/integration/test_lists/waives.txt: Removes obsolete waivers and changes hardware-specific waivers. Verify waiver applicability.
tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes.py: Uses Qwen3.5-4B. Verify multi-node completion behavior.
tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes_service_discovery.py: Uses Qwen3.5-4B. Verify discovery and completion behavior.
tests/unittest/llmapi/apps/_test_openai_chat_guided_decoding.py: Removes Llama coverage. Verify remaining model selection.
tests/unittest/llmapi/apps/_test_openai_multi_nodes.py: Deletes multi-node chat, completion, streaming, and consistency tests. Verify equivalent coverage.
tests/unittest/llmapi/apps/_test_trtllm_serve_benchmark.py: Restricts coverage to GPT-OSS-20B. Verify the pre-Hopper skip.
tests/unittest/llmapi/test_llm.py: Removes Llama incremental detokenization coverage. Verify another model covers the path.
tests/unittest/llmapi/test_llm_pytorch.py: Removes the Llama FP8 LoRA test. Verify equivalent FP8 LoRA coverage.
tests/unittest/llmapi/test_session_reuse.py: Replaces Llama torch-compile identifiers with Qwen3 cases. Verify both privacy expectations and the remaining Llama Eagle3 selector.
tests/unittest/metrics/test_collector.py: Uses Qwen3-8B metadata. Verify metric labels and served-name behavior.
tests/unittest/scripts/test_check_model_registry.py: Uses Qwen3-8B for duplicate-name validation. Verify the assertion remains unchanged.

Description

https://docs.google.com/spreadsheets/d/17qLRr5oQESqn1rT9LrIUnpjvhH8wcC59273YEPT9k44/edit?gid=177162358#gid=177162358

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…n and unit tests Fully removes Llama-3.1-8B / Llama-3.1-8B-Instruct from the test suite, including dedicated accuracy test classes, shared multi-model parametrizations in test_e2e.py/disaggregated/examples/llmapi tests, corresponding test-list entries (waives.txt, qa/*, test-db/*), accuracy reference YAMLs, the RocketKV sparse-attention unit test, spec-decoding and AutoDeploy smoke tests, and the Llama architecture unit test (test_modeling_llama.py, with its shared Scenario/reduce_llama_config helpers preserved for test_modeling_mllama.py). Files left with no remaining tests were deleted outright. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Cleanup fallout from the Llama-3.1-8B test removal conflict resolution: - Drop now-unused imports (llm_models_root, os, ExperimentConfig/main). - Add the sys/os imports actually needed by the sys.path.append call kept in test_sa.py. - Remove stale TestMiniMaxM2/TestMiniMaxM2_5 list entries: those classes were pruned upstream and were mistakenly reintroduced while merging a conflict block that legitimately needed to keep an unrelated, adjacent TestLlama4SpeculativeDecoding entry. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Sort the MSA sparse-attention waive entries alphabetically to match the repo's required ordering. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…e test test_super_mtp_smoke still needs hf_id_to_local_model_dir, ExperimentConfig, and main; these were dropped by mistake while resolving the Llama-3.1-8B removal merge conflict in this file. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…loy Llama-3.1-8B changes Restore all files under these two directories to upstream/main, undoing the speculative-decoding and AutoDeploy smoke-test deletions from the Llama-3.1-8B cleanup. Narrows this branch's scope back to tests/integration/defs and the remaining tests/unittest files already touched elsewhere. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…on Qwen3-8B Re-add test_async_eagle3_full_model_handoff (test_ad_disagg.py) and test_disaggregated_logprobs_serving (test_disaggregated.py), both previously deleted as part of the Llama-3.1-8B retirement, using Qwen3-8B (+ Qwen3/qwen3_8b_eagle3 draft for the Eagle3 case) instead. test_async_eagle3_full_model_handoff now compares against a freshly computed aggregate run using the same speculative_config instead of hardcoded golden text/token IDs, since no such goldens exist yet for this model pairing. eagle3_layers_to_capture was scaled proportionally from the retired Llama config and is flagged as needing GPU validation. test_disaggregated_logprobs_serving drops the llama_model_root indirect fixture in favor of the simpler direct setup_model_symlink pattern already used elsewhere in this file, and adds a new disagg_config_ctxtp2_gentp2_qwen3_8b.yaml config. Both tests still need a real GPU run to confirm behavior before they can be trusted in CI. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…of file Keep it as a pure addition instead of an in-place replacement of the retired Llama-3.1-8B version, so it doesn't shift the line position of test_disaggregated_cancel_large_context_requests and everything after it. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…gregated_logprobs_serving Rebuild the test from upstream's current version and swap only the model-specific pieces, so the diff stays minimal and no other test in the file shifts position. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…gregated_spec_dec_batch_slot_limit Restore this Eagle3 disaggregated batch-slot-release test on Qwen3-8B instead of the retired Llama-3.1-8B + EAGLE3-LLaMA3.1-Instruct-8B pairing, and add back its l0_h100 and QA test-list entries under the new parametrize ID. Also restores the renamed test_disaggregated_logprobs_serving entry in llm_function_core.txt. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…de disagg completion tests Restore test_openai_disagg_multi_nodes_completion, test_openai_disagg_multi_nodes_completion_service_discovery, and their underlying _test_disagg_serving_multi_nodes*.py helpers on Qwen3.5-4B instead of the retired Llama-3.1-8B, adding skip_pre_hopper since Qwen3.5-4B requires it. Also restores the three QA test-list entries in llm_function_multinode.txt. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Remove waivers for TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm(_eagle)] on GB300 (nvbugs/6572838), test_overlap_scheduler_consistency on GB300 (nvbugs/6608387), and the MiniMax M3 MSA sparse-attention decode tests (nvbugs/6669902). Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

…ids_are_private Replace the TestGemma3_1BInstruct placeholder (introduced when retiring Llama-3.1-8B coverage) with a real, currently-existing torch_compile parametrized test ID from TestQwen3_30B_A3B::test_fp8, instead of substituting an unrelated model class. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com> # Conflicts: # tests/integration/defs/disaggregated/test_ad_disagg.py # tests/integration/defs/examples/test_ad_speculative_decoding.py # tests/integration/test_lists/qa/llm_function_core.txt # tests/integration/test_lists/test-db/l0_h100.yml

tensorrt-cicd · 2026-09-17T04:12:59Z

PR_Github #73946 [ run ] completed with state ABORTED. Commit: 300d54d

Link to invocation

tensorrt-cicd · 2026-09-17T08:13:45Z

PR_Github #73992 [ run ] completed with state SUCCESS. Commit: 4f99b4b
/LLM/main/L0_MergeRequest_PR pipeline #60842 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-17T08:16:22Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-17T08:23:46Z

PR_Github #74071 [ run ] triggered by Bot. Commit: 4f99b4b Link to invocation

tensorrt-cicd · 2026-09-17T13:41:25Z

PR_Github #74071 [ run ] completed with state SUCCESS. Commit: 4f99b4b
/LLM/main/L0_MergeRequest_PR pipeline #60913 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-18T01:13:57Z

/bot run --disable-fail-fast

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-09-18T02:11:46Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-18T02:17:39Z

PR_Github #74261 [ run ] triggered by Bot. Commit: fc88f0d Link to invocation

tensorrt-cicd · 2026-09-18T06:20:03Z

PR_Github #74261 [ run ] completed with state FAILURE. Commit: fc88f0d
/LLM/main/L0_MergeRequest_PR pipeline #61087 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-18T06:25:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-18T06:32:09Z

PR_Github #74345 [ run ] triggered by Bot. Commit: 07073c6 Link to invocation

tensorrt-cicd · 2026-09-19T02:24:34Z

PR_Github #74345 [ run ] completed with state SUCCESS. Commit: 07073c6
/LLM/main/L0_MergeRequest_PR pipeline #61164 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-09-19T09:21:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-19T09:27:24Z

PR_Github #74538 [ run ] triggered by Bot. Commit: 9f45d7f Link to invocation

tensorrt-cicd · 2026-09-19T13:51:06Z

PR_Github #74538 [ run ] completed with state SUCCESS. Commit: 9f45d7f
/LLM/main/L0_MergeRequest_PR pipeline #61330 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-19T13:55:46Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-19T14:01:52Z

PR_Github #74560 [ run ] triggered by Bot. Commit: 9f45d7f Link to invocation

tensorrt-cicd · 2026-09-19T16:18:46Z

PR_Github #74560 [ run ] completed with state SUCCESS. Commit: 9f45d7f
/LLM/main/L0_MergeRequest_PR pipeline #61350 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-20T01:06:19Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-20T01:12:33Z

PR_Github #74597 [ run ] triggered by Bot. Commit: 9f45d7f Link to invocation

tensorrt-cicd · 2026-09-20T06:23:14Z

PR_Github #74597 [ run ] completed with state SUCCESS. Commit: 9f45d7f
/LLM/main/L0_MergeRequest_PR pipeline #61386 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-09-20T06:25:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-20T06:31:54Z

PR_Github #74609 [ run ] triggered by Bot. Commit: 9f45d7f Link to invocation

github-actions Bot assigned xinhe-nv Sep 3, 2026

xinhe-nv force-pushed the worktree-TRTLLM-15124 branch 2 times, most recently from 9b4f3b1 to 3ae19d5 Compare September 11, 2026 11:46

xinhe-nv added 6 commits September 13, 2026 12:20

[None][fix] Fix waives.txt ordering

1739246

Sort the MSA sparse-attention waive entries alphabetically to match the repo's required ordering. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv force-pushed the worktree-TRTLLM-15124 branch from 940366a to bceeaa5 Compare September 13, 2026 04:30

xinhe-nv added 6 commits September 13, 2026 12:59

xinhe-nv changed the title ~~[None][test] Remove deprecated test_cli_flow.py and unused Llama-3.1…~~ [None][test] Prune Llama-3.1 8b tests from tests Sep 13, 2026

xinhe-nv changed the title ~~[None][test] Prune Llama-3.1 8b tests from tests~~ [None][test] Prune Llama-3.1 8b model from tests Sep 13, 2026

xinhe-nv marked this pull request as ready for review September 14, 2026 01:27

xinhe-nv requested review from a team as code owners September 14, 2026 01:27

xinhe-nv requested review from asfiyab-nvidia, chienchunhung and zhaoyuanh-nvidia September 14, 2026 01:27

xinhe-nv added 3 commits September 18, 2026 09:24

Merge branch 'main' into worktree-TRTLLM-15124

c186817

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

Merge branch 'main' into worktree-TRTLLM-15124

c013735

Merge branch 'main' into worktree-TRTLLM-15124

fc88f0d

xinhe-nv added 2 commits September 18, 2026 14:22

Merge branch 'main' into worktree-TRTLLM-15124

2dc9208

Merge branch 'main' into worktree-TRTLLM-15124

07073c6

xinhe-nv added 2 commits September 19, 2026 17:19

Merge branch 'main' into worktree-TRTLLM-15124

1a75993

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

Update waives.txt

9f45d7f

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

Conversation

xinhe-nv commented Sep 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

tensorrt-cicd commented Sep 17, 2026

Uh oh!

tensorrt-cicd commented Sep 17, 2026

Uh oh!

xinhe-nv commented Sep 17, 2026

Uh oh!

tensorrt-cicd commented Sep 17, 2026

Uh oh!

tensorrt-cicd commented Sep 17, 2026

Uh oh!

xinhe-nv commented Sep 18, 2026

Uh oh!

xinhe-nv commented Sep 18, 2026

Uh oh!

tensorrt-cicd commented Sep 18, 2026

Uh oh!

tensorrt-cicd commented Sep 18, 2026

Uh oh!

xinhe-nv commented Sep 18, 2026

Uh oh!

tensorrt-cicd commented Sep 18, 2026

Uh oh!

tensorrt-cicd commented Sep 19, 2026

Uh oh!

xinhe-nv commented Sep 19, 2026

Uh oh!

tensorrt-cicd commented Sep 19, 2026

Uh oh!

tensorrt-cicd commented Sep 19, 2026

Uh oh!

xinhe-nv commented Sep 19, 2026

Uh oh!

tensorrt-cicd commented Sep 19, 2026

Uh oh!

tensorrt-cicd commented Sep 19, 2026

Uh oh!

xinhe-nv commented Sep 20, 2026

Uh oh!

tensorrt-cicd commented Sep 20, 2026

Uh oh!

tensorrt-cicd commented Sep 20, 2026

Uh oh!

xinhe-nv commented Sep 20, 2026

Uh oh!

tensorrt-cicd commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

xinhe-nv commented Sep 3, 2026 •

edited by coderabbitai Bot

Loading