[NVBUG-6448152][test] Improve consensus efficiency in Python transceiver (DO NOT REVIEW YET)

[chienchunhung]

Purpose

TEST ONLY; DO NOT REVIEW.

Validate the production candidate for generation-first Python request-cohort activation before updating the official review branch.

The candidate keeps resource-free context requests in a pre-active lane, lets rank zero choose a capacity-bounded peer-ready cohort in queue priority order, serializes the exact activation IDs through the existing pipeline schedule, and requires followers to materialize that exact cohort or fail before mutation. Strict READY agreement remains the gate before scheduler visibility.

Focused CI acceptance

The first CI run is limited to the six H100 PyTorch unit-test shards containing the affected suites. It must collect and pass the focused tests covering:

• capacity-independent pre-active cohort extraction;
• rank-zero canonical activation selection;
• exact follower materialization;
• follower fail-closed behavior before mutation;
• authoritative READY release and schedule activation;
• metadata leases, TTL cleanup, cancellation, session handoff, and shutdown.

Passing these unit tests proves the intended branches execute and preserve their local invariants. It does not establish distributed liveness or performance.

Follow-up validation

Only after focused CI passes, run the exact generation-first GB300 workload with matched flags-off control and combined treatment. A valid result requires every submitted request to succeed, identical activation on all four context pipeline ranks, READY and terminal agreement, clean shutdown, and an official throughput record.

No throughput claim is made by this test-only pull request.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61137 [ run ] triggered by Bot. Commit: 0453b6d Link to invocation

[tensorrt-cicd]

PR_Github #61137 [ run ] completed with state SUCCESS. Commit: 0453b6d
/LLM/main/L0_MergeRequest_PR pipeline #49389 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61190 [ run ] triggered by Bot. Commit: 9f71926 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61363 [ run ] triggered by Bot. Commit: e026195 Link to invocation

[tensorrt-cicd]

PR_Github/16766-9f71926 #61190 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

[tensorrt-cicd]

PR_Github #61363 [ run ] completed with state FAILURE. Commit: e026195
/LLM/main/L0_MergeRequest_PR pipeline #49591 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61381 [ run ] triggered by Bot. Commit: a2928c4 Link to invocation

[tensorrt-cicd]

PR_Github #61381 [ run ] completed with state SUCCESS. Commit: a2928c4
/LLM/main/L0_MergeRequest_PR pipeline #49607 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation