Skip to content

test(gpu): assert the parts-download counter is zero on the happy path - #938

Merged
ColoCarletti merged 1 commit into
gpu-cliff-recoveryfrom
review/parts-downloads-assert
Aug 18, 2026
Merged

test(gpu): assert the parts-download counter is zero on the happy path#938
ColoCarletti merged 1 commit into
gpu-cliff-recoveryfrom
review/parts-downloads-assert

Conversation

@MauroToscano

Copy link
Copy Markdown
Contributor

Small follow-up to #935, targeted at its branch so it can go in with the PR.

#935 replaces the device-only hard aborts with a silent download-and-continue. That is the right call, but the abort was also the detector for the LOCKSTEP rule in device_only_gate's doc — add a decline condition to a dispatch without mirroring it into the gate, and the crash made it impossible to miss. With the recovery in place, the counters are the only thing left that surfaces it.

GPU_DEVICE_ONLY_DOWNGRADES (trace side) already has its == 0 guard in gpu_device_only_residency_fires_and_verifies. Its parts-side counterpart, GPU_COMPOSITION_PARTS_DOWNLOADS, does not — its only readers are the > 0 assertions in cuda_fallback_tests, which run with a fault deliberately armed. So on a healthy prove nothing reads it.

Concretely, what slips through today: a decline in try_build_comp_poly_tree_gpu_from_dev for device-only tables. materialize_composition_parts_host repopulates the evals, bumps only the parts counter, the proof verifies, the test passes green — while every device-only table pays a full parts D2H plus a CPU commit_bit_reversed, and gpu_composition_tree goes None so the R4 composition openings revert to the host tree too. A silent reversion of the win residency exists to deliver.

Scope: this closes the R2 commit site and the R3 parts-OOD site. The R4 DEEP site (prover.rs:2392) is already covered transitively — reaching it requires the trace to be device-only as well, and the only host_trace_empty true→false transition in the tree is set_host_data, one line above the GPU_DEVICE_ONLY_DOWNGRADES bump.

Zero is the right expectation: materialize_composition_parts_host early-returns without bumping when the part evals are already populated, so the counter never moves for a non-device-only table, even if both comp-tree arms decline.

The message names both causes instead of blaming the gate, matching the counter-family doc #935 rewrote, which now admits a transient VRAM decline as well as a gate miss. Worth considering the same wording for the two existing messages beside it — they still say "the gate should mirror the missing condition", which would misdiagnose a transient decline on a busy box.

Also refreshes the test's doc comment, which still said "at R3/R4 it panics one of the guards". After #935 that is only true on the R4 opening path.

Not verified on hardware — I have no CUDA device. cargo fmt --check and cargo clippy -p lambda-vm-prover --all-targets --features cuda -- -D warnings are clean; the assertion itself only runs under make test-cuda-integration on the GPU box.

The device-only cliff recoveries replace hard aborts with a silent
download-and-continue, so the counters are now the only thing that
surfaces a gate/dispatch lockstep break. GPU_DEVICE_ONLY_DOWNGRADES
(trace side) already has its == 0 guard here; its parts-side counterpart
did not, and its only readers were the > 0 assertions in
cuda_fallback_tests, which run with a fault deliberately armed.

Without this, a decline in the R2 comp-poly tree build on a device-only
table recovers, verifies and passes green, while every such table pays a
full parts D2H plus a CPU commit_bit_reversed and loses the resident
composition tree. The R4 DEEP site is already covered transitively (it
needs the trace to be device-only too, which moves the trace counter),
so this closes the R2 commit and R3 parts-OOD sites.

Zero is the right expectation: materialize_composition_parts_host
early-returns without bumping when the part evals are already populated,
so the counter only moves for a device-only table that had to pull its
parts back.

The message names both causes rather than blaming the gate, matching the
counter's own doc, which now allows a transient VRAM decline as well as
a gate miss.
@ColoCarletti
ColoCarletti merged commit 7863a67 into gpu-cliff-recovery Aug 18, 2026
8 checks passed
@ColoCarletti
ColoCarletti deleted the review/parts-downloads-assert branch August 18, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants