test(gpu): assert the parts-download counter is zero on the happy path - #938
Merged
Merged
Conversation
The device-only cliff recoveries replace hard aborts with a silent download-and-continue, so the counters are now the only thing that surfaces a gate/dispatch lockstep break. GPU_DEVICE_ONLY_DOWNGRADES (trace side) already has its == 0 guard here; its parts-side counterpart did not, and its only readers were the > 0 assertions in cuda_fallback_tests, which run with a fault deliberately armed. Without this, a decline in the R2 comp-poly tree build on a device-only table recovers, verifies and passes green, while every such table pays a full parts D2H plus a CPU commit_bit_reversed and loses the resident composition tree. The R4 DEEP site is already covered transitively (it needs the trace to be device-only too, which moves the trace counter), so this closes the R2 commit and R3 parts-OOD sites. Zero is the right expectation: materialize_composition_parts_host early-returns without bumping when the part evals are already populated, so the counter only moves for a device-only table that had to pull its parts back. The message names both causes rather than blaming the gate, matching the counter's own doc, which now allows a transient VRAM decline as well as a gate miss.
ColoCarletti
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Small follow-up to #935, targeted at its branch so it can go in with the PR.
#935 replaces the device-only hard aborts with a silent download-and-continue. That is the right call, but the abort was also the detector for the LOCKSTEP rule in
device_only_gate's doc — add a decline condition to a dispatch without mirroring it into the gate, and the crash made it impossible to miss. With the recovery in place, the counters are the only thing left that surfaces it.GPU_DEVICE_ONLY_DOWNGRADES(trace side) already has its== 0guard ingpu_device_only_residency_fires_and_verifies. Its parts-side counterpart,GPU_COMPOSITION_PARTS_DOWNLOADS, does not — its only readers are the> 0assertions incuda_fallback_tests, which run with a fault deliberately armed. So on a healthy prove nothing reads it.Concretely, what slips through today: a decline in
try_build_comp_poly_tree_gpu_from_devfor device-only tables.materialize_composition_parts_hostrepopulates the evals, bumps only the parts counter, the proof verifies, the test passes green — while every device-only table pays a full parts D2H plus a CPUcommit_bit_reversed, andgpu_composition_treegoesNoneso the R4 composition openings revert to the host tree too. A silent reversion of the win residency exists to deliver.Scope: this closes the R2 commit site and the R3 parts-OOD site. The R4 DEEP site (
prover.rs:2392) is already covered transitively — reaching it requires the trace to be device-only as well, and the onlyhost_trace_emptytrue→false transition in the tree isset_host_data, one line above theGPU_DEVICE_ONLY_DOWNGRADESbump.Zero is the right expectation:
materialize_composition_parts_hostearly-returns without bumping when the part evals are already populated, so the counter never moves for a non-device-only table, even if both comp-tree arms decline.The message names both causes instead of blaming the gate, matching the counter-family doc #935 rewrote, which now admits a transient VRAM decline as well as a gate miss. Worth considering the same wording for the two existing messages beside it — they still say "the gate should mirror the missing condition", which would misdiagnose a transient decline on a busy box.
Also refreshes the test's doc comment, which still said "at R3/R4 it panics one of the guards". After #935 that is only true on the R4 opening path.
Not verified on hardware — I have no CUDA device.
cargo fmt --checkandcargo clippy -p lambda-vm-prover --all-targets --features cuda -- -D warningsare clean; the assertion itself only runs undermake test-cuda-integrationon the GPU box.