[CI] Run the CPU quarantine shards on the oldest and newest Python only#3977
Open
vmoens wants to merge 1 commit into
Open
[CI] Run the CPU quarantine shards on the oldest and newest Python only#3977vmoens wants to merge 1 commit into
vmoens wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3977
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit f15bab8 with merge base a74b478 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
github-actions
Bot
added
the
CI
This was referenced Jul 9, 2026
Contributor
Benchmark Results: PR
|
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] |
184.09 | 54.37 | -70.47% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] |
60.49 | 86.70 | +43.32% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,669 | 3,554 | +33.17% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] |
28.23 | 35.78 | +26.74% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
49.05 | 37.35 | -23.86% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,566 | 3,052 | +18.92% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
2,609 | 3,062 | +17.37% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,130 | 1,825 | -14.31% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] |
8.1579 | 9.2905 | +13.88% |
benchmarks/test_collectors_benchmark.py::test_single_with_rb |
7.3618 | 6.5549 | -10.96% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
829.53 | 902.14 | +8.75% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] |
10.46 | 11.33 | +8.27% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] |
76.52 | 82.49 | +7.80% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
1,894 | 2,036 | +7.51% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] |
208.65 | 224.20 | +7.45% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
2,599 | 2,782 | +7.03% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,508 | 2,683 | +6.97% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] |
8.9690 | 8.3993 | -6.35% |
benchmarks/test_envs_benchmark.py::test_simple |
1.8111 | 1.6967 | -6.32% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] |
4.0161 | 4.2573 | +6.00% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] |
24,413 | 22,967 | -5.93% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] |
249.99 | 264.69 | +5.88% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
1,883 | 1,773 | -5.83% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] |
481.46 | 454.97 | -5.50% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] |
12,385 | 11,714 | -5.42% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
1,842 | 1,941 | +5.35% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,526 | 2,655 | +5.14% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] |
0.5258 | 0.4991 | -5.08% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
435.38 | 457.43 | +5.07% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] |
136.25 | 129.70 | -4.81% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] |
112.52 | 117.89 | +4.77% |
benchmarks/test_collectors_benchmark.py::test_sync_preempt |
16.62 | 15.88 | -4.49% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
1,717 | 1,794 | +4.48% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] |
272.29 | 284.18 | +4.36% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] |
725.42 | 756.92 | +4.34% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] |
7,228 | 6,916 | -4.31% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] |
524.25 | 546.30 | +4.21% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None] |
215.62 | 224.63 | +4.18% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,825 | 2,941 | +4.12% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
2.8899 | 3.0060 | +4.02% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,947 | 3,065 | +3.99% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] |
1.9231 | 1.9986 | +3.93% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
249.10 | 258.61 | +3.82% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] |
170.06 | 163.62 | -3.79% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
400.89 | 415.39 | +3.62% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None] |
273.69 | 283.13 | +3.45% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
1,018 | 1,051 | +3.25% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] |
38,324 | 37,113 | -3.16% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] |
49,614 | 51,175 | +3.15% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] |
680.13 | 701.16 | +3.09% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] |
7,200 | 6,978 | -3.08% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] |
623.19 | 642.14 | +3.04% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] |
683.55 | 704.32 | +3.04% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] |
58.08 | 59.81 | +2.98% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] |
43,544 | 42,256 | -2.96% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] |
49,828 | 51,248 | +2.85% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] |
273.20 | 280.95 | +2.84% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] |
321.18 | 330.29 | +2.84% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] |
752.08 | 730.91 | -2.81% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-backward] |
62.75 | 61.01 | -2.78% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] |
547.08 | 531.99 | -2.76% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] |
56,656 | 55,106 | -2.74% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] |
4,417 | 4,300 | -2.67% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] |
29,101 | 29,867 | +2.63% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] |
267.52 | 260.72 | -2.54% |
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] |
25.12 | 24.50 | -2.47% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] |
302.05 | 294.62 | -2.46% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] |
113.56 | 116.31 | +2.42% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] |
134.28 | 137.47 | +2.37% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
966.45 | 989.17 | +2.35% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] |
51.97 | 50.77 | -2.32% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] |
91.57 | 89.45 | -2.31% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] |
2,529 | 2,471 | -2.30% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] |
5.8176 | 5.9502 | +2.28% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16] |
5.0112 | 5.1228 | +2.23% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-backward] |
53.04 | 54.21 | +2.21% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] |
29,942 | 30,600 | +2.20% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] |
682.76 | 697.57 | +2.17% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] |
205.40 | 209.77 | +2.12% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] |
3.0588 | 2.9938 | -2.12% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-None] |
672.53 | 686.78 | +2.12% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] |
546.51 | 558.08 | +2.12% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] |
20,892 | 20,453 | -2.10% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] |
393.66 | 385.69 | -2.03% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] |
18.21 | 18.58 | +2.01% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] |
31,615 | 32,247 | +2.00% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] |
4.5483 | 4.6381 | +1.97% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] |
254.59 | 259.59 | +1.96% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_catframes_offline |
517.72 | 527.77 | +1.94% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-None] |
157.05 | 160.03 | +1.90% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] |
81.64 | 83.18 | +1.89% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] |
22.76 | 23.19 | +1.86% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] |
4,350 | 4,270 | -1.84% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cpu... |
80.36 | 78.89 | -1.83% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] |
32,576 | 31,992 | -1.79% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] |
51.68 | 52.60 | +1.79% |
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] |
7,758 | 7,625 | -1.72% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] |
24,323 | 23,905 | -1.72% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] |
111.75 | 113.66 | +1.71% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] |
30,270 | 30,774 | +1.67% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] |
163.38 | 160.74 | -1.62% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] |
31,905 | 32,418 | +1.61% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] |
18,776 | 19,077 | +1.61% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] |
28,039 | 28,488 | +1.60% |
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] |
54.73 | 55.59 | +1.59% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] |
0.6021 | 0.5928 | -1.54% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] |
20,684 | 20,366 | -1.53% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] |
114.68 | 116.44 | +1.53% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] |
12.21 | 12.39 | +1.53% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] |
2,601 | 2,562 | -1.51% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] |
194.03 | 196.94 | +1.50% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-True] |
0.2143 | 0.2112 | -1.48% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] |
80.33 | 81.51 | +1.47% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[done_aware] |
1,528 | 1,550 | +1.45% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] |
77,906 | 76,779 | -1.45% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] |
20.12 | 20.41 | +1.43% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
56.64 | 57.45 | +1.42% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] |
494.11 | 487.22 | -1.39% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16] |
47.52 | 48.18 | +1.38% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] |
41.39 | 41.95 | +1.37% |
| ... | ... | ... | Showing 120 of 220 comparisons, sorted by absolute change. |
GPU
Compared 230 benchmarks. Regressions over 5%: 12. Improvements over 5%: 17.
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] |
82.85 | 112.45 | +35.72% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
3,464 | 2,724 | -21.36% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,906 | 3,404 | +17.13% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
3,221 | 3,754 | +16.56% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
1,993 | 2,301 | +15.43% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] |
526.58 | 457.05 | -13.20% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
44.27 | 49.63 | +12.12% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
3,119 | 3,496 | +12.09% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] |
6.4000 | 7.1498 | +11.72% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,018 | 2,669 | -11.54% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,227 | 2,879 | -10.78% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] |
364.99 | 400.98 | +9.86% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
1,856 | 2,028 | +9.27% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] |
735.30 | 803.06 | +9.22% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
3,003 | 2,746 | -8.57% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] |
3,833 | 4,133 | +7.81% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
3,446 | 3,177 | -7.79% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] |
349.86 | 376.00 | +7.47% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,098 | 1,945 | -7.30% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] |
24,220 | 22,587 | -6.74% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] |
12,374 | 11,602 | -6.24% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
814.82 | 863.11 | +5.93% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] |
500.24 | 529.68 | +5.89% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] |
44,179 | 41,739 | -5.52% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] |
48.44 | 45.77 | -5.50% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] |
166.26 | 175.18 | +5.37% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_catframes_offline |
529.74 | 557.38 | +5.22% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] |
418.45 | 440.06 | +5.17% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] |
785.50 | 745.84 | -5.05% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] |
324.72 | 340.89 | +4.98% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] |
48.95 | 51.31 | +4.81% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] |
133.70 | 140.08 | +4.77% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] |
698.20 | 730.08 | +4.57% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] |
269.04 | 280.97 | +4.43% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] |
731.79 | 701.22 | -4.18% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] |
7,255 | 6,958 | -4.09% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] |
17.57 | 16.87 | -4.01% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] |
138.38 | 143.87 | +3.96% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] |
41.24 | 42.87 | +3.94% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,144 | 2,060 | -3.92% |
benchmarks/test_envs_benchmark.py::test_simple |
1.2466 | 1.1978 | -3.91% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
21.71 | 22.55 | +3.85% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... |
1,511 | 1,454 | -3.77% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] |
238.42 | 247.37 | +3.75% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] |
76.85 | 79.69 | +3.69% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] |
33,748 | 34,952 | +3.57% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] |
274.32 | 283.74 | +3.43% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] |
0.5273 | 0.5093 | -3.41% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] |
10.36 | 10.70 | +3.26% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] |
27,593 | 26,721 | -3.16% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] |
51.64 | 53.21 | +3.05% |
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] |
11,347 | 11,679 | +2.93% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
3,601 | 3,497 | -2.89% |
benchmarks/test_envs_benchmark.py::test_parallel |
0.5371 | 0.5525 | +2.88% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] |
626.82 | 608.80 | -2.87% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] |
23,744 | 23,063 | -2.87% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] |
1,840 | 1,892 | +2.87% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] |
11.98 | 12.31 | +2.77% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[200-img_shape1-large_batch] |
8.7986 | 8.5552 | -2.77% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] |
38,184 | 37,134 | -2.75% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
230.01 | 236.31 | +2.74% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] |
20.80 | 21.34 | +2.61% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16] |
47.11 | 48.33 | +2.58% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[200-img_shape1-large_batch] |
6.6837 | 6.5122 | -2.57% |
benchmarks/test_replaybuffer_benchmark.py::test_replay_buffer_direct_client_identity |
7,694,497 | 7,886,338 | +2.49% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] |
271.15 | 277.84 | +2.47% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] |
21.67 | 22.21 | +2.46% |
benchmarks/test_envs_benchmark.py::test_transformed |
0.7125 | 0.6951 | -2.45% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] |
57,901 | 59,306 | +2.43% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] |
43,076 | 42,033 | -2.42% |
benchmarks/test_collectors_benchmark.py::test_sync |
10.45 | 10.20 | -2.41% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] |
192.08 | 196.66 | +2.38% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[done_aware] |
1,550 | 1,586 | +2.37% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] |
4.4702 | 4.5757 | +2.36% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] |
4,436 | 4,331 | -2.36% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] |
8.5685 | 8.7687 | +2.34% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[no_done] |
1,584 | 1,621 | +2.33% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[100-img_shape0-atari] |
13.42 | 13.11 | -2.33% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] |
155.46 | 159.06 | +2.32% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] |
20,099 | 19,642 | -2.27% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] |
1,350 | 1,320 | -2.21% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
478.92 | 489.34 | +2.17% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] |
23.13 | 23.63 | +2.13% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] |
19,892 | 19,473 | -2.11% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cuda] |
2,240 | 2,194 | -2.05% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] |
14.89 | 15.19 | +2.04% |
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] |
19.73 | 20.11 | +1.94% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-True] |
0.2159 | 0.2117 | -1.94% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] |
399.08 | 391.41 | -1.92% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] |
29,517 | 28,952 | -1.91% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] |
4,790 | 4,698 | -1.91% |
benchmarks/test_collectors_benchmark.py::test_async |
10.84 | 11.04 | +1.88% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] |
52.80 | 51.81 | -1.87% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] |
35.57 | 36.23 | +1.85% |
benchmarks/test_envs_benchmark.py::test_serial |
0.4205 | 0.4283 | +1.84% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] |
737.44 | 751.03 | +1.84% |
benchmarks/test_collectors_benchmark.py::test_single_with_rb_pixels |
5.1976 | 5.2932 | +1.84% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] |
18,828 | 18,485 | -1.82% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
908.31 | 892.14 | -1.78% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] |
34,561 | 33,951 | -1.77% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] |
151.51 | 154.16 | +1.75% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] |
37,915 | 38,572 | +1.73% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] |
18,044 | 17,745 | -1.66% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
700.20 | 711.80 | +1.66% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
101.33 | 99.67 | -1.63% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] |
823.84 | 836.91 | +1.59% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
1,001 | 1,016 | +1.56% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-buffers-True] |
0.5243 | 0.5161 | -1.55% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] |
5,968 | 5,876 | -1.54% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-no-buffers-True] |
0.6058 | 0.5965 | -1.53% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] |
31,607 | 31,129 | -1.51% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-backward] |
68.28 | 69.29 | +1.49% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] |
1,911 | 1,939 | +1.48% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] |
464.51 | 471.07 | +1.41% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] |
527.68 | 535.07 | +1.40% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] |
366,724 | 371,670 | +1.35% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] |
326.11 | 321.72 | -1.35% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] |
283.35 | 287.13 | +1.33% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cuda] |
1,067 | 1,053 | -1.32% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] |
31,388 | 31,800 | +1.31% |
| ... | ... | ... | Showing 120 of 230 comparisons, sorted by absolute change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
All five tests-cpu Python versions previously paid the full serial
quarantine on every PR push. Keep the xdist bulk (24k tests in ~5 min)
on all five versions as the middle-version smoke subset, and run the
expensive collectors/mp quarantine shards on 3.10 and 3.14 only, which
bracket the supported range.
A separate tests-cpu-quarantine-mid job covers 3.11-3.13 and is skipped
on pull requests: pushes to main and the nightly orchestrator retain
full five-version quarantine coverage, and workflow_dispatch provides
an on-demand full run for a PR branch. No PR-label gate: a "labeled"
trigger type would retrigger the whole workflow and the ref-keyed
cancel-in-progress group would cancel in-flight runs.
upload-codecov now also waits on the mid job so its coverage artifacts
are included on non-PR events (skipped needs do not block if: always()).
Saves roughly two linux.12xlarge runner-hours per PR push.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com