[CI] Fan out the tests-cpu serial quarantine into parallel jobs#3976
Open
vmoens wants to merge 1 commit into
Open
[CI] Fan out the tests-cpu serial quarantine into parallel jobs#3976vmoens wants to merge 1 commit into
vmoens wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3976
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit dbac2d1 with merge base a74b478 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
github-actions
Bot
added
the
CI
This was referenced Jul 9, 2026
Contributor
Benchmark Results: PR
|
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] |
185.09 | 56.81 | -69.30% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] |
56.91 | 87.40 | +53.58% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] |
752.06 | 1,039 | +38.20% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
2,804 | 3,524 | +25.66% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
49.38 | 36.85 | -25.37% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] |
205.94 | 249.63 | +21.21% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] |
5.6134 | 6.4829 | +15.49% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,559 | 3,100 | -12.91% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
371.53 | 418.73 | +12.71% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
1,023 | 896.03 | -12.39% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
3,322 | 2,952 | -11.15% |
benchmarks/test_collectors_benchmark.py::test_single_with_rb |
7.2703 | 6.5023 | -10.56% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] |
11.30 | 12.37 | +9.44% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] |
504.74 | 549.82 | +8.93% |
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] |
98.94 | 90.29 | -8.75% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] |
9.0821 | 9.8439 | +8.39% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,796 | 3,024 | +8.16% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] |
4,891 | 5,272 | +7.78% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] |
262.10 | 282.08 | +7.62% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
840.64 | 899.04 | +6.95% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
1,762 | 1,882 | +6.78% |
benchmarks/test_envs_benchmark.py::test_simple |
1.7948 | 1.6817 | -6.30% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] |
109.85 | 116.73 | +6.26% |
benchmarks/test_collectors_benchmark.py::test_sync |
15.23 | 16.17 | +6.16% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_catframes_offline |
530.49 | 561.93 | +5.93% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] |
79.63 | 84.22 | +5.76% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
3,071 | 3,229 | +5.15% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,127 | 2,236 | +5.14% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
943.48 | 991.21 | +5.06% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
3,162 | 3,002 | -5.04% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] |
23,410 | 22,255 | -4.94% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] |
29,160 | 30,484 | +4.54% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] |
2,304 | 2,408 | +4.50% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] |
259.62 | 271.17 | +4.45% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] |
209.68 | 218.59 | +4.25% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-backward] |
53.17 | 55.40 | +4.20% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] |
560.06 | 536.61 | -4.19% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] |
642.98 | 669.87 | +4.18% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] |
3.2485 | 3.1152 | -4.10% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] |
286.56 | 274.87 | -4.08% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] |
47,314 | 49,236 | +4.06% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] |
17,847 | 18,544 | +3.91% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] |
48.33 | 50.21 | +3.89% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,812 | 2,918 | +3.77% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] |
445.03 | 461.57 | +3.72% |
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] |
7,876 | 7,593 | -3.59% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] |
205.88 | 213.11 | +3.51% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] |
27,809 | 28,776 | +3.48% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None] |
211.01 | 218.29 | +3.45% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] |
307.12 | 317.60 | +3.41% |
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] |
37.26 | 36.02 | -3.32% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
2.8596 | 2.9529 | +3.26% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] |
278.41 | 287.26 | +3.18% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] |
19,373 | 19,988 | +3.18% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[no_done] |
1,616 | 1,565 | -3.16% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] |
121.76 | 125.38 | +2.97% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] |
326.09 | 335.58 | +2.91% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] |
53,997 | 55,551 | +2.88% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] |
699.48 | 719.33 | +2.84% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] |
1,849 | 1,798 | -2.80% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,032 | 2,088 | +2.74% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] |
774.65 | 753.58 | -2.72% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-None] |
91.78 | 94.24 | +2.68% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] |
20,876 | 20,320 | -2.66% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] |
1,792 | 1,744 | -2.65% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] |
24.89 | 25.54 | +2.59% |
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] |
25.05 | 24.41 | -2.57% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] |
30,988 | 31,770 | +2.52% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] |
2,176 | 2,229 | +2.47% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] |
257.78 | 264.07 | +2.44% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] |
6,911 | 7,077 | +2.41% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] |
341,361 | 333,546 | -2.29% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] |
81.98 | 83.83 | +2.26% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
3,434 | 3,358 | -2.20% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] |
138.47 | 135.43 | -2.20% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
256.15 | 261.68 | +2.16% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-backward] |
59.84 | 61.12 | +2.14% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] |
330.75 | 337.81 | +2.13% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] |
273.42 | 267.68 | -2.10% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] |
24.38 | 24.89 | +2.09% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] |
195.99 | 191.91 | -2.08% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] |
86.66 | 88.44 | +2.06% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] |
21,670 | 21,226 | -2.05% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] |
52.87 | 51.79 | -2.05% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] |
0.4889 | 0.4792 | -1.99% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] |
770.12 | 754.89 | -1.98% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-backward] |
27.85 | 27.31 | -1.92% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] |
23,160 | 22,734 | -1.84% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] |
647.91 | 636.04 | -1.83% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] |
109.93 | 107.92 | -1.83% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] |
28,904 | 29,428 | +1.81% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] |
525.57 | 516.05 | -1.81% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] |
52,461 | 53,389 | +1.77% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,070 | 2,034 | -1.73% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-backward] |
32.34 | 32.87 | +1.67% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] |
177.98 | 175.13 | -1.60% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] |
8.2947 | 8.1617 | -1.60% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,069 | 2,102 | +1.60% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] |
114.58 | 116.41 | +1.59% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] |
59.11 | 60.04 | +1.58% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] |
462.15 | 469.45 | +1.58% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[50-img_shape0-small] |
3,500 | 3,445 | -1.57% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] |
42.94 | 42.27 | -1.57% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari] |
29.27 | 29.71 | +1.52% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] |
0.5896 | 0.5984 | +1.49% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] |
130.36 | 132.30 | +1.49% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] |
37,104 | 36,551 | -1.49% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
454.81 | 461.24 | +1.42% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] |
6,845 | 6,940 | +1.39% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] |
120.73 | 122.40 | +1.38% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] |
53.15 | 52.42 | -1.36% |
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] |
55.17 | 55.89 | +1.31% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
58.76 | 57.99 | -1.30% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] |
33,514 | 33,080 | -1.29% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] |
34,194 | 34,636 | +1.29% |
benchmarks/test_envs_benchmark.py::test_transformed |
0.9023 | 0.8906 | -1.29% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None] |
285.57 | 289.24 | +1.28% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] |
549.69 | 556.71 | +1.28% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cpu] |
97.19 | 98.43 | +1.27% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] |
41,978 | 41,450 | -1.26% |
| ... | ... | ... | Showing 120 of 220 comparisons, sorted by absolute change. |
GPU
Compared 230 benchmarks. Regressions over 5%: 6. Improvements over 5%: 16.
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] |
43.18 | 191.33 | +343.06% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
882.75 | 2,972 | +236.65% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
1,918 | 525.18 | -72.61% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] |
59.82 | 87.99 | +47.10% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] |
81.04 | 112.49 | +38.81% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,595 | 3,306 | +27.36% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,959 | 3,669 | +24.02% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,013 | 2,312 | +14.88% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,037 | 2,334 | +14.59% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] |
4,315 | 3,724 | -13.70% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,149 | 3,565 | +13.20% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
893.52 | 1,004 | +12.36% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] |
230.45 | 252.35 | +9.51% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] |
766.90 | 699.81 | -8.75% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] |
774.02 | 720.43 | -6.92% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] |
423.82 | 452.25 | +6.71% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] |
346.71 | 368.06 | +6.16% |
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] |
471.76 | 444.38 | -5.80% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] |
6.6534 | 7.0315 | +5.68% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] |
175.29 | 165.75 | -5.44% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] |
4,243 | 4,461 | +5.14% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] |
355.18 | 373.01 | +5.02% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
791.16 | 830.70 | +5.00% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
459.28 | 481.46 | +4.83% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] |
369,553 | 351,941 | -4.77% |
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] |
49.79 | 47.47 | -4.66% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] |
33,427 | 31,972 | -4.35% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cuda] |
2,287 | 2,189 | -4.31% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] |
30,867 | 29,613 | -4.06% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] |
0.5268 | 0.5055 | -4.04% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
3,652 | 3,506 | -4.00% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] |
49.97 | 47.97 | -3.99% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] |
66,004 | 63,374 | -3.98% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
102.37 | 98.39 | -3.89% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
22.87 | 22.00 | -3.78% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] |
399.22 | 384.39 | -3.72% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] |
22.10 | 21.28 | -3.72% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] |
52.17 | 50.26 | -3.67% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] |
420.87 | 405.47 | -3.66% |
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] |
12.60 | 12.15 | -3.63% |
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] |
20.70 | 19.97 | -3.57% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,546 | 3,420 | -3.56% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] |
19,961 | 19,275 | -3.44% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] |
49.05 | 47.38 | -3.42% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] |
559.33 | 540.23 | -3.41% |
benchmarks/test_envs_benchmark.py::test_simple |
1.2402 | 1.1980 | -3.40% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] |
65,773 | 63,550 | -3.38% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... |
1,507 | 1,457 | -3.34% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] |
528.15 | 510.96 | -3.25% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] |
785.28 | 759.90 | -3.23% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] |
139.54 | 135.08 | -3.20% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] |
148.07 | 152.74 | +3.16% |
benchmarks/test_collectors_benchmark.py::test_single |
6.8126 | 6.6098 | -2.98% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] |
278.27 | 270.01 | -2.97% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] |
37,958 | 36,838 | -2.95% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
992.51 | 964.13 | -2.86% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] |
28,280 | 29,073 | +2.80% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] |
526.28 | 511.57 | -2.79% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
190.55 | 195.83 | +2.77% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-backward] |
129.37 | 132.92 | +2.75% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] |
1,269 | 1,235 | -2.73% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] |
53.49 | 52.04 | -2.73% |
benchmarks/test_envs_benchmark.py::test_transformed |
0.7200 | 0.7007 | -2.69% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] |
34,478 | 33,554 | -2.68% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] |
1.3319 | 1.3673 | +2.66% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] |
966.04 | 991.71 | +2.66% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] |
4.4987 | 4.6166 | +2.62% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] |
405.61 | 395.05 | -2.60% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] |
405.22 | 395.09 | -2.50% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] |
21,017 | 20,508 | -2.42% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] |
23.52 | 22.96 | -2.39% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] |
30,634 | 29,935 | -2.28% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] |
24,134 | 23,591 | -2.25% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] |
497.78 | 508.94 | +2.24% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
710.16 | 694.29 | -2.24% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
229.69 | 234.81 | +2.23% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-4] |
180.42 | 176.40 | -2.23% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] |
41,690 | 42,608 | +2.20% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[done_aware] |
1,541 | 1,574 | +2.16% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64] |
3.0358 | 3.1005 | +2.13% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] |
41.31 | 42.17 | +2.09% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] |
695.32 | 680.93 | -2.07% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] |
10.41 | 10.62 | +2.00% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] |
812.84 | 828.96 | +1.98% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] |
31,913 | 31,284 | -1.97% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] |
282.66 | 288.19 | +1.96% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] |
1,907 | 1,870 | -1.95% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] |
76.27 | 77.74 | +1.94% |
benchmarks/test_collectors_benchmark.py::test_async |
10.85 | 11.06 | +1.90% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] |
0.5857 | 0.5968 | +1.89% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] |
57,437 | 56,355 | -1.88% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] |
630.00 | 641.66 | +1.85% |
benchmarks/test_collectors_benchmark.py::test_single_with_rb |
6.0020 | 5.8913 | -1.84% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] |
23.18 | 22.76 | -1.83% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] |
81.27 | 79.78 | -1.83% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] |
18,924 | 18,579 | -1.83% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] |
281.22 | 276.11 | -1.82% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16] |
47.36 | 48.20 | +1.79% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] |
21.78 | 21.39 | -1.78% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] |
18.07 | 18.39 | +1.75% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] |
3,609 | 3,547 | -1.74% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] |
120.93 | 118.85 | -1.72% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-no-buffers-True] |
0.6061 | 0.5957 | -1.71% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] |
100.73 | 99.03 | -1.69% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] |
22,220 | 21,849 | -1.67% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] |
236.23 | 240.14 | +1.66% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[no_done] |
1,567 | 1,593 | +1.65% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] |
77,013 | 78,279 | +1.64% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] |
18,264 | 17,965 | -1.63% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] |
161.13 | 163.72 | +1.61% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-buffers-True] |
0.5250 | 0.5166 | -1.60% |
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] |
863.55 | 849.72 | -1.60% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] |
33,855 | 33,314 | -1.60% |
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] |
11,861 | 11,673 | -1.59% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-no-buffers-False] |
0.6971 | 0.6860 | -1.58% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-32-512] |
1,277 | 1,298 | +1.58% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] |
4,903 | 4,829 | -1.51% |
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False] |
862.74 | 850.07 | -1.47% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] |
822.03 | 834.06 | +1.46% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] |
757.23 | 746.16 | -1.46% |
| ... | ... | ... | Showing 120 of 230 comparisons, sorted by absolute change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
After the xdist stack, tests-cpu wall time is dominated by the serial
process-spawning quarantine that runs after the ~5 min parallel bulk
(sum instead of max): 33-55 min depending on the Python version, with
test/test_collectors.py alone accounting for 66-79% of it.
Split TORCHRL_TEST_SHARD into three additional values and fan tests-cpu
out over them as a matrix dimension:
half of the previous "all" run);
The union of the three equals the previous "all" selection (verified
with pytest --collect-only: 29,587 tests, identical set, no overlap).
Numeric shards 1/2/3 and "all" are unchanged, so GPU jobs and local
usage keep their behavior; shard 2 still runs the whole quarantine.
Job wall becomes max(bulk, collectors, mp) instead of their sum. The
collectors shard may still exceed 30 min on 3.12; splitting that file
by class is deferred until this layout produces fresh duration data.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com