[CI] Shard the SOTA smoke tests into two parallel jobs#3980
Open
vmoens wants to merge 2 commits into
Open
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3980
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Unclassified FailureAs of commit 2ff191d with merge base a74b478 ( NEW FAILURE - The following job has failed:
UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
github-actions
Bot
added
the
CI
This was referenced Jul 9, 2026
meta-cla
Bot
added
the
CLA Signed
Contributor
Benchmark Results: PR
|
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
49.35 | 195.58 | +296.33% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] |
194.38 | 50.79 | -73.87% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] |
59.89 | 87.99 | +46.92% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
3,407 | 2,533 | -25.63% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,793 | 3,504 | +25.45% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,478 | 2,655 | -23.67% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] |
36.05 | 28.51 | -20.94% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
930.69 | 742.71 | -20.20% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
868.49 | 1,003 | +15.46% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512] |
29.01 | 33.07 | +14.02% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
370.29 | 419.93 | +13.41% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] |
552.07 | 624.31 | +13.09% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-1] |
71.34 | 80.66 | +13.06% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
1,684 | 1,902 | +12.95% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] |
7.0362 | 6.2137 | -11.69% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] |
10.53 | 9.3204 | -11.47% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] |
468.18 | 517.75 | +10.59% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,854 | 3,112 | +9.07% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
1,844 | 1,989 | +7.88% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] |
499.79 | 538.86 | +7.82% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] |
125.86 | 135.02 | +7.28% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_catframes_offline |
543.79 | 581.71 | +6.97% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
1,888 | 2,019 | +6.93% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] |
12.12 | 11.35 | -6.35% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
3,070 | 2,883 | -6.07% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
2,788 | 2,946 | +5.67% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False] |
44,538 | 47,016 | +5.56% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] |
72.14 | 68.32 | -5.30% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] |
119.62 | 125.93 | +5.28% |
benchmarks/test_envs_benchmark.py::test_simple |
1.8013 | 1.7077 | -5.20% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cpu] |
98.29 | 93.52 | -4.85% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] |
23,669 | 22,522 | -4.85% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] |
30,575 | 32,018 | +4.72% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] |
40,098 | 38,216 | -4.69% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] |
2,381 | 2,269 | -4.69% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] |
540.84 | 565.84 | +4.62% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
1,965 | 2,055 | +4.57% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,832 | 2,955 | +4.34% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] |
1,776 | 1,852 | +4.27% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] |
33,015 | 31,609 | -4.26% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
248.54 | 259.03 | +4.22% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
1,087 | 1,041 | -4.22% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[done_aware] |
1,467 | 1,528 | +4.20% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] |
440.38 | 458.07 | +4.02% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,700 | 2,806 | +3.92% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] |
19.49 | 20.25 | +3.88% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] |
114.93 | 119.13 | +3.65% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] |
773.26 | 746.11 | -3.51% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] |
1,728 | 1,787 | +3.45% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] |
548.35 | 567.04 | +3.41% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] |
1,065 | 1,100 | +3.36% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] |
689.64 | 712.26 | +3.28% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] |
30,796 | 29,786 | -3.28% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] |
58,442 | 56,545 | -3.24% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] |
643.05 | 663.58 | +3.19% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_action_chunk_transform[no_done] |
1,504 | 1,552 | +3.19% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] |
113.23 | 116.71 | +3.08% |
benchmarks/test_replaybuffer_benchmark.py::test_replay_buffer_direct_client_identity |
7,728,148 | 7,965,301 | +3.07% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] |
20,610 | 21,230 | +3.01% |
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] |
24.58 | 23.88 | -2.85% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] |
4,333 | 4,456 | +2.84% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch] |
10.08 | 10.36 | +2.81% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] |
1.3590 | 1.3211 | -2.78% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] |
50,017 | 48,626 | -2.78% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] |
329.50 | 338.58 | +2.76% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] |
12,105 | 12,435 | +2.73% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] |
65,500 | 63,725 | -2.71% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] |
245.00 | 251.45 | +2.63% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] |
43,011 | 41,898 | -2.59% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] |
278.65 | 285.62 | +2.50% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] |
34,344 | 35,195 | +2.48% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] |
165.40 | 161.42 | -2.40% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] |
19,338 | 19,782 | +2.30% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
58.88 | 57.53 | -2.29% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] |
23.15 | 23.68 | +2.25% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] |
174.75 | 170.86 | -2.23% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-lstm] |
0.8753 | 0.8560 | -2.21% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] |
567.75 | 555.39 | -2.18% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] |
30,399 | 29,737 | -2.18% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] |
50,337 | 49,269 | -2.12% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] |
553.30 | 564.81 | +2.08% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] |
2,242 | 2,196 | -2.03% |
benchmarks/test_envs_benchmark.py::test_serial |
0.5747 | 0.5861 | +1.99% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] |
216.46 | 220.68 | +1.95% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] |
38,542 | 37,799 | -1.93% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] |
328.99 | 335.33 | +1.93% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] |
35,177 | 34,514 | -1.88% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] |
64,636 | 63,434 | -1.86% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] |
37,724 | 37,028 | -1.85% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] |
309.25 | 314.93 | +1.84% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] |
52.36 | 51.40 | -1.83% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] |
18,196 | 17,864 | -1.82% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] |
122.62 | 120.42 | -1.79% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-backward] |
55.26 | 56.20 | +1.70% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-backward] |
27.61 | 28.06 | +1.63% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] |
207.50 | 210.85 | +1.62% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-4] |
180.76 | 183.68 | +1.61% |
benchmarks/test_envs_benchmark.py::test_parallel |
0.9401 | 0.9550 | +1.58% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] |
0.5914 | 0.6007 | +1.58% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] |
462.68 | 469.89 | +1.56% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] |
170.11 | 167.45 | -1.56% |
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] |
36.41 | 35.86 | -1.53% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] |
493.80 | 501.25 | +1.51% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] |
32,614 | 33,103 | +1.50% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] |
7,109 | 7,003 | -1.50% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
458.74 | 465.60 | +1.49% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] |
145.53 | 143.36 | -1.49% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[50-img_shape0-small] |
868.50 | 855.62 | -1.48% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
3.0136 | 3.0579 | +1.47% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] |
188.64 | 185.92 | -1.44% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] |
24.14 | 24.49 | +1.43% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] |
5,128 | 5,055 | -1.42% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] |
168.18 | 165.84 | -1.39% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] |
259.48 | 263.09 | +1.39% |
benchmarks/test_collectors_benchmark.py::test_sync |
16.72 | 16.49 | -1.39% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] |
284.09 | 280.14 | -1.39% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] |
88.59 | 87.37 | -1.38% |
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-None] |
94.11 | 92.82 | -1.37% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] |
692.78 | 702.28 | +1.37% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] |
7.9407 | 8.0462 | +1.33% |
| ... | ... | ... | Showing 120 of 220 comparisons, sorted by absolute change. |
GPU
Compared 230 benchmarks. Regressions over 5%: 12. Improvements over 5%: 16.
| Benchmark | main ops | PR ops | Change |
|---|---|---|---|
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] |
42.11 | 199.36 | +373.47% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] |
195.89 | 38.08 | -80.56% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] |
81.78 | 111.72 | +36.61% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
2,983 | 3,746 | +25.60% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,226 | 1,826 | -17.95% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
2,834 | 3,219 | +13.57% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] |
2,621 | 2,971 | +13.35% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] |
3,089 | 2,711 | -12.22% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] |
2,228 | 1,974 | -11.39% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] |
4,258 | 4,679 | +9.89% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] |
3,762 | 3,412 | -9.31% |
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] |
376.66 | 410.47 | +8.98% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] |
456.53 | 496.06 | +8.66% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] |
797.04 | 728.96 | -8.54% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] |
0.4936 | 0.5342 | +8.22% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] |
585.54 | 539.61 | -7.84% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,152 | 1,985 | -7.77% |
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] |
49.57 | 45.84 | -7.53% |
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] |
267.12 | 287.23 | +7.53% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] |
724.17 | 776.65 | +7.25% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] |
907.43 | 962.93 | +6.12% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] |
327.83 | 347.52 | +6.01% |
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] |
226.79 | 239.68 | +5.68% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] |
2,164 | 2,042 | -5.64% |
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] |
20.63 | 19.48 | -5.59% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
2,829 | 2,681 | -5.21% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] |
627.22 | 659.92 | +5.21% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] |
33,671 | 35,357 | +5.01% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] |
40,133 | 38,136 | -4.98% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] |
11,991 | 12,587 | +4.97% |
benchmarks/test_envs_benchmark.py::test_parallel |
0.5291 | 0.5545 | +4.80% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] |
1.3407 | 1.4018 | +4.55% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] |
186.32 | 194.77 | +4.53% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] |
19,802 | 20,696 | +4.52% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] |
455.23 | 474.53 | +4.24% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] |
158.79 | 152.58 | -3.91% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] |
671.01 | 696.81 | +3.84% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] |
7,017 | 7,279 | +3.73% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] |
283.75 | 273.36 | -3.66% |
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] |
1,873 | 1,941 | +3.64% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] |
21,538 | 20,766 | -3.59% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] |
733.69 | 759.36 | +3.50% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] |
861.60 | 891.36 | +3.45% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] |
4,456 | 4,302 | -3.45% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] |
166.36 | 171.97 | +3.37% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] |
358.06 | 346.22 | -3.31% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] |
52,477 | 50,751 | -3.29% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] |
22.89 | 22.15 | -3.21% |
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] |
23,142 | 23,884 | +3.20% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] |
22.94 | 22.22 | -3.14% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-triton-False-0-gru] |
20.72 | 20.07 | -3.14% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] |
1,045 | 1,013 | -3.06% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] |
473.93 | 459.43 | -3.06% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] |
17.66 | 18.18 | +2.99% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] |
1,023 | 1,053 | +2.96% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cuda] |
2,205 | 2,268 | +2.89% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64] |
1.2118 | 1.2465 | +2.87% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... |
1,465 | 1,507 | +2.86% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] |
19,658 | 20,219 | +2.86% |
benchmarks/test_envs_benchmark.py::test_transformed |
0.7234 | 0.7028 | -2.85% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] |
271.31 | 278.85 | +2.78% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] |
22,894 | 22,260 | -2.77% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] |
39,818 | 38,732 | -2.73% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] |
43,693 | 42,502 | -2.73% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] |
138.82 | 142.59 | +2.72% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] |
172.14 | 176.79 | +2.70% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] |
39,930 | 38,873 | -2.65% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] |
4,707 | 4,830 | +2.63% |
benchmarks/test_envs_benchmark.py::test_simple |
1.2441 | 1.2120 | -2.59% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] |
825.12 | 846.46 | +2.59% |
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None] |
863.80 | 884.97 | +2.45% |
benchmarks/test_replaybuffer_benchmark.py::TestWindowingTransformsBenchmark::test_catframes_offline |
585.50 | 599.52 | +2.40% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] |
822.35 | 842.02 | +2.39% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] |
170.80 | 174.82 | +2.35% |
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] |
11,757 | 11,480 | -2.35% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] |
21,285 | 20,797 | -2.29% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] |
52.36 | 53.54 | +2.24% |
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] |
1,352 | 1,322 | -2.24% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] |
65,753 | 64,290 | -2.23% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] |
404.22 | 395.22 | -2.23% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] |
43,865 | 42,904 | -2.19% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] |
18,980 | 18,566 | -2.18% |
benchmarks/test_envs_benchmark.py::test_serial |
0.4263 | 0.4354 | +2.15% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] |
166.61 | 170.18 | +2.14% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] |
75.01 | 76.55 | +2.06% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] |
51,619 | 50,566 | -2.04% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] |
48.00 | 48.97 | +2.02% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] |
372.10 | 364.70 | -1.99% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] |
28,457 | 27,905 | -1.94% |
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] |
236.33 | 240.86 | +1.92% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] |
23,835 | 24,285 | +1.89% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False] |
0.2221 | 0.2263 | +1.89% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] |
4.4227 | 4.5052 | +1.86% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] |
21.60 | 22.00 | +1.86% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] |
171.29 | 174.48 | +1.86% |
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] |
81.90 | 83.38 | +1.81% |
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cpu_sampler] |
90.23 | 88.60 | -1.80% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[200-img_shape1-large_batch] |
8.6393 | 8.7933 | +1.78% |
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] |
102.37 | 100.63 | -1.71% |
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] |
3,610 | 3,671 | +1.70% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] |
70.55 | 71.73 | +1.67% |
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] |
12.54 | 12.33 | -1.67% |
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] |
0.6025 | 0.5925 | -1.66% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] |
52.54 | 53.41 | +1.65% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] |
20.11 | 19.79 | -1.59% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64] |
3.0409 | 2.9930 | -1.58% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] |
508.56 | 500.76 | -1.53% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] |
783.80 | 771.80 | -1.53% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] |
17.47 | 17.74 | +1.51% |
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] |
122.64 | 120.81 | -1.49% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] |
52.61 | 51.84 | -1.48% |
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] |
12.24 | 12.06 | -1.46% |
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] |
15.21 | 15.43 | +1.43% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] |
3,485 | 3,534 | +1.42% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] |
496.07 | 489.08 | -1.41% |
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] |
23.63 | 23.96 | +1.39% |
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] |
103.93 | 105.36 | +1.38% |
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] |
705.03 | 695.79 | -1.31% |
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] |
32,133 | 31,716 | -1.30% |
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] |
387.28 | 382.34 | -1.27% |
| ... | ... | ... | Showing 120 of 230 comparisons, sorted by absolute change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
The SOTA smoke list runs as a single serial job (33-36 minutes) that
grows with every algorithm. Fan it out over two jobs: test_sota.py
selects an interleaved slice of the sorted command list via SOTA_SHARD
/ SOTA_NUM_SHARDS (unset = run everything, so local behavior is
unchanged), and the workflow adds shard: ["1", "2"] to the matrix.
Verified with pytest --collect-only: shards of 18 and 17 tests, union
identical to the full list, no overlap, and the two heavy dreamer
entries land on different shards thanks to the interleaving.
The per-recipe openvla unit tests now run on shard 1 only, and the
smoke/sota JSON reports get distinct per-shard file names (previously
both invocations wrote test-results-sota.json, the second overwriting
the first, so the flaky tracker never saw the smoke results).
Costs one duplicated GPU environment build per push; the wall-clock
halving of the dominant pytest phase outweighs it.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com