Repository navigation
Reuse dictionary value hashes in RepartitionExec - #26816
Draft
Rich-T-kid wants to merge 3 commits into
Draft
Rich-T-kid wants to merge 3 commits into
Rich-T-kid wants to merge 3 commits into
Conversation
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
#26816 (comment) @asolimando I think I found it 😄 |
Contributor
Author
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"Results will be posted here when complete File an issue against this benchmark runner |
Rich-T-kid
commented
Oct 27, 2026
Rich-T-kid
left a comment
Rich-T-kid
left a comment
Contributor
Author
There was a problem hiding this comment.
this PR itself shows good promise but I think a PR that breaks apart the hash vs spreading of the keys will make the other PRs related to #26780 far smaller & simpler & non dependent on each other
| #[cfg(not(feature = "force_hash_collisions"))] | ||
| use arrow::{downcast_dictionary_array, downcast_primitive_array}; | ||
| use foldhash::fast::FixedState; | ||
| #[cfg(not(feature = "force_hash_collisions"))] |
Contributor
Author
There was a problem hiding this comment.
we can trim this. we just need to break apart the actually hashing of values from the spreading of the keys.
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 14d1378 (14d1378) to 5d9b7fd diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "5d9b7fd45ba0e27225effa91fd4e7a0ae4eeeff9"
changed:
ref: "14d1378f2bdc9b4f64a5c43dbf189d031a706fa0"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/repartition-dictionary-hash-cache
branch
from
October 27, 2026 18:38
14d1378 to
81d2bb6
Compare
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d" |
github-actions
Bot
added
common
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 81d2bb6 (81d2bb6) to acbad14 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "acbad1400bb75b5837a2c389ea622d5ff4b6f77c"
changed:
ref: "81d2bb625063861415e9382ceff01df526e9e55d"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/repartition-dictionary-hash-cache
branch
from
October 27, 2026 19:40
81d2bb6 to
d4b5712
Compare
Member
It looks to me that you are working the same angle as @jayzhan211 in #26703 (Q18/Q33/Q34 ~ 1.8–1.9×, total −28 % vs −29 % here), maybe you two could join forces on this one? I didn't have time to go through all the details of both PRs, but if you check and still have doubts, one quick way to check is to rebase this on #26703 and re-run Also, both runs compare against a baseline with |
Rich-T-kid
force-pushed
the
rich-T-kid/repartition-dictionary-hash-cache
branch
from
October 27, 2026 19:56
d4b5712 to
4b4472c
Compare
Rich-T-kid
changed the title
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "4b4472c7bcc38d5b478cc7fbf64c6780efc979e4" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 4b4472c (4b4472c) to 4ede583 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "4b4472c7bcc38d5b478cc7fbf64c6780efc979e4"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d648959 (d648959) to 9f6f79f diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "9f6f79f717429e45575fcae6223bdc31671d3890"
changed:
ref: "d648959fac2d32cb1daf4b0f9f4b51ff461f0829"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/repartition-dictionary-hash-cache
branch
from
October 28, 2026 04:40
d648959 to
887fb88
Compare
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned |
Reuse dictionary value hashes while the values Arc is unchanged so operators do not rehash the same dictionary for every batch.
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"Results will be posted here when complete File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/repartition-dictionary-hash-cache
branch
from
October 28, 2026 04:52
887fb88 to
ffec3be
Compare
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpcds |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Runner log (last 40 lines)Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Runner log (last 40 lines)Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 887fb88 (887fb88) to 1284065 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "12840651e7e6736f6eb055b3206622397308891b"
changed:
ref: "887fb88c3cce6152dc2399f608458b7bffad1965"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing ffec3be (ffec3be) to af752e5 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "af752e5d5620b13e759e1713c6c0b72c43b4fb9b"
changed:
ref: "ffec3beac0409b03770b0e5449ebf41aef6bda02"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #26814.
Depends on #26817.
Rationale for this change
Dictionary batches often share one values array while only their keys change.
RepartitionExeccurrently hashes all dictionary values again for every batch.What changes are included in this PR?
BatchPartitionerkeeps oneCachingArrayHasherper hash expression. Dictionary columns reuse value hashes while the valuesArcis unchanged and replace the cache when it changes. Non-dictionary columns use the existing hashing path, and composite-key hashing keeps the same combine order.The operator commit is 76 insertions and 8 deletions. The rest of the stacked diff is #26817.
Results
The original hosted GKE run compared
9f6f79f7withd648959f, with RLE-to-dictionary enabled on both sides.ClickBench improves Q33 by 1.93x, Q18 by 1.90x, Q34 by 1.84x, Q16 by 1.62x, and Q17 by 1.59x. TPC-DS Q72 improves by 1.06x. No primary-summary query regresses.
These results predate the helper and main rebases. The operator logic is unchanged, but this is not final-head benchmark evidence.
What is the testing strategy for this PR?
One end-to-end test covers shared dictionary values across batches, null keys, and partition routing. It passes on the rebased head.
Are there any user-facing changes?
No.