Repository navigation
WIP: Reuse dictionary value hashes in hash joins - #26825
Rich-T-kid wants to merge 2 commits into
Conversation
0917893 to
8fccf69
Compare
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5" |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
run benchmark clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmark clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 8fccf69 (8fccf69) to 4ede583 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"
changed:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmark clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"
changed:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 4ede583 (4ede583) to 8fccf69 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"
changed:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 4ede583 (4ede583) to 8fccf69 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "8fccf69b78cfcddf53df0025958176ecdca556b5"
changed:
ref: "4ede5833f68dad5bc8169492dedc5749fcc4b8d5"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
WIP: this is stacked on #26817. Full-suite regression checks are complete; a targeted dictionary-key join benchmark is still needed.
Why
With Parquet RLE-to-dictionary enabled, consecutive batches can share the same dictionary values.
HashJoinExeccurrently hashes those values again for every build and probe batch. This addresses the hash-join part of #26780 and closes #26815. #26816 demonstrates the cost of repeated value hashing in repartitioning, but this PR intentionally uses a narrower join-local design.What changed
A small cache keeps only the last dictionary values
Arcand its hashes for each join key. Pointer equality identifies shared values; a different values array is hashed normally and replaces the entry. Build and probe state each own their cache, so the fixed hash state is implicit in the cache lifetime. Non-dictionary columns and collision checking are unchanged.The full stacked GitHub diff, including #26817, is 116 additions and 16 deletions.
Results
All hosted comparisons used RLE-to-dictionary on both exact refs (
4ede5833→8fccf69b).These suites are useful regression checks, but they do not measure the target workload: TPC-H/TPC-DS join on numeric surrogate keys and ClickBench has no joins. A low/medium-cardinality dictionary-key join benchmark is still needed to quantify the speedup.
Test coverage