feat: support filtered batches in LimitedBatchCoalescer#23856
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23856 +/- ##
=======================================
Coverage ? 80.70%
=======================================
Files ? 1090
Lines ? 370214
Branches ? 370214
=======================================
Hits ? 298786
Misses ? 53617
Partials ? 17811 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing filter-push-batch-with-filter (23beb6a) to 05b7e11 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing filter-push-batch-with-filter (23beb6a) to 05b7e11 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing filter-push-batch-with-filter (23beb6a) to 05b7e11 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Which issue does this PR close?
push_batch_with_filterin LimitedBatchCoalescer /FilterExec#19876.Rationale for this change
Allow
FilterExecto use Arrow's optimized filtered coalescing path and avoid materializing an intermediate filtered batch when supported.What changes are included in this PR?
LimitedBatchCoalescer::push_batch_with_filterand use it inFilterExec.fetchsemantics for filtered rows.Are these changes tested?
Yes. New unit tests cover the added behavior.
Are there any user-facing changes?
No changes to SQL behavior or results. This adds a new method to
LimitedBatchCoalescer.