Enable dynamic filters for range-partitioned joins#23854
Conversation
…/hash-join-dynamic-filter-with-range-partition
| let mut filters: Vec<Arc<dyn PhysicalExpr>> = Vec::with_capacity(thresholds.len()); | ||
|
|
||
| let mut prev_sort_expr: Option<Arc<dyn PhysicalExpr>> = None; | ||
| for (sort_expr, value) in sort_exprs.iter().zip(thresholds.iter()) { |
There was a problem hiding this comment.
meganit: this is a pretty long for loop with a lot of logic in it, is it possible to make this more concise? Specifically I think having a lot of mutable variables can make this a bit hard to reason about - but that could just be me & my smooth brain lol
There was a problem hiding this comment.
actually I just move the TopK::build_filter_expression to here, so this can be shared to build the comparison expression.
|
cc @gene-bordegaray @jayshrivastava @stuhood @adriangb — kindly pinging on this one. Please take a look if you have time, thanks! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23854 +/- ##
==========================================
- Coverage 80.71% 80.71% -0.01%
==========================================
Files 1090 1091 +1
Lines 369398 370486 +1088
Branches 369398 370486 +1088
==========================================
+ Hits 298170 299038 +868
- Misses 53458 53620 +162
- Partials 17770 17828 +58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Which issue does this PR close?
Rationale for this change
Partitioned hash joins build one dynamic filter per build partition. Existing routing uses
hash(key) % N, which cannot reproduce a Range partitioning layout.Compatible Range co-partitioned joins instead need to route probe rows using their existing ordering and split points.
What changes are included in this PR?
CASEexpression that routes probe rows to the corresponding partition filter using the Range ordering and split points.Are these changes tested?
unit test.
Are there any user-facing changes?
no