Fix join filter pushdown - #2249
Draft
ianton-ru wants to merge 3 commits into
Draft
Conversation
…in header Unused-column removal and `JoinStepLogical` aliases can hide a one-sided `WHERE` from `get_available_columns_for_filter`. Include those names so existing split and remap can push the predicate under the JOIN. Co-authored-by: Cursor <cursoragent@cursor.com>
…r can prune files Initiator listing runs on the wrap subquery (`SELECT cols FROM icebergCluster`), which previously had no WHERE. A left-only predicate on `count()` of `SELECT * … JOIN` never reached min/max file listing. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the duplicated WHERE walker and the PK-walk-through-JOIN remapping. Wrap listing still uses collectFiltersForAnalysis and tryAddClusterWrapFilter. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix join filter pushdown
Documentation entry for user-facing changes
Solved #2245
Push left-only JOIN filters when column names do not match the join header
A left-only
WHEREoncount()ofSELECT * … JOINwas not pushed under the JOIN. The filtered column often disappeared from the JOIN output (unused-column removal aftercount()ofSELECT *), while the Filter DAG still referenced it — sometimes under an identifier name such as__table1.arather thana.get_available_columns_for_filterrequired the name to appear in the JOIN header, sosplitActionsForJOINFilterPushDownnever saw it.That is not Iceberg-specific. On MergeTree the same shape skipped
Prewhere/ index analysis on the left read. The filter still ran after the JOIN, so the result was correct but the left table was scanned without the predicate.JoinStepLogicalcan also alias a side input (a) to a JOIN-output / filter name (__table1.a). Pushdown matches filter inputs against the available-column list, thenfix_predicate_for_join_logical_stepremaps aliases back to input names. The output alias has to be listed or the split never runs.JOIN filter pushdown
In
tryPushDownOverJoinStep:JoinStepLogical, output actions thatfromLeft()/fromRight()are added to that list (including aliases). The existing split and remap then push the predicate under the JOIN.Covered by
04673_join_filter_pushdown_count_subquery.sql: MergeTree left + Memory right,count()ofSELECT * … LEFT JOIN … WHERE foo.a < 40(and the same with an extra(SELECT * FROM t_left) AS foowrap).EXPLAIN actions = 1must containPrewhere.icebergClusterfile listingicebergCluster(IStorageCluster) lists files on the initiator. The planner wraps the left cluster table so remotes do not get the JOIN (SELECT cols FROM icebergCluster). That wrap had noWHERE, so initiator listing stayed unfiltered even after JOIN pushdown.Left-only
WHERE/PREWHEREis copied onto the wrap withremoveExpressionsThatDoNotDependOnTableIdentifiers(same helper asIStorageCluster::updateQueryWithJoinToSendIfNeeded). Wrap planning runscollectFiltersForAnalysis. Listing-only filters are attached on the wrap source without adding aFilterStepthat would drop unused wrap columns.Covered by
test_cluster_join_filter_minmax_pruning.py:IcebergMinMaxIndexPrunedFilesforicebergS3Clusterwith a plainWHERE, withJOIN … WHERE, and with outercount()ofSELECT * … JOIN … WHERE.CI/CD Options
Exclude tests:
Regression jobs to run: