[AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter#2164
Open
weimingdiit wants to merge 1 commit intoapache:masterfrom
Open
Conversation
22c1061 to
8116033
Compare
8116033 to
e19e1a9
Compare
…s via scan pruning and post-scan native filter
e19e1a9 to
4fe640b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2163
Rationale for this change
The previous behavior was too conservative for Iceberg scans with residual filters. Even when the scan could still be executed natively and the remaining filter logic could be handled above the scan, the planner would fall back entirely.
This PR improves native coverage for Iceberg reads by:
This is an incremental improvement to Iceberg native execution, not full Iceberg feature coverage.
What changes are included in this PR?
This PR:
alwaysTrueresidual filtersIcebergScanPlanto carrypruningPredicatesNativeIcebergTableScanExecNativeFilterpathIN-based pruningSupported predicate scope in this PR
The scan-pruning conversion added here supports a limited subset of Iceberg expressions, including:
ANDORNOTIS NULLIS NOT NULLIS NANNOT NAN=,!=,<,<=,>,>=INNOT INThe current implementation intentionally avoids pushing some types through scan pruning, including:
StringTypeBinaryTypeDecimalTypeUnsupported predicates are not pushed into scan pruning and are instead left for post-scan native filtering.
How was this patch tested?
Integration coverage was added in
AuronIcebergIntegrationSuite