[format] Decline ORC pushdown of empty IN/NOT IN instead of crashing the reader - #10015
Open
zhuxiangyi wants to merge 1 commit into
Open
zhuxiangyi wants to merge 1 commit into
zhuxiangyi wants to merge 1 commit into
Conversation
…the reader
PredicateBuilder.in(idx, emptyList()) is a legitimate always-false leaf, and
notIn inherits it through negate(). OrcPredicateFunctionVisitor.visitIn turned
that empty list into an empty Object[] and handed it to Hive's
SearchArgument.Builder.in(...), which rejects a zero-length call with
IllegalArgumentException("Can't create in expression with no arguments") while
the reader is being created, so any ORC read carrying IN () or NOT IN ()
failed outright.
visitIn now declines the pushdown (Optional.empty()) on an empty literal list,
the same way it already declines every other predicate it cannot express.
Correctness is unaffected: engines re-evaluate data filters after the scan,
so the always-false / always-true semantics still hold; only the stripe-level
pruning for that predicate is given up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Bug fix: any ORC read that carries an
IN ()orNOT IN ()predicate fails while the reader is being created.PredicateBuilder.in(idx, emptyList())is a legitimate always-false leaf (the builder already special-cases an empty list, exactly as it does for more than 20 literals), andnotIninherits it throughnegate(). ButOrcPredicateFunctionVisitor.visitInturns that empty list into an emptyObject[]and hands it to Hive'sSearchArgument.Builder.in(...), which rejects a zero-length call outright:So a predicate that should simply match nothing (or everything, for
NOT IN ()) instead takes the whole read down. Nothing nested orTransform-based is needed to hit it — a plain index-basedin/notInon an ORC table is enough.The fix has
visitIndecline the pushdown (Optional.empty()) on an empty literal list, the same way the visitor already declines every other predicate it cannot express in a SARG.visitNotIndelegates tovisitIn, so it is covered by the same branch. Correctness is unaffected: engines re-evaluate data filters after the scan (Spark keeps every data filter as a post-scan predicate; Paimon's own row-level filtering is opt-in viaTableRead.executeFilter()), so the always-false / always-true semantics still hold — only the stripe-level pruning for that one predicate is given up.This is the ORC counterpart of the
ParquetFilters.visitIn/visitNotInguard added in #9423 (parquet-mr'sSetColumnFilterPredicaterefuses an empty set the same way). It is independent of that PR and predates it.Tests
OrcFilterConverterTest.testInPredicateWithEmptyValuesIsNotPushedDown—in(idx, [])andnotIn(idx, [])both resolve toOptional.empty(); also pins that an emptyInleaf insideor(...)/and(...)declines the whole compound rather than crashing (the builder only foldsAlwaysFalse.INSTANCE, so the leaf does survive into a compound).OrcFormatReadWriteTest.testEmptyInAndNotInPredicatesDoNotCrashTheReader— writes a real ORC file and opens a reader with each predicate throughOrcFileFormat.createReaderFactory; reproduces theIllegalArgumentExceptionbefore the fix, returns every row unfiltered after it.API and Format
No changes.
Documentation
No changes.