Skip to content

[format] Decline ORC pushdown of empty IN/NOT IN instead of crashing the reader - #10015

Open
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:orc-empty-in-crash
Open

zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:orc-empty-in-crash

Conversation

@zhuxiangyi

Copy link
Copy Markdown
Contributor

Purpose

Bug fix: any ORC read that carries an IN () or NOT IN () predicate fails while the reader is being created.

PredicateBuilder.in(idx, emptyList()) is a legitimate always-false leaf (the builder already special-cases an empty list, exactly as it does for more than 20 literals), and notIn inherits it through negate(). But OrcPredicateFunctionVisitor.visitIn turns that empty list into an empty Object[] and hands it to Hive's SearchArgument.Builder.in(...), which rejects a zero-length call outright:

java.lang.IllegalArgumentException: Can't create in expression with no arguments
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.in(SearchArgumentImpl.java:419)
	at org.apache.paimon.format.orc.filter.OrcFilters$In.add(OrcFilters.java:328)
	at org.apache.paimon.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:466)

So a predicate that should simply match nothing (or everything, for NOT IN ()) instead takes the whole read down. Nothing nested or Transform-based is needed to hit it — a plain index-based in/notIn on an ORC table is enough.

The fix has visitIn decline the pushdown (Optional.empty()) on an empty literal list, the same way the visitor already declines every other predicate it cannot express in a SARG. visitNotIn delegates to visitIn, so it is covered by the same branch. Correctness is unaffected: engines re-evaluate data filters after the scan (Spark keeps every data filter as a post-scan predicate; Paimon's own row-level filtering is opt-in via TableRead.executeFilter()), so the always-false / always-true semantics still hold — only the stripe-level pruning for that one predicate is given up.

This is the ORC counterpart of the ParquetFilters.visitIn/visitNotIn guard added in #9423 (parquet-mr's SetColumnFilterPredicate refuses an empty set the same way). It is independent of that PR and predates it.

Tests

  • OrcFilterConverterTest.testInPredicateWithEmptyValuesIsNotPushedDownin(idx, []) and notIn(idx, []) both resolve to Optional.empty(); also pins that an empty In leaf inside or(...)/and(...) declines the whole compound rather than crashing (the builder only folds AlwaysFalse.INSTANCE, so the leaf does survive into a compound).
  • OrcFormatReadWriteTest.testEmptyInAndNotInPredicatesDoNotCrashTheReader — writes a real ORC file and opens a reader with each predicate through OrcFileFormat.createReaderFactory; reproduces the IllegalArgumentException before the fix, returns every row unfiltered after it.

API and Format

No changes.

Documentation

No changes.

…the reader

PredicateBuilder.in(idx, emptyList()) is a legitimate always-false leaf, and
notIn inherits it through negate(). OrcPredicateFunctionVisitor.visitIn turned
that empty list into an empty Object[] and handed it to Hive's
SearchArgument.Builder.in(...), which rejects a zero-length call with
IllegalArgumentException("Can't create in expression with no arguments") while
the reader is being created, so any ORC read carrying IN () or NOT IN ()
failed outright.

visitIn now declines the pushdown (Optional.empty()) on an empty literal list,
the same way it already declines every other predicate it cannot express.
Correctness is unaffected: engines re-evaluate data filters after the scan,
so the always-false / always-true semantics still hold; only the stripe-level
pruning for that predicate is given up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant