[core] Do not prune TopN splits with unknown sort-column statistics - #10039
Merged
JingsongLi merged 1 commit intoSep 21, 2026
Merged
Conversation
TopNDataSplitEvaluator orders splits by the sort column's aggregate min/max/null-count and keeps the best `limit` ones, which is exact only if every aggregate is a true bound of the split. DataSplit.minValue and friends aggregate by silently skipping files without statistics, so a split containing a stats.mode=counts (or otherwise stats-less) file produced a value that looked like a bound but was not one: on a pure counts-mode table all splits tie and an arbitrary subset is kept — the split holding the true top row can be dropped and ORDER BY ... LIMIT n silently misses rows; a mixed-stats table could drop the known-bound split for the same reason. Aggregate the statistics per file inside the evaluator and treat a split whose statistics are incomplete (any file lacking min/max or null-count for the sort column) like a gate failure: always read it, never order it against the others. A file with nullCount equal to its row count provably holds no non-null value, so it legitimately contributes nothing to min/max — all-null columns and files written before an ALTER TABLE ADD COLUMN stay prunable, and splits whose statistics are complete keep the exact ordering behavior. Assisted-by: GLM-5.3
JingsongLi
reviewed
Sep 21, 2026
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Requirement fit: SUPPORTED (triage: GO)
Implementation: CLEAN
This prevents wrong results for Spark ORDER BY ... LIMIT when a split contains a file without sort-column statistics. The evaluator now treats that split as unprunable instead of ranking a fabricated aggregate bound, while preserving pruning for complete statistics and provably all-null files. The mixed-statistics and counts-only regression cases cover the failure described in #10038. I found no blocking issue in the changed path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #10038
TopNDataSplitEvaluatorsupports Spark single-key TopN pushdown (ORDER BY col LIMIT non an append-only table, a min/max-comparable sort type, limit ≤ 100). It orders the splits by the sort column's aggregate min/max/null-count and keeps the bestlimit. It reads those aggregates fromDataSplit.minValue/maxValue/nullCount, which aggregate across a split's files by silently skipping files without statistics. A split containing a stats-less file (astats.mode=countsfile, or one written before the column was added) therefore gets an aggregate that looks like a true bound but is not: the file's real values are ignored. Ordering by that fabricated bound can prune a split that may hold a row in the true top-N. On a pure counts-mode table every bound is fabricated, all splits tie, and an arbitrary subset survives, so the split with the true top row can be dropped andORDER BY ... LIMIT nsilently misses rows.This aggregates the statistics per file inside the evaluator and tracks completeness. If any file of a split lacks min/max or null-count for the sort column, the split has no trustworthy bound and is read unconditionally (never pruned), the same way splits that fail the min/max gate are already handled. A file that is provably all-null for the column (its null count equals its row count) legitimately contributes nothing to min/max and does not make the split incomplete. For a split whose files all have statistics, the aggregated min/max/null-count is the same as before, so fully-statted tables are unaffected.
Tests
TableScanTest.testPushDownTopNMultiFileSplitWithMixedStatsIsAlwaysRead: one split holds a full-statistics file (min 50) plus a counts-mode file with no min/max. Before the fixDataSplit.minValueskipped the counts file and reported 50, so atLIMIT 1the split ranked last and was pruned; the test asserts it is now always read (the counts file may hold a value below every other split's min). It fails against the pre-fix code, which returns only the other split.TableScanTest.testPushDownTopNCountsModeStatsDisablesPruning: every split is counts-mode (min/max unknown), so no bound is trustworthy and all splits must be read; the pre-fix code keeps an arbitrary subset atLIMIT 1.TableScanTest.testPushDownTopNNullsLastKeepsStatsUnknownSplitFirst(updated): an unknown-bound split is always read, and the best known-bound split still wins the remaining limit slot.