Skip to content

[core] Do not prune TopN splits with unknown sort-column statistics - #10039

Merged
JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/topn-stats-unknown-splits
Sep 21, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/topn-stats-unknown-splits

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #10038

TopNDataSplitEvaluator supports Spark single-key TopN pushdown (ORDER BY col LIMIT n on an append-only table, a min/max-comparable sort type, limit ≤ 100). It orders the splits by the sort column's aggregate min/max/null-count and keeps the best limit. It reads those aggregates from DataSplit.minValue / maxValue / nullCount, which aggregate across a split's files by silently skipping files without statistics. A split containing a stats-less file (a stats.mode=counts file, or one written before the column was added) therefore gets an aggregate that looks like a true bound but is not: the file's real values are ignored. Ordering by that fabricated bound can prune a split that may hold a row in the true top-N. On a pure counts-mode table every bound is fabricated, all splits tie, and an arbitrary subset survives, so the split with the true top row can be dropped and ORDER BY ... LIMIT n silently misses rows.

This aggregates the statistics per file inside the evaluator and tracks completeness. If any file of a split lacks min/max or null-count for the sort column, the split has no trustworthy bound and is read unconditionally (never pruned), the same way splits that fail the min/max gate are already handled. A file that is provably all-null for the column (its null count equals its row count) legitimately contributes nothing to min/max and does not make the split incomplete. For a split whose files all have statistics, the aggregated min/max/null-count is the same as before, so fully-statted tables are unaffected.

Tests

TableScanTest.testPushDownTopNMultiFileSplitWithMixedStatsIsAlwaysRead: one split holds a full-statistics file (min 50) plus a counts-mode file with no min/max. Before the fix DataSplit.minValue skipped the counts file and reported 50, so at LIMIT 1 the split ranked last and was pruned; the test asserts it is now always read (the counts file may hold a value below every other split's min). It fails against the pre-fix code, which returns only the other split.

TableScanTest.testPushDownTopNCountsModeStatsDisablesPruning: every split is counts-mode (min/max unknown), so no bound is trustworthy and all splits must be read; the pre-fix code keeps an arbitrary subset at LIMIT 1.

TableScanTest.testPushDownTopNNullsLastKeepsStatsUnknownSplitFirst (updated): an unknown-bound split is always read, and the best known-bound split still wins the remaining limit slot.

TopNDataSplitEvaluator orders splits by the sort column's aggregate
min/max/null-count and keeps the best `limit` ones, which is exact only
if every aggregate is a true bound of the split. DataSplit.minValue and
friends aggregate by silently skipping files without statistics, so a
split containing a stats.mode=counts (or otherwise stats-less) file
produced a value that looked like a bound but was not one: on a pure
counts-mode table all splits tie and an arbitrary subset is kept — the
split holding the true top row can be dropped and ORDER BY ... LIMIT n
silently misses rows; a mixed-stats table could drop the known-bound
split for the same reason.

Aggregate the statistics per file inside the evaluator and treat a
split whose statistics are incomplete (any file lacking min/max or
null-count for the sort column) like a gate failure: always read it,
never order it against the others. A file with nullCount equal to its
row count provably holds no non-null value, so it legitimately
contributes nothing to min/max — all-null columns and files written
before an ALTER TABLE ADD COLUMN stay prunable, and splits whose
statistics are complete keep the exact ordering behavior.

Assisted-by: GLM-5.3

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)
Implementation: CLEAN

This prevents wrong results for Spark ORDER BY ... LIMIT when a split contains a file without sort-column statistics. The evaluator now treats that split as unprunable instead of ranking a fabricated aggregate bound, while preserving pruning for complete statistics and provably all-null files. The mixed-statistics and counts-only regression cases cover the failure described in #10038. I found no blocking issue in the changed path.

@JingsongLi
JingsongLi merged commit 7120607 into apache:master Sep 21, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] TopN pushdown prunes splits using a fabricated bound from stats-less files, dropping the true top row

2 participants