Search before asking
Paimon version
master
Compute Engine
Spark (TopN / ORDER BY ... LIMIT pushdown)
Minimal reproduce step
On an append-only table, run a query that pushes a single-key TopN into Paimon: SELECT * FROM t ORDER BY col DESC LIMIT 1 (Spark's DESC defaults to NULLS LAST), or an explicit ORDER BY col ASC NULLS LAST LIMIT 1. Arrange for at least two splits where one split's col is entirely NULL (its files' min/max are null while the null count is tracked, so minmaxAvailable still passes) and another split holds the real top value.
TopNDataSplitEvaluator gates: append-only table, one sort key, a min/max-comparable type, limit ≤ 100.
What doesn't meet your expectations?
The query returns a NULL row instead of the true top value.
TopNDataSplitEvaluator.pickTopNSplits orders the splits by the sort column's aggregate min/max and keeps the first limit. In the NULLS LAST branches the comparison falls to ascCompare / descCompare, which treat a null min/max as the smallest value. A split whose sort column is entirely NULL therefore sorts to the FRONT, takes one of the kept limit slots, and displaces the split that holds the real top value. With LIMIT 1 and two or more splits, the reader keeps only the all-null split and emits a NULL row.
A related defect in the same comparator: ascCompare / descCompare / nullsFirstCompare / nullsLastCompare return -1 when the left operand is null without checking the right, so compare(x, y) and compare(y, x) can both return -1 for two nulls. That violates the Comparator contract and can surface as IllegalArgumentException: Comparison method violates its general contract during the sort.
Expected: under NULLS LAST a split whose sort column is provably all null (null count equals row count) is the worst candidate and sorts last, so it never displaces a split with real values. A split whose bound is merely unknown (null min/max but not provably all-null, e.g. stats.mode=counts) must still be read, so it sorts first conservatively.
Anything else?
Fix direction: add an allNull flag (null count equals row count) to the split wrapper and, in the NULLS LAST branches, order all-null splits last before falling back to the min/max comparison; and return 0 when both comparator operands are null so the comparator obeys its contract.
Are you willing to submit a PR?
Search before asking
Paimon version
master
Compute Engine
Spark (TopN / ORDER BY ... LIMIT pushdown)
Minimal reproduce step
On an append-only table, run a query that pushes a single-key TopN into Paimon:
SELECT * FROM t ORDER BY col DESC LIMIT 1(Spark's DESC defaults to NULLS LAST), or an explicitORDER BY col ASC NULLS LAST LIMIT 1. Arrange for at least two splits where one split'scolis entirely NULL (its files' min/max are null while the null count is tracked, sominmaxAvailablestill passes) and another split holds the real top value.TopNDataSplitEvaluatorgates: append-only table, one sort key, a min/max-comparable type, limit ≤ 100.What doesn't meet your expectations?
The query returns a NULL row instead of the true top value.
TopNDataSplitEvaluator.pickTopNSplitsorders the splits by the sort column's aggregate min/max and keeps the firstlimit. In the NULLS LAST branches the comparison falls toascCompare/descCompare, which treat a null min/max as the smallest value. A split whose sort column is entirely NULL therefore sorts to the FRONT, takes one of the keptlimitslots, and displaces the split that holds the real top value. WithLIMIT 1and two or more splits, the reader keeps only the all-null split and emits a NULL row.A related defect in the same comparator:
ascCompare/descCompare/nullsFirstCompare/nullsLastComparereturn-1when the left operand is null without checking the right, socompare(x, y)andcompare(y, x)can both return-1for two nulls. That violates theComparatorcontract and can surface asIllegalArgumentException: Comparison method violates its general contractduring the sort.Expected: under NULLS LAST a split whose sort column is provably all null (null count equals row count) is the worst candidate and sorts last, so it never displaces a split with real values. A split whose bound is merely unknown (null min/max but not provably all-null, e.g.
stats.mode=counts) must still be read, so it sorts first conservatively.Anything else?
Fix direction: add an
allNullflag (null count equals row count) to the split wrapper and, in the NULLS LAST branches, order all-null splits last before falling back to the min/max comparison; and return0when both comparator operands are null so the comparator obeys its contract.Are you willing to submit a PR?