[Vedin]

Parquet statistics leave NaN out of floating-point columns: arrow-rs skips NaN when it computes min/max, as the Parquet spec asks writers to. DataFusion orders floats with arrow's total order, where a NaN with its sign bit clear (the 'NaN' literal) sorts above every other value, so NaN > 2 is true. Snowflake agrees: over {1.0, NaN, 1.0}, COUNT_IF(k > 2) is 1 and MAX(k) is NaN.

A row group, page or file holding 1.0 and NaN therefore has max 1.0. When that max is trusted, three things go wrong:

• WHERE c > 2 prunes it and loses the NaN row;
• MAX(c) is answered from file statistics as 1.0;
• min = max lets the opener read the column as the constant 1.0, which turns the NaN rows into 1.0.

The new sqllogictest reproduces all three on the base branch.

Change

A float column's Parquet maximum is now treated as unknown in three places:

• row-group pruning (RowGroupPruningStatistics::max_values);
• page pruning (PagesPruningStatistics::max_values);
• file statistics built from the footer (statistics_from_parquet_metadata, where max_value becomes Absent).

The minimum is still a valid lower bound for such a NaN, so c < x still prunes. A zero minimum is now marked inexact: writers record it as -0.0 whatever the sign of the stored zeros, so MIN over {0.0, 1.0} came back as -0.0 from footer statistics. Non-float columns are unchanged.

A NaN with the sign bit set (e.g. -'NaN', or inf - inf on x86) sorts below every number in arrow's total order. That is a separate, pre-existing difference from Snowflake (which treats every NaN alike), and this change does not alter it.

Cost

• Float columns lose pruning by their maximum, because Parquet statistics carry no NaN count that could prove a row group NaN-free.
• Filter cardinality estimates on float columns can no longer use the maximum. In parquet_statistics.slt, a float equality filter now estimates 5 rows instead of 1.

Tests

• row_group_float_maximum_does_not_prune (new): c1 > 15 keeps both row groups, and c1 < 5 still prunes by the minimum.
• float_maximum_is_absent_from_file_statistics (new): a real file holding [1.0, NaN] gives max Absent and min 1.0, while the int column keeps its max.
• parquet_float_nan_statistics.slt (new): count(*) WHERE c > 2 = 1, max = NaN, ORDER BY = 1, 1, NaN, and c < 1 = 0. The first three fail on the base branch.
• parquet_float_nan_statistics.slt also covers ORDER BY c DESC LIMIT 1 = NaN (the TopK path) and min over {0.0, 1.0} = 0.0. The zero case fails without the minimum change.
• Updated expectations that relied on the float maximum:
  • prune_f64_scalar_fun_and_gt, row-group and page variants (f >= 0.1);
  • two parquet_statistics.slt plans;
  • two opener tests, which now prune the float column by its minimum.
• Local runs:
  • all tests of datafusion, datafusion-datasource-parquet, datafusion-pruning and datafusion-physical-optimizer, including parquet_integration (223) and core_integration (1,092);
  • the full sqllogictest suite, 508 files;
  • clippy on the changed crates.

🤖 Generated with Claude Code