GH-51097: Fix Parquet null counts for fixed-width leaves

[AnuragRaut08]

Rationale for this change

Parquet null-count statistics can be incorrect for fixed-width leaf columns nested below a repeated ancestor, such as list<struct<...>>.

When a list is null or empty, its descendant leaf does not produce a value in the leaf values buffer. DefLevelsToBitmap intentionally excludes these repeated-ancestor nulls from its null_count. MaybeCalculateValidityBits was using that value directly for the Parquet column statistics, causing the null count to be undercounted.

For example, a list<struct<string, int32>> column can report an incorrect null_count for the int32 leaf even though the encoded definition levels correctly represent the null and empty list entries.

What changes are included in this PR?

• Update MaybeCalculateValidityBits to calculate the total null count from batch_size - out_values_to_write.
• This includes nulls represented by repeated ancestors, such as null or empty lists, while preserving the existing validity bitmap behavior.
• Add a regression test covering a fixed-width int32 leaf under list<struct<...>>.

Are these changes tested?

Yes.

Added a regression test that verifies the int32 leaf reports:

• null_count = 3
• num_values = 2

The focused regression test passes:

StatisticsTest.FixedWidthLeafUnderListStructNullCount

The existing parquet-writer-test also passes.

Are there any user-facing changes?

Yes. This fixes incorrect Parquet column statistics for affected nested fixed-width columns. The change does not alter the encoded data or public APIs.

This PR contains a "Critical Fix".

This fixes a bug that produces incorrect Parquet statistics. The underlying data remains correct, but the reported null_count for affected fixed-width leaf columns can be incorrect.

Was AI used for this PR?

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human

• AI

• Not reviewed

• GitHub Issue: [Python][C++][Parquet] null_count statistics undercount for a fixed-width leaf under list<struct<...>> when lists are null or empty #51097

[github-actions]

⚠️ GitHub issue #51097 has been automatically assigned in GitHub to PR creator.

[AnuragRaut08]

Hi @wgtmac ,
I’ve addressed this point in the latest push. The compact Arrow null_count is now kept separate from the total Parquet null count: null_count is used by MaybeReplaceValidity, while total_null_count includes repeated-ancestor nulls and is used for WriteValuesSpaced and Parquet page/statistics metadata.
This keeps the Arrow validity semantics unchanged while fixing the Parquet null count for null/empty repeated ancestors. Could you please take another look when you get a chance? Thanks!

[wgtmac]

Thanks @AnuragRaut08 for the quick update! I think the code change looks good now. I have some comments for better test coverage.

[wgtmac]

Can we run this test for both data page V1 and V2 and then check both leaf columns? The old change fixed i32 but broke the string leaf, so the current checks would not catch that error.

[wgtmac]

Please add the same nested test case with a dictionary leaf. Run it for data page V1 and V2 by checking the statistics and reading the values back.

[wgtmac]

Can we rename total_null_count to parquet_null_count because it means number of nulls in the parquet leaf column? null_count is for the Arrow leaf array but renaming it would result in a larger change.