Support content-defined chunking (CDC) in the Parquet writer #3817

Describe the enhancement requested

Add content-defined chunking (CDC) to the Parquet writer. It is already available in Parquet C++ (apache/arrow#45750, PyArrow >= 21) and parquet-rs (apache/arrow-rs#9450). With CDC, data page boundaries come from a rolling hash of the column values rather than fixed sizes. Pages then stay stable when rows are inserted or appended, so content-addressable storage (e.g. Hugging Face Xet) can deduplicate them. The output is a regular Parquet file, with no format change.

It would help to use the same parameters as the other implementations:

• min_chunk_size (default 256 KiB)
• max_chunk_size (default 1 MiB)
• norm_level (default 0)

Motivation: Iceberg is adding write.parquet.content-defined-chunking.* table properties (apache/iceberg-rust#2375, apache/iceberg-python#3889). Iceberg Java cannot honor them until parquet-java supports CDC.

Component(s)

Core