Skip to content

feat: add support for _partition metadata column#2668

Open
parthchandra wants to merge 2 commits into
apache:mainfrom
parthchandra:metadata-columns
Open

feat: add support for _partition metadata column#2668
parthchandra wants to merge 2 commits into
apache:mainfrom
parthchandra:metadata-columns

Conversation

@parthchandra

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Implements the _partition metadata column for table scans. This is a struct column whose type is the union of all partition fields across all partition specs (handling partition evolution). Each row gets the
partition values for its data file.

  • Adds compute_unified_partition_type() to compute the union of partition fields across all specs (equivalent to Java's Partitioning.partitionType())
  • Adds PartitionColumnConstant and build_partition_column_constant() for pre-computing the struct values per file
  • Adds ColumnSource::AddStructConstant variant to RecordBatchTransformer for materializing struct columns
  • Threads the unified partition type through scan planning and populates the constant in FileScanTask
  • Pipeline detects RESERVED_FIELD_ID_PARTITION in projected fields and injects the struct constant

Are these changes tested?

Because we do not have write support yet, I made the corresponding change to comet and then tested by adding tests in Comet which uses iceberg-java to write files and then iceberg-rust to read them back.
https://github.com/parthchandra/datafusion-comet/blob/iceberg-metadata-columns/spark/src/test/resources/sql-tests/iceberg/metadata_column_partition.sql

@parthchandra parthchandra marked this pull request as draft June 18, 2026 00:34
@parthchandra

Copy link
Copy Markdown
Author

@advancedxy fyi

@parthchandra parthchandra marked this pull request as ready for review June 18, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant