Skip to content

feat(datafusion): make PartitionExpr reconstructible - #3001

Open
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:feat/serializable-partition-expr
Open

feat(datafusion): make PartitionExpr reconstructible#3001
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:feat/serializable-partition-expr

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

PartitionExpr wraps a PartitionValueCalculator, which is a live object with
no serializable form. Once the expression is built, nothing on it says which
partition spec and schema produced it, so a caller holding the expression
cannot rebuild an equivalent one.

Retain both inputs on the expression and expose them:

  • try_new(partition_spec, table_schema) replaces new(calculator, spec) and
    builds the calculator itself, so the two inputs and the calculator cannot
    drift apart.
  • partition_spec() and table_schema() read them back. Both are self-contained
    iceberg spec types, so a distributed engine can serialize them and rebuild
    the expression on a worker with try_new.

project_with_partition is unchanged from the outside; it now hands the two
inputs to try_new instead of constructing the calculator first.

Are these changes tested?

  • test_partition_expr_rebuilds_from_its_retained_parts: reads partition_spec() and table_schema() off a PartitionExpr, builds a second one from them via try_new, and asserts both evaluate a RecordBatch to the same array.

The existing project tests cover the refactor itself, as it now constructs through try_new and still pass unchanged.

AI Disclosure

  • Helped write the test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PartitionExpr cannot be reconstructed from its public API

1 participant