Repository navigation
Conversation
Appending an Arrow table with string_view or binary_view columns, which pa.table() produces from a Polars DataFrame, worked for unpartitioned tables but failed for partitioned ones: PyArrow has no group by, filter or take kernels for view types, and pyiceberg-core's bucket and truncate transforms reject them. Cast view types, including nested ones, to their large counterparts before splitting the table into partitions.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4120
Rationale for this change
string_viewandbinary_viewcolumns are accepted on write, and appending them to an unpartitioned table works. Appending them to a partitioned table fails in_determine_partitions, because PyArrow has no group by, filter or take kernels for view types, and the pyiceberg-core bucket and truncate transforms do not accept them either.pa.table(polars_df)produces exactly this shape, since Polars exports strings and binaries as view types over the PyCapsule interface.Depending on the spec, the append fails with
Keys of type string_view,Function 'array_filter' has no kernel matching input types (string_view, bool), orUnsupported data type for bucket transform: Utf8View.This change casts view types, including ones nested in structs, lists and maps, to
large_string/large_binarybefore the table is split into partitions. Tables without view types are not cast. The data files are written the same as forlarge_string/large_binaryinput.One remaining gap is outside this repo: truncate on a binary column already fails for
large_binaryinput in pyiceberg-core (apache/iceberg-rust#3119), and nowbinary_viewinput hits that same error.Are these changes tested?
Yes:
test_determine_partitions_with_view_types: identity, bucket and truncate partitions on astring_viewcolumn, withbinary_viewand nested view columns alongside.test_append_view_types_to_partitioned_table: end-to-end append to a partitioned in-memory table, then read back.Both fail on
mainand pass with this change, on pyarrow 25.0.1 and on the minimum supported pyarrow 18.1.0.Are there any user-facing changes?
Yes. Appending, overwriting, and dynamically overwriting partitioned tables with Arrow view type columns now works where it used to raise.