Upgrade to Arrow/Parquet 58 and DataFusion 54.1 - #137
Merged
Conversation
HarukiMoriarty
marked this pull request as draft
August 14, 2026 16:51
HarukiMoriarty
force-pushed
the
upgrade-arrow-59
branch
from
August 14, 2026 17:18
743c8d9 to
8ac4fe0
Compare
HarukiMoriarty
changed the title
Dependency changes: - arrow, parquet, arrow-csv: 56 -> 58 - datafusion: 50.2 -> 54.1 - object_store: 0.12.4 -> 0.13.2 - drop the direct arrow-array/arrow-schema deps in favour of the arrow crate's re-exports, so the Arrow version is declared in one place DataFusion 54.1 depends on Arrow 58, so targeting 58 rather than 59 keeps a single Arrow version in the dependency graph. API updates required by those releases: - parquet: get_column_writers (deprecated in 57) -> ArrowRowGroupWriterFactory. The factory borrows a live SerializedFileWriter, so the writer is now built up front and moved into the blocking writer task, and row groups are enumerated to supply the row group index the factory requires. - parquet: set_max_row_group_size -> set_max_row_group_row_count. - object_store: put/get moved to the ObjectStoreExt trait. Parquet output is unchanged. Geometry columns carry no Arrow extension metadata, so Arrow 58's native GEOMETRY logical type is not engaged and WKB is still written as plain BYTE_ARRAY. Signed-off-by: Nemo Yu <zyu379@wisc.edu> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HarukiMoriarty
force-pushed
the
upgrade-arrow-59
branch
from
August 14, 2026 17:20
8ac4fe0 to
adb3da4
Compare
HarukiMoriarty
marked this pull request as ready for review
August 14, 2026 17:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Sedona maintainers, thanks in advance for taking a look at this one.
What
Moves the workspace up to Arrow/Parquet 58, DataFusion 54.1, and object_store 0.13.2.
arrow,parquet,arrow-csvdatafusionobject_storearrow-array,arrow-schemaarrowre-exportsThe direct
arrow-array/arrow-schemadeps are dropped since they pinned the Arrow version in three separate places and thearrowcrate already re-exports both.API changes these releases required
parquet:get_column_writerswas deprecated in 57 in favour ofArrowRowGroupWriterFactory. The factory borrows a liveSerializedFileWriter, so the writer is now built before the row-group stream and moved into the blocking writer task, and row groups areenumerate()d for the index the factory needs. That accounts for most of the churn inparquet.rs— the data path itself is unchanged.parquet:set_max_row_group_size→set_max_row_group_row_count(Some(n)), same semantics.object_store:put/getmoved onto theObjectStoreExttrait.Output unchanged
Arrow 58 added native
GEOMETRY/GEOGRAPHYlogical types, so I checked this explicitly: geometry fields carry noARROW:extension:namemetadata, and a generatedzone.parquetstill writesz_boundaryas plainoptional binary.