chore: Update to arrow/parquet 59.0.0#22744
Draft
alamb wants to merge 2 commits into
Draft
Conversation
12 tasks
github-actions
Bot
added
physical-expr
alamb
commented
Jun 3, 2026
| .bloom_filter_properties(&ColumnPath::from("")) | ||
| .expect("expected bloom properties!") | ||
| .fpp, | ||
| .fpp(), |
Contributor
Author
There was a problem hiding this comment.
These fields are made private in
| }; | ||
| if let Some(bloom_filter_ndv) = bloom_filter_ndv { | ||
| builder = builder.set_bloom_filter_ndv(*bloom_filter_ndv); | ||
| builder = builder.set_bloom_filter_max_ndv(*bloom_filter_ndv); |
Contributor
Author
There was a problem hiding this comment.
- due to feat(parquet): add BloomFilterPropertiesBuilder arrow-rs#9877 which deprecated set_bloom_filtr_ndv
| ndv: DEFAULT_BLOOM_FILTER_NDV | ||
| }), | ||
| Some( | ||
| &BloomFilterProperties::builder() |
Contributor
Author
There was a problem hiding this comment.
Properties are now built with builder:
| &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], | ||
| &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 5, 6], | ||
| ]))], | ||
| vec![Arc::new( |
Contributor
Author
There was a problem hiding this comment.
from impl was removed because it could panic
| scale: 2, | ||
| precision: 9, | ||
| }) | ||
| .with_logical_type(LogicalType::decimal(2, 9)) |
Contributor
Author
There was a problem hiding this comment.
have to use new helpers added in
| ] | ||
|
|
||
| [[package]] | ||
| name = "integer-encoding" |
Contributor
Author
There was a problem hiding this comment.
yay for removing older deps
| "cfg-if", | ||
| ] | ||
|
|
||
| [[package]] |
Contributor
Author
There was a problem hiding this comment.
no more thrift! We now use the entirely new thrift encoder and not the thrift generator
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
alamb
force-pushed
the
alamb/update_arrow_59
branch
from
55c7eb3 to
e616241
Compare
alamb
commented
Jun 4, 2026
| | alltypes_plain.parquet | 1851 | 8882 | 2 | page_index=false | | ||
| | alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true | | ||
| | lz4_raw_compressed_larger.parquet | 380836 | 1339 | 2 | page_index=false | | ||
| | alltypes_plain.parquet | 1851 | 8794 | 2 | page_index=false | |
Contributor
Author
There was a problem hiding this comment.
I think this changed (smaller in memory size) due to the representation change of CompressionCodec in this pr
It changes from Compression which also carries the compression level: ZSTD(ZstdLevel), GZIP(GzipLevel), BROTLI(BrotliLevel) — and ZstdLevel(i32) / GzipLevel(u32) / BrotliLevel(u32) and are 4-byte wrappers. So Compression = 4-byte discriminant + 4-byte level = 8 bytes.
To a fieldless enum CompressionCodec -- 1 byte
alamb
commented
Jun 4, 2026
| Total Requests: 2 | ||
| - GET (opts) path=parquet_table.parquet head=true | ||
| - GET (ranges) path=parquet_table.parquet ranges=1064-1481,1481-1594,1594-2011,2011-2124 | ||
| - GET (ranges) path=parquet_table.parquet ranges=1064-1594,1594-2124 |
Contributor
Author
There was a problem hiding this comment.
this seems like an improvement -- contiguous ranges are coalesced into fewer ranges. I tracked it down to this PR from @HippoBaro
alamb
force-pushed
the
alamb/update_arrow_59
branch
from
e616241 to
4d274ef
Compare
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/update_arrow_59 (4d274ef) to 18e7c8e (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/update_arrow_59 (4d274ef) to 18e7c8e (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/update_arrow_59 (4d274ef) to 18e7c8e (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark clickbench_partitioned |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/update_arrow_59 (4d274ef) to 18e7c8e (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
Performance looks about the same |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
59.0.0(May 2026) arrow-rs#9110WIP: I am using this PR to test the arrow release
Rationale for this change
Update to latest version of arrow/parquet
What changes are included in this PR?
Are these changes tested?
By CI
Are there any user-facing changes?
New dependency