Parquet: Add reader and writer without Hadoop dependency#16588
Conversation
efc4cf5 to
1342666
Compare
|
Hey, just got pointed to this PR by @nssalian (thanks!). You might be interested in the Hardwood project, whose explicit goal is to implement a Parquet reader (and soon writer) without any mandatory dependencies. We're on the homestretch towards 1.0 Final, with quite comprehensive support for most Parquet features at this point. Would love to hear from folks here whether Hardwood could be interesting for the purposes here (we're already talking to Flink folks about integrating Hardwood for its Parquet usage, too). |
|
Use of Hardwood project in Iceberg warrants a discussion in the dev list. @ebyhr / @gunnarmorling would you like to start a thread? |
80c4b3f to
5bca7ce
Compare
89b8239 to
75d8ba7
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Summary
Adds native Parquet file reader and writer implementations that use Iceberg's
InputFile/OutputFileabstractions instead of Hadoop'sFileSystem, enabling usage withoutparquet-hadoopruntime dependency.This allows systems like Trino (which prohibits Hadoop dependencies) to use Iceberg's Parquet module by setting
iceberg.parquet-client=NATIVEJVM property.Hardwood isn't ready for replacement, as far as I've confirmed. From Trino's perspective (one of the major downstream projects), reducing the dependency is preferable to replacing parquet-hadoop with another library. Trino won't use Hardwood anyway.
🤖 Generated with Claude Code