Skip to content

Parquet: Add reader and writer without Hadoop dependency#16588

Draft
ebyhr wants to merge 2 commits into
apache:mainfrom
ebyhr:ebi/hadoop
Draft

Parquet: Add reader and writer without Hadoop dependency#16588
ebyhr wants to merge 2 commits into
apache:mainfrom
ebyhr:ebi/hadoop

Conversation

@ebyhr

@ebyhr ebyhr commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

Adds native Parquet file reader and writer implementations that use Iceberg's InputFile/OutputFile abstractions instead of Hadoop's FileSystem, enabling usage without parquet-hadoop runtime dependency.

This allows systems like Trino (which prohibits Hadoop dependencies) to use Iceberg's Parquet module by setting iceberg.parquet-client=NATIVE JVM property.

Hardwood isn't ready for replacement, as far as I've confirmed. From Trino's perspective (one of the major downstream projects), reducing the dependency is preferable to replacing parquet-hadoop with another library. Trino won't use Hardwood anyway.

🤖 Generated with Claude Code

@ebyhr ebyhr changed the title Add parquet reader / writer without Hadoop dependency Parquet: Add parquet reader / writer without Hadoop dependency May 28, 2026
@ebyhr ebyhr changed the title Parquet: Add parquet reader / writer without Hadoop dependency Parquet: Add reader and writer without Hadoop dependency May 28, 2026
@ebyhr ebyhr force-pushed the ebi/hadoop branch 3 times, most recently from efc4cf5 to 1342666 Compare May 28, 2026 13:20
@gunnarmorling

Copy link
Copy Markdown

Hey, just got pointed to this PR by @nssalian (thanks!). You might be interested in the Hardwood project, whose explicit goal is to implement a Parquet reader (and soon writer) without any mandatory dependencies. We're on the homestretch towards 1.0 Final, with quite comprehensive support for most Parquet features at this point. Would love to hear from folks here whether Hardwood could be interesting for the purposes here (we're already talking to Flink folks about integrating Hardwood for its Parquet usage, too).

@anuragmantri

Copy link
Copy Markdown
Collaborator

Use of Hardwood project in Iceberg warrants a discussion in the dev list. @ebyhr / @gunnarmorling would you like to start a thread?

@ebyhr ebyhr force-pushed the ebi/hadoop branch 3 times, most recently from 80c4b3f to 5bca7ce Compare June 4, 2026 12:45
@ebyhr ebyhr force-pushed the ebi/hadoop branch 4 times, most recently from 89b8239 to 75d8ba7 Compare June 7, 2026 07:08
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decouple Iceberg from Hadoop codebase

3 participants