Skip to content

[python] Add opt-in late materialization using Parquet OffsetIndex - #9962

Draft
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read
Draft

XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What

  • Add opt-in late materialization for row-ID-based Parquet reads. Once Paimon identifies the requested rows, use OffsetIndex to fetch only their data pages and let PyArrow decode them.
  • Support flat columns and standard two-binary-field VARIANT (value, metadata). A single row is a valid one-row range. Disjoint ranges, full-row-group reads, missing indexes, unsupported schemas, and selections that would read too many bytes use the ordinary reader.
  • Reuse parquet.filter.columnindex.enabled, disabled by default. This does not add ColumnIndex predicate filtering or change the table format.

I/O trade-off

The optimization trades more small range requests for fewer transferred bytes. Read-only single-row tests returned identical results and showed lower file-read and point-lookup latency, but issued more GET requests. The extra requests read page indexes and may outweigh byte savings on high-latency storage or less selective reads. This is why the path remains opt-in and falls back when it is unlikely to help. These measurements are not a full application or training-sample benchmark.

Validation

  • Related read tests: 108 passed, 18 subtests passed (PyArrow 18); page-index tests: 61 passed (PyArrow 19).
  • Python lint, Python 3.6 syntax check, and git diff --check passed.

This is a draft because the remote I/O behavior and fallback thresholds deserve community review before enabling it by default.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add opt-in Parquet OffsetIndex reads for selected rows [python] Add opt-in late materialization using Parquet OffsetIndex Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant