Repository navigation
feat(parquet): add row number reader for selected row groups - #1006
Conversation
| // Metadata columns belong to top-level rows; nested fields need no positions. | ||
| ICEBERG_ASSIGN_OR_RAISE( | ||
| projected_array, | ||
| ProjectNestedArray(parquet_array, output_arrow_type, nested_type, | ||
| field_projection.children, metadata_context, pool)); | ||
| field_projection.children, metadata_context, | ||
| /*row_positions=*/nullptr, pool)); |
There was a problem hiding this comment.
not satisfied about these nullptr changes yet, however I don't find a more elegant way to passing row positions as it will touch all the ProjectedXXXMethods.
happy to take suggestions.
There was a problem hiding this comment.
A cleaner option would be to keep row positions only in a top-level struct projection helper, and use a separate ProjectNestedArray overload without the row_positions parameter for nested fields.
There was a problem hiding this comment.
good idea， let me address this
There was a problem hiding this comment.
Let me just merge this PR. You can create followup PR if you wish :)
There was a problem hiding this comment.
You can create followup PR if you wish :)
thanks. Let me create a follow-up pr then.
2 participants
What
Add a
RowNumberReader, modeled after parquet-rs, that emits absolute file row positions from the selected Parquet row groups.Use the resulting position array for
_posand inherited_row_idin record-batch projection. Preserve physical lineage values and cross-row-group batching. Restore consumed ranges if array construction fails so the batch can be retried.Why
Follow-up to the row-position tracking discussion in #956.
Updating
next_file_posby the emitted batch size assumes contiguous file rows. Row-group pruning can introduce gaps, so each selected row must retain its original position in the file.Testing
parquet_test,arrow_test, andavro_testpassed.git diff --checkpassed.Limitations and follow-up
The Iceberg-side
RowNumberReadercan account for row-group-level skipping only: it knows the selected row groups and their row counts. It cannot account for page-level skipping inside Arrow C++ because the skipped row ranges are not exposed alongside the output batches; batch lengths alone are insufficient to recover the original positions.Supporting page-level skipping requires integrating row-number generation into Arrow C++'s Parquet reader, where it can stay synchronized with page skipping and the actual rows emitted. This is the planned follow-up. Iceberg C++ can then consume that position array for
_posand inherited_row_id, replacing its localRowNumberReader. Keeping position generation separate from Iceberg metadata semantics prepares for that integration.