WIP: Arrow: vectorized read of lists - #17723
Conversation
# Conflicts: # arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Dictionary encoding is not supported yet Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Also rename VectorizedListBuilder to ListVectorBuilder and slim the interface Co-Authored-By: Claude <noreply@anthropic.com>
| * repetition-level histogram); used to size the nullability holder and repetition-level | ||
| * vector | ||
| */ | ||
| void prepareBatch(int numRowsToRead, int estimatedSize) { |
There was a problem hiding this comment.
What happens if the estimated size is too small?
There was a problem hiding this comment.
The Arrow vectors will be resized automatically (with "safe" method calls), but the NullabilityHolder will not so it will lead to an index out of bound exception. It happens when the list is nested repetition level histogram is not present in the parquet metadata.
I don't know what is the better approach in this case to a) over allocate by value count (it can be far from the actual number of lists on the level or b) scale responsively.
|
|
||
| if (listRepetitionLevels != null) { | ||
| listRepetitionLevels.close(); | ||
| } |
There was a problem hiding this comment.
in a new file, please leave an empty line after the block close. See: https://iceberg.apache.org/contribute/#block-spacing
There was a problem hiding this comment.
Oh I thought Spotless would fix that, but I'll add newlines where missing.
There was a problem hiding this comment.
Where do we handle this change?
There was a problem hiding this comment.
Which change? I don't see any lines referenced.
2 participants
This is WIP. I want to improve on the design of NullabilityHolder and the reading of repetition levels.