Core: Simplify list handling in manifest read and write paths - #17581
Conversation
Use Lists.newArrayListWithCapacity in ManifestsTable.partitionSummariesToRows and ManifestFiles to avoid intermediate ArrayList resizes when the target size is already known.
ebyhr
left a comment
There was a problem hiding this comment.
Looks good to me, but there are still same patterns in other places.
|
Limiting the scope is perfectly acceptable, but I recommend providing an explanation of the intention in the PR description. This way, it becomes clear whether the limitation was intentional or not. |
|
cc @szehon-ho PTAL |
There was a problem hiding this comment.
Both changes are correct and behavior-preserving, and newArrayListWithCapacity is the right helper here since the sizes are exact rather than estimates.
For ManifestFiles.writeParallel there's a better version of the change: the list being pre-sized doesn't need to exist at all, and removing it would also bring the method in line with every other parallel fan-out in the codebase. Details inline.
Two small, behavior-preserving cleanups to how manifest code builds result lists:
ManifestsTable#partitionSummariesToRows— therowslist is filled with exactly one row per entry insummaries, a size known up front, so it is allocated withLists.newArrayListWithCapacity(summaries.size())instead of the default-capacitynewArrayList(), avoiding intermediate resizes.ManifestFiles#writeParallel— replaced the intermediateList<Pair<Integer, List<F>>>(built only to carry each group's index) withTasks.range(groups.size())indexing directly intogroups. This drops the list, its build loop, onePairallocation per group, the now-unusedPairimport, and the comment that existed only to explain the pairing, and it makes the input-order guarantee visible in therunbody. It also brings the method in line with theTasks.range(n)idiom used by every other parallel fan-out in the codebase (ManifestMergeManager,ManifestFilterManager,SnapshotProducer,DVUtil, and the Spark/Flink planners);writeParallelwas the sole outlier, having diverged when it was extracted fromSnapshotProducerin Core: extract parallel manifest write logic out of SnapshotProducer #16730. TheAtomicReferenceArrayresults assembly is unchanged.The
ManifestsTablepre-size is intentionally limited to this one spot rather than a codebase-wide sweep of every under-sized list, to keep the change small and easy to review.