perf: spill every shuffle partition of a task into one file - #5916

Open

peterxcli wants to merge 8 commits into

apache:mainfrom

peterxcli:perf/shuffle-single-spill-file-per-round

Commits on Sep 9, 2026

perf: return native shuffle partition offsets over JNI instead of an index file

peterxcli
and
claude
committed
1cbe9c2
review: address feedback on naming and layering

peterxcli
and
claude
committed
6e1fb10
fix: size shuffle partition lengths from the offsets the writer returned

peterxcli
and
claude
committed
f83eca4
review: drop the reserved declarations for the retired index path

peterxcli
and
claude
committed
313fdc2

Commits on Sep 10, 2026

review: trim comments and close the gap left by the retired field number

peterxcli
and
claude
committed
7e4c5e4

Commits on Sep 13, 2026

perf: spill every shuffle partition of a task into one file

peterxcli
and
claude
committed
377f99b
perf: merge short spill ranges with one positional read

peterxcli
and
claude
committed
fc9d64b
perf: buffer spill writes across partitions

peterxcli
and
claude
committed
a524afb