perf(merge_insert): stream missing partial-schema columns instead of collecting them in the join#7630
Conversation
Benchmark: peak memory before vs afterMemory experiment on a synthetic dataset shaped like the failing workload (wide binary payload column, partial-schema source). Setup:
The key structural difference: before, peak memory is ~3× the total payload size of the whole table regardless of how few rows are updated (the CollectLeft build side materializes the payload column for every target row, then join/write copies stack on top). After, peak memory tracks the number of matched rows (4× fewer updated rows → ~3× lower peak) and no longer grows with the payload width of the target table at all. Extrapolated to the motivating workload (hundreds of GB of payload columns, updating 30–40% of rows via a partial-schema source): before the fix the build side alone must hold the table's full payload in RAM, which OOMs any reasonable machine; after the fix the merge streams. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…collecting them in the join
A partial-schema merge_insert filled the dataset columns missing from
the source by copying them out of the target side of the join
(with_column(col("target.x"))). The target scan feeds the CollectLeft
build side of the hash join, so every payload column of every target
row was collected into memory before the source streamed through, and
the plan executes with an unbounded memory pool. On wide tables (rows
carrying large binary columns, datasets in the hundreds of GB) this
reliably OOM-kills the process even when the merge only touches a
small fraction of rows.
The join now carries only the join keys, _rowid/_rowaddr, the source
columns, and the action column. The write exec fetches the missing
columns per output batch with a take by row address
(TargetColumnFiller): updated rows read the matched target row's
values, inserted rows fill NULL. Peak memory is bounded by the batch
size, and only the rows actually rewritten are read, instead of the
full column for every target row.
ytyky
force-pushed
the
fix-merge-insert-partial-schema-oom
branch
from
264a161 to
1e49757
Compare
1 participant
A partial-schema merge_insert filled the dataset columns missing from the source by copying them out of the target side of the join (with_column(col("target.x"))). The target scan feeds the CollectLeft build side of the hash join, so every payload column of every target row was collected into memory before the source streamed through, and the plan executes with an unbounded memory pool. On wide tables (rows carrying large binary columns, datasets in the hundreds of GB) this reliably OOM-kills the process even when the merge only touches a small fraction of rows.
The join now carries only the join keys, _rowid/_rowaddr, the source columns, and the action column. The write exec fetches the missing columns per output batch with a take by row address (TargetColumnFiller): updated rows read the matched target row's values, inserted rows fill NULL. Peak memory is bounded by the batch size, and only the rows actually rewritten are read, instead of the full column for every target row.
Related to #1983 — fixes the partial-schema OOM; a configurable memory pool for the remaining paths is follow-up work.