Skip to content

perf: optimize arrays_zip_inner in datafusion-comet-spark-expr#4922

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/arrays_zip_inner-datafusion-comet-20260714-105432
Draft

perf: optimize arrays_zip_inner in datafusion-comet-spark-expr#4922
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/arrays_zip_inner-datafusion-comet-20260714-105432

Conversation

@andygrove

Copy link
Copy Markdown
Member

This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.

Which issue does this PR close?

N/A — autonomous exploratory PR.

Rationale for this change

Borrow list offsets/null buffers instead of materializing per-column Vecs, size the MutableArrayData builders exactly, and coalesce unpadded rows into bulk extends so equal-length rows cost one copy per column instead of one call per row.

What changes are included in this PR?

Borrow list offsets/null buffers instead of materializing per-column Vecs, size the MutableArrayData builders exactly, and coalesce unpadded rows into bulk extends so equal-length rows cost one copy per column instead of one call per row.

How are these changes tested?

Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).

Benchmark (criterion):

  • arrays_zip_uniform: 20.538% faster (base 362344ns -> cand 287927ns)
  • arrays_zip_ragged: 11.864% faster (base 421085ns -> cand 371129ns)

Full criterion output:

arrays_zip/uniform      time:   [287.27 µs 287.91 µs 288.91 µs]
                        change: [−20.756% −20.538% −20.270%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe

arrays_zip/ragged       time:   [370.88 µs 371.03 µs 371.19 µs]
                        change: [−12.181% −11.864% −11.603%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low severe
  1 (1.00%) high mild
  4 (4.00%) high severe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant