Skip to content

perf: optimize spark_regexp_extract_all in datafusion-comet-spark-expr#4923

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_regexp_extract_all-datafusion-comet-20260714-111804
Draft

perf: optimize spark_regexp_extract_all in datafusion-comet-spark-expr#4923
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_regexp_extract_all-datafusion-comet-20260714-111804

Conversation

@andygrove

Copy link
Copy Markdown
Member

This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.

Which issue does this PR close?

N/A — autonomous exploratory PR.

Rationale for this change

Reuse a single CaptureLocations across all matches and rows instead of letting captures_iter heap-allocate capture slots per match, and use find_iter when only the whole match (idx=0) is needed.

What changes are included in this PR?

Reuse a single CaptureLocations across all matches and rows instead of letting captures_iter heap-allocate capture slots per match, and use find_iter when only the whole match (idx=0) is needed.

How are these changes tested?

Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).

Benchmark (criterion):

  • capture_group: 24.244% faster (base 1052449ns -> cand 797294ns)
  • whole_match: 61.571% faster (base 1046798ns -> cand 402278ns)

Full criterion output:

regexp_extract_all/capture_group
                        time:   [797.09 µs 797.74 µs 798.40 µs]
                        change: [−24.419% −24.244% −24.094%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
regexp_extract_all/whole_match
                        time:   [402.08 µs 402.21 µs 402.42 µs]
                        change: [−61.621% −61.571% −61.519%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant