Skip to content

perf: optimize date_trunc (>2x faster)#4915

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/date_trunc-datafusion-comet-20260714-032832
Open

perf: optimize date_trunc (>2x faster)#4915
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/date_trunc-datafusion-comet-20260714-032832

Conversation

@andygrove

@andygrove andygrove commented Jul 14, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expression.

What changes are included in this PR?

Resolve the date_trunc format keyword without allocating an uppercased String per row (ASCII case-insensitive match plus a last-format memo), removing a heap allocation from every row of the array-format path.

How are these changes tested?

Existing tests.

Benchmark (criterion):

  • month: 60.556% faster (base 143134ns -> cand 56457ns)
  • week: 64.05% faster (base 138437ns -> cand 49768ns)
  • mixed: 54.003% faster (base 157749ns -> cand 72560ns)
  • year: 67.326% faster (base 134453ns -> cand 43931ns)
  • quarter: 53.059% faster (base 159586ns -> cand 74911ns)

Full criterion output:

date_trunc_array_fmt/year
                        time:   [43.888 µs 43.941 µs 44.010 µs]
                        change: [−67.368% −67.326% −67.273%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  4 (4.00%) high mild
  7 (7.00%) high severe
date_trunc_array_fmt/quarter
                        time:   [74.951 µs 74.980 µs 75.019 µs]
                        change: [−53.137% −53.059% −52.998%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe
date_trunc_array_fmt/month
                        time:   [56.438 µs 56.467 µs 56.513 µs]
                        change: [−60.672% −60.556% −60.480%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
date_trunc_array_fmt/week
                        time:   [49.729 µs 49.783 µs 49.882 µs]
                        change: [−64.092% −64.050% −64.005%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe
date_trunc_array_fmt/mixed
                        time:   [72.540 µs 72.573 µs 72.631 µs]
                        change: [−54.045% −54.003% −53.946%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe

@andygrove andygrove changed the title perf: optimize date_trunc in datafusion-comet-spark-expr perf: optimize date_trunc (>2x faster) Jul 14, 2026
@andygrove andygrove marked this pull request as ready for review July 14, 2026 14:51

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, thanks @andygrove!

Comment on lines +345 to +360
fn date_trunc_fn_for_format(format: &str) -> Result<DateTruncFn, SparkError> {
let key: Cow<str> = if format.is_ascii() {
Cow::Borrowed(format)
} else {
Cow::Owned(format.to_uppercase())
};
DATE_TRUNC_FORMATS
.iter()
.find(|(name, _)| name.eq_ignore_ascii_case(&key))
.map(|(_, trunc_fn)| *trunc_fn)
.ok_or_else(|| {
SparkError::Internal(format!(
"Unsupported format: {format:?} for function 'date_trunc'"
))
})
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let key: Cow<str> = if format.is_ascii() {
    Cow::Borrowed(format)
} else {
    Cow::Owned(format.to_uppercase())
};
DATE_TRUNC_FORMATS
    .iter()
    .find(|(name, _)| name.eq_ignore_ascii_case(&key))

Every entry in DATE_TRUNC_FORMATS is ASCII. eq_ignore_ascii_case only folds ASCII bytes, so a non-ASCII key can never equal an ASCII key regardless of whether it was to_uppercased first. The else branch allocates a String and then cannot change the outcome: a non-ASCII format always falls through to the error. The comment claiming non-ASCII "still needs full Unicode case folding to fold the same way Spark does" is incorrect for this table, because Spark also only matches ASCII literals after folding, so a non-ASCII input is invalid in Spark too.

Suggested change: drop the Cow and the is_ascii() split entirely and match on the borrowed &str:

fn date_trunc_fn_for_format(format: &str) -> Result<DateTruncFn, SparkError> {
    DATE_TRUNC_FORMATS
        .iter()
        .find(|(name, _)| name.eq_ignore_ascii_case(format))
        .map(|(_, trunc_fn)| *trunc_fn)
        .ok_or_else(|| {
            SparkError::Internal(format!(
                "Unsupported format: {format:?} for function 'date_trunc'"
            ))
        })
}

This removes the use std::borrow::Cow; import, removes an allocation on the non-ASCII path, and is exactly as Spark-compatible.

fn ntz_trunc_fn_for_format(
format: &str,
) -> Result<fn(NaiveDateTime) -> Option<NaiveDateTime>, SparkError> {
match format.to_uppercase().as_str() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

native/spark-expr/src/kernels/temporal.rs:591 (ntz_trunc_fn_for_format uses format.to_uppercase().as_str()) and 832 (the tz array-format helper still calls $formats.value(index).to_uppercase().as_str() per row). These are the timestamp_trunc siblings of the code this PR just optimized, and they carry the identical per-row String allocation the PR is removing. Leaving them behind means the "resolve once into a table plus memo" pattern this PR introduces is applied to one of three structurally identical loops, so the improvement is narrower than the general problem.

Suggested change: either extend this PR to build a TIMESTAMP_TRUNC_FORMATS table plus the same last-format memo for the NTZ and tz array paths, or file a follow-up issue and reference it in this PR so the remaining allocations are tracked. I would fold at least the NTZ path in here since it is a direct analog of DATE_TRUNC_FORMATS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants