[EPIC] Optimize native scalar expressions used in TPC-DS

[andygrove]

Background

This is an umbrella issue for performance-tuning the native scalar expression implementations in datafusion-comet-spark-expr (native/spark-expr/), prioritizing the expressions that carry the most per-row cost in TPC-DS.

The methodology is documented in the contributor guide (Optimizing Scalar Expressions): benchmark-first with criterion, keep output bit-identical to main (existing unit tests are the correctness gate), cover the input shapes where fast paths regress (no-null / sparse-null / dense-null, short / long), and do not submit a change that meaningfully regresses any shape. Tuned expressions are recorded as Performance (tuned ...) entries in the per-expression audit pages.

Scope

TPC-DS per-row scalar work is dominated by decimal arithmetic, casts, and a handful of string/conditional expressions. Aggregates (sum/avg/count/stddev_samp) are the biggest overall cost but are out of scope here (aggregate, not scalar).

Candidate expressions (to be confirmed and benchmarked individually):

• CheckOverflow (math_funcs/internal/checkoverflow.rs) - wraps the result of every decimal + - * /, sum, and avg. The common non-ANSI path always allocates a new Decimal128 array via null_if_overflow_precision, even when nothing overflows. Opportunity: a no-overflow zero-copy fast path. No benchmark exists yet.
• Decimal rescale (math_funcs/internal/decimal_rescale_check.rs) - used by decimal-to-decimal casts; audit for per-row allocation.
• Further casts used heavily in TPC-DS (decimal casts), triaged against work already merged or in flight.

Already optimized / in flight

Many scalar expressions have already been tuned in a prior campaign (casts, substring, date_trunc, floor/ceil, to_json, parse_url, size, unhex, and others). New work under this epic should check the per-expression audit pages first to avoid re-treading tuned expressions.

How to contribute

Each expression is a small, self-contained PR:

1. Add a criterion benchmark under native/spark-expr/benches/ covering the shapes above.
2. Capture a baseline on main, apply the optimization, confirm bit-identical output, re-measure.
3. Only submit a strict improvement (no meaningful regression on any shape).
4. Record a Performance (tuned ...) entry in the relevant audit page.