Skip to content

Support custom simple CASE equality with ordered literal matching - #169

Merged
osipovartem merged 4 commits into
df55-physical-udf-instancefrom
feat/case-custom-comparator
Oct 27, 2026
Merged

osipovartem merged 4 commits into
df55-physical-udf-instancefrom
feat/case-custom-comparator

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

  • Add an optional session-scoped equality planner for simple CASE, so an engine can use its predicate semantics without changing the default DataFusion path.
  • Preserve the comparator through simplification, physical planning, expression rewrites, and physical-plan serialization.
  • Allow immutable scalar UDFs to opt into an ordered literal matcher. Dynamic, volatile, unsupported, and one-branch cases retain lazy evaluation.

Why

Snowflake VARIANT predicate equality between exact NUMBER and FLOAT is not transitive at precision boundaries. Replacing a simple CASE base with a hash/set key changes its answer. The hook keeps this behavior local to the engine that needs it; ordinary DataFusion CASE queries remain on their existing path.

Validation

  • Focused DataFusion tests cover nulls, first-match ordering, optimizer folding and volatility, expression rewrite, and physical-plan protobuf round-trip.
  • cargo +1.95.0 check -p datafusion-physical-expr and cargo +1.95.0 fmt --all -- --check passed locally.
  • Rustice integration passed 4,251/4,251 local SQL cases, including five new VARIANT CASE cases checked by focused live Snowflake queries.
  • Eight repeated local 1M-row probes (old vs new median, 1/8/64 numeric WHEN branches): column input 17/18/23 ms vs 21/17/22 ms; scalar input 5/5/9 ms vs 5/5/8 ms. The 1-branch path is not indexed; its 4 ms regression is narrow and required for correct predicate semantics. These timings are regression checks, not general speedup claims.
  • Independent read-only review approved correctness, performance tradeoff, API compatibility, and tests.

The consuming Rustice PR pins this fork commit and documents the focused Snowflake evidence. The full live corpus was not rerun for this change.

@osipovartem
osipovartem merged commit d749807 into df55-physical-udf-instance Oct 27, 2026
10 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant