feat: add Spark date_from_unix_date function to datafusion-spark#23852
feat: add Spark date_from_unix_date function to datafusion-spark#23852andygrove wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23852 +/- ##
==========================================
- Coverage 80.74% 80.71% -0.03%
==========================================
Files 1089 1091 +2
Lines 369207 369498 +291
Branches 369207 369498 +291
==========================================
+ Hits 298102 298231 +129
- Misses 53347 53485 +138
- Partials 17758 17782 +24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…invocation consumers work
|
Leaving this as draft for now. I am assuming that the SLT tests will be using the |
…_unix_date slt The function lowers to an Int32 -> Date32 cast in simplify(), so the default sqllogictest path never reaches invoke_with_args. Add a second section that disables the logical optimizer via datafusion.optimizer.max_passes = 0 so the function survives into the physical plan, with EXPLAIN assertions in each section to show which path is being exercised.
I updated the slt test to test both paths |
| ########## | ||
|
|
||
| statement ok | ||
| set datafusion.optimizer.max_passes = 0; |
There was a problem hiding this comment.
we prob need to document that we don't want simplify to be called.
There was a problem hiding this comment.
Maybe I should just remove the simplify path from this PR, but my concern is that someone will later implement it and remove invoke_with_args and then Comet will no longer be able to leverage this work
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { |
There was a problem hiding this comment.
the tests might be not needed as it should have already been covere by slt, however nice test would be to test simplify output and invoke_with_args_output are the same
Which issue does this PR close?
Part of the datafusion-spark epic: #15914
Rationale for this change
date_from_unix_dateis a Spark built-in that creates a date from the number of days since the Unix epoch (1970-01-01). It is the inverse of the existingunix_datefunction and is not yet available in thedatafusion-sparkcrate.What changes are included in this PR?
SparkDateFromUnixDateunderdatafusion/spark/src/function/datetime/, registered in the datetime module. The implementation lowers to a singleInt32 -> Date32cast insimplify(), mirroring the existingunix_dateidiom, since both types represent days since epoch.test_files/spark/datetime/date_from_unix_date.sltfor scalar, array, negative, and NULL inputs.Are these changes tested?
Yes, via sqllogictest with scalar and array inputs (including negative and NULL cases).