feat: Add support for explode_outer function for arrays#22100
feat: Add support for explode_outer function for arrays#22100athlcode wants to merge 27 commits into
explode_outer function for arrays#22100Conversation
github-actions
Bot
added
sql
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @athlcode explode_outer is not like other builtin function so it has to be in the core.
Please add tests, more tests the better, including mix with nulls, nested combinations with explode and explode_outer and supported datatypes
…/explode_outer
…han7/datafusion into feat/explode_outer
thank you, added more test cases |
…/explode_outer
|
Thanks @athlcode my bad, I just realized I haven't clarified tests needs to be added to |
Thanks @comphead for the clarification. None of the test cases are currently reproducible via SLT because there's no SQL surface for NullHandling::PreserveAndExpandEmpty. To enable SLT coverage we need to add SQL recognition of explode_outer(col) (and probably explode(col) for symmetry) and map them to UNNEST with the right NullHandling variant. |
|
we would still need to have |
github-actions
Bot
added
logical-expr
Added slt tests at datafusion/sqllogictest/test_files/spark/generator/explode_outer.slt, exercises explode_outer via the Spark SQL dialect path |
github-actions
Bot
added
the
documentation
| pub format: FormatOptions, | ||
| /// Spark-compatibility options (functions under `datafusion/spark` and | ||
| /// Spark-specific planner behavior) | ||
| pub spark: SparkOptions, |
There was a problem hiding this comment.
I think we should not be adding spark into the core, it would inverse the dependencies
comphead
changed the title
|
we might also update the |
…/explode_outer
github-actions
Bot
removed
the
documentation
The mechanism has to live in core, but it isn't Spark-specific. NullHandling::PreserveAndExpandEmpty is just a new option on UnnestOptions in datafusion-common, where UnnestOptions already lives, and UnnestExec in datafusion-physical-plan is the only place the per-row logic can run. Expr::Unnest.outer is a new field on a core Expr variant, and downstream crates can't add fields to a pub enum variant in Rust, so the flag has to live where the variant is defined. Neither references Spark by name. They're the generic mechanism for "unnest that preserves empty arrays as NULL rows," which Hive EXPLODE OUTER, Snowflake FLATTEN(OUTER => true), and BigQuery UNNEST ... WITH OFFSET all need too. The Spark-flavored surface (the explode / explode_outer SQL aliases and the allow_multiple_generators config) is the only part that's truly dialect-specific. Happy to move those into datafusion-spark as a planner extension and a custom config registered via Extensions if you'd prefer the core diff to be purely the mechanism. |
github-actions
Bot
added
the
documentation
Thanks @athlcode sorry I really missed this comments, it makes sense to me, lets remove Spark from comments in the core, as this would be confusing, and other than that, the PR would be good to go |
|
@comphead thanks, I have removed spark from the comments, |
comphead
changed the title
explode_outer function for arrays
|
Thanks @athlcode. I checked some of engines for Lets move tests from |
…/explode_outer
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #22100 +/- ##
==========================================
- Coverage 80.76% 80.75% -0.01%
==========================================
Files 1089 1089
Lines 368797 369268 +471
Branches 368797 369268 +471
==========================================
+ Hits 297850 298210 +360
- Misses 53201 53309 +108
- Partials 17746 17749 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
thanks @comphead. moved tests into |
4 participants
Which issue does this PR close?
explode_outerfunction for arrays #19053.Rationale for this change
DataFusion's 'unnest' had no way to express Spark 'explode_outer' semantics, empty input lists were silently dropped, even with 'preserve_nulls' = true.
What changes are included in this PR?
Adds a third unnest behavior that produces a
NULLrow for empty input lists, by replacingUnnestOptions.preserve_nulls: boolwith aNullHandling { Drop, Preserve, PreserveAndExpandEmpty }enum. Thewith_preserve_nulls(bool)builder is kept as a backward-compat shim.Are these changes tested?
Yes, new unit test for the empty-list case, extended longest-length and DataFrame
unnest_column_nullstests, and existing proto round-trip coverage.Are there any user-facing changes?
The
preserve_nullsfield onUnnestOptionsis renamed tonull_handling. Thewith_preserve_nulls(bool)builder is preserved, so most callers are unaffected. Add theapi changelabel for the field rename.