Skip to content

fix(substrait): consume chained window functions whose default names collide - #25181

Open
ShayanGho wants to merge 2 commits into
apache:mainfrom
ShayanGho:substrait-chained-window-names
Open

fix(substrait): consume chained window functions whose default names collide#25181
ShayanGho wants to merge 2 commits into
apache:mainfrom
ShayanGho:substrait-chained-window-names

Conversation

@ShayanGho

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A logical plan that contains two window aggregates with the same default schema name, distinguished only by aliases, executes fine in DataFusion but cannot be consumed back after a Substrait round trip. The consumer fails with:

SchemaError(DuplicateUnqualifiedField { name: "avg(data.b) PARTITION BY [data.a] ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW" })

DataFusion's Substrait producer uses positional projections and does not preserve intermediate window aliases: a ProjectRel carries expressions and an output mapping. The aliases that kept the two columns apart in DataFusion (AS avg1, AS avg2) are dropped on the wire. When the consumer rebuilds the outer projection, the inherited window column and the new window expression both come back under the same default name, and Window::try_new rejects the plan.

The SQL from the issue reproduces it, and so does a minimal plan built with LogicalPlanBuilder that stacks two row_number() windows.

What changes are included in this PR?

Two small changes, one per crate:

  • datafusion-expr: group_window_expr_by_sort_keys (used by LogicalPlanBuilder::window_plan) now looks through a single Expr::Alias to derive the sort key and keeps the aliased expression in its group, so window_plan can build a Window node whose output field carries an alias. LogicalPlanBuilder::window, filter pushdown, and the physical planner already handle aliased window expressions, so this makes window_plan consistent with them. Nested aliases are still rejected, matching what filter pushdown tolerates.
  • datafusion-substrait consumer: from_project_rel reserves the input schema's field names in a NameTracker (new reserve_schema helper), aliases any new window expression whose default name would collide, and rewrites the projection to reference the alias while preserving the projection's own output names via NamePreserver. Plans without a collision are built exactly as before.

What is the testing strategy for this PR?

  • datafusion-expr: a unit test (test_group_window_expr_by_sort_keys_aliased_window_expr) asserting that an aliased window expression is grouped by the inner function's sort key and kept aliased. It failed on main with Impossibly got non-window expr.
  • datafusion-substrait: two round-trip tests in roundtrip_logical_plan.rs, stacked_windows_with_same_default_name_via_builder (minimal LogicalPlanBuilder form) and chained_windows_with_same_default_name (the SQL from the issue). Both failed before this change with the DuplicateUnqualifiedField error above. They assert Arrow schema equality (names, types, nullability) and execute the consumed plan. They do not compare plan text, because the consumer must synthesize an alias for the colliding window, so the text legitimately differs. They compare Arrow schemas rather than full DFSchemas because qualifiers and inferred functional dependencies can change during reconstruction: DataFusion's Substrait producer does not preserve subquery aliases, and the rebuilt Window nodes infer dependencies the original plan did not record.
  • The full datafusion-substrait test run passes unchanged (273 passed, 6 ignored across the lib, integration, and doc tests), confirming the non-colliding path is unaffected.

This change was developed with AI assistance. I have reviewed and understand every line and stand behind it.

Are there any user-facing changes?

No API changes. Plans consumed from Substrait that previously failed with DuplicateUnqualifiedField now consume successfully. Synthesized aliases of the form <default name>__temp__N may appear in the decoded logical plan; final output column names are preserved. Plans without such a collision are unchanged.

ShayanGho and others added 2 commits September 10, 2026 22:24
group_window_expr_by_sort_keys rejected any expression that was not a bare
WindowFunction, so LogicalPlanBuilder::window_plan could not build a Window
node whose output field carries an alias, even though LogicalPlanBuilder::window
already accepts such expressions and both filter pushdown and the physical
planner unwrap them. Look through one alias to derive the sort key and keep the
aliased expression in the group. Nested aliases are still rejected, matching
what filter pushdown tolerates.

Co-authored-by: AI assistants <ai-assistants@users.noreply.github.com>
Substrait projections use positional references and omit intermediate
window aliases. An inherited window column and a new window expression
can therefore share the same default schema name, causing Window::try_new
to reject the rebuilt plan with DuplicateUnqualifiedField.

Reserve input schema names in a NameTracker, alias colliding window
expressions, and rewrite their projection references. Use NamePreserver
to preserve the projection's output names. Leave noncolliding window
expressions unaliased.

Add builder and SQL regressions checking output schemas and execution
for chained windows with identical default names.

Closes apache#23007

Co-authored-by: AI assistants <ai-assistants@users.noreply.github.com>
@github-actions github-actions Bot added logical-expr Logical plan and expressions substrait Changes to the substrait crate labels Sep 11, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.56627% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (7e5f40a) to head (4dbab95).

Files with missing lines Patch % Lines
...trait/src/logical_plan/consumer/rel/project_rel.rs 90.47% 0 Missing and 4 partials ⚠️
datafusion/expr/src/utils.rs 91.66% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25181      +/-   ##
==========================================
- Coverage   81.91%   81.91%   -0.01%     
==========================================
  Files        1132     1132              
  Lines      421117   421189      +72     
  Branches   421117   421189      +72     
==========================================
+ Hits       344961   345016      +55     
- Misses      55767    55776       +9     
- Partials    20389    20397       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait consumer: DuplicateUnqualifiedField when chaining two Window relations with a carried window column

2 participants