Skip to content

fix: Preserve metadata when a cross-join is swapped#23605

Merged
alamb merged 2 commits into
apache:mainfrom
mkleen:metadata-join-swap
Jul 20, 2026
Merged

fix: Preserve metadata when a cross-join is swapped#23605
alamb merged 2 commits into
apache:mainfrom
mkleen:metadata-join-swap

Conversation

@mkleen

@mkleen mkleen commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When a cross-join is swapped, the metadata is rebuilt based on the inverted inputs. This can cause a change of metadata when the metadata has conflicting values e.g.:

// left: 1000 rows, schema metadata {"key": "left"}
// right: 2 rows, schema metadata {"key": "right"}
// Cross join will swap the sides
ctx.sql("select * from left cross join right").await?.collect().await?;
// -> Internal error: ... 'join_selection' failed. Schema mismatch.
// Metadata should be {"key" : "right" } but changed to {"key" : "left" } 
// because of the swap of sides and the rebuilt of the metadata starting from the other direction.

The cross-join should preserve the original metdata when it's swapped.

What changes are included in this PR?

  • This pr avoids rebuilding the metadata based on the swapped inputs and uses the original metadata instead
  • Test

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 15, 2026
@mkleen mkleen changed the title fix: Preserve schema and metadata when a cross-join is swapped fix: Preserve metadata when a cross-join is swapped Jul 15, 2026
@mkleen
mkleen marked this pull request as ready for review July 15, 2026 10:17
@mkleen
mkleen force-pushed the metadata-join-swap branch from 5176f86 to 8fe3468 Compare July 15, 2026 11:19
pub fn swap_inputs(&self) -> Result<Arc<dyn ExecutionPlan>> {
let new_join =
CrossJoinExec::new(Arc::clone(&self.right), Arc::clone(&self.left));
// Rebuild schema with columns from right to left, preserve existing metadata

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rebuilds the schema in the same way as CrossJoinExec::new does and preserves the metadata.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can make a helper (as a follow on) to handle this logic rather than sprinkling the same fix through the code in many places 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want the helper as part of this PR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do it as a follow on PR perhaps

@alamb

alamb commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

FYI @fangchenli

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me -- thank you @mkleen

pub fn swap_inputs(&self) -> Result<Arc<dyn ExecutionPlan>> {
let new_join =
CrossJoinExec::new(Arc::clone(&self.right), Arc::clone(&self.left));
// Rebuild schema with columns from right to left, preserve existing metadata

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can make a helper (as a follow on) to handle this logic rather than sprinkling the same fix through the code in many places 🤔

@mkleen

mkleen commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

@alamb Thanks for the review!

@alamb
alamb added this pull request to the merge queue Jul 20, 2026
@alamb

alamb commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Thanks @mkleen

Merged via the queue into apache:main with commit d75742e Jul 20, 2026
42 checks passed
@fangchenli

Copy link
Copy Markdown

Sorry, I didn't get a chance to look at it. Thanks for the PR @mkleen and thanks for the review @alamb

@2010YOUY01

Copy link
Copy Markdown
Contributor

Do we have an end-to-end bug reproducer (SQL/query level)? This looks like a major bug, and I think the given UT can't guarantee the test goal, here is the reason: #23759 (comment)

I can help to look into it further.

@mkleen

mkleen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Do we have an end-to-end bug reproducer (SQL/query level)? This looks like a major bug, and I think the given UT can't guarantee the test goal, here is the reason: #23759 (comment)

I can help to look into it further.

@2010YOUY01 See #23857

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants