[james-willis]

Which issue does this PR close?

• Part of CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860.

Rationale for this change

COALESCE, NVL/IFNULL, and NVL2 drop the field metadata of their arguments when computing their return field: return_field_from_args builds the output Field from the argument data types only. For columns carrying Arrow extension types (ARROW:extension:name / ARROW:extension:metadata — e.g. arrow.uuid, geoarrow.wkb), the planned schema of any expression containing these functions silently loses the extension-type identity, and metadata-aware UDFs that dispatch on their argument Fields fail to plan over them (see #24860 for a full reproducer).

This is one of several independent metadata drop sites listed in #24860. It is the plan-time schema fix for the conditional functions. These functions simplify into CASE, whose logical and physical drop sites are addressed separately in #26145 and #26158.

What changes are included in this PR?

• datafusion_common::metadata::coerced_fields_metadata: computes the metadata for an expression whose result may come from any one of several inputs coerced to a common type. It lives next to the related metadata utilities so that CASE (in datafusion-expr) can share it:
  • Metadata is merged across the inputs whose data type is not Null (an untyped NULL literal carries no metadata and cannot contribute a typed value), with earlier inputs taking precedence for repeated keys. Unrelated keys are not required to agree, so metadata that arrived with the data (e.g. from an Arrow file) cannot cause an extension type to be dropped.
  • The extension type keys are handled as a unit, as in Expr::Cast output field derivation: differing values of ARROW:extension:name are a planning error, and ARROW:extension:metadata always comes from the input that supplied the extension name (byte-wise differences in it are tolerated, since extension parameters are often JSON).
• coalesce: return_field_from_args attaches the merged metadata of its arguments. Return type and nullability are unchanged. NVL/IFNULL delegate to coalesce and are fixed by the same change.
• nvl2: same, over the second and third arguments only — the first argument is only tested for NULL and never contributes a value, so its metadata is excluded.

What is the testing strategy for this PR?

• Unit tests for coerced_fields_metadata in metadata.rs: merge precedence, untyped NULL inputs skipped, empty result, error on differing extension names, and extension name/metadata kept paired.
• Unit tests in coalesce.rs and nvl2.rs exercise return_field_from_args: metadata propagated, untyped NULL arguments not blocking propagation, conflicting extension types rejected, NVL2's test argument excluded, and nullability unchanged.

There is deliberately no sqllogictest yet: arrow_metadata() reads its argument field at execution time, and by then these functions have been simplified into CASE, so end-to-end coverage depends on the CASE fixes (#26145, #26158). This ordering is safe: the optimizer's schema invariant ignores metadata (assert_expected_schema / logically_equivalent_names_and_types), so a plan whose coalesce carries metadata that the simplified CASE does not still plans.

Are there any user-facing changes?

• The planned schema (e.g. DataFrame::schema()) of expressions containing COALESCE/NVL/IFNULL/NVL2 over metadata-bearing columns now preserves that metadata.
• COALESCE/NVL/IFNULL/NVL2 over arguments with differing ARROW:extension:name values (e.g. an arrow.uuid column and an arrow.json column) is now a planning error. Previously these planned and silently dropped the extension type.
• New public function datafusion_common::metadata::coerced_fields_metadata. No other API changes.

[james-willis]

In this PR, if the branches of the coalesce, etc don't agree on the metadata, we drop extension metadata defensively.

My opinion is that this should actually be a planning time error unless all rows will hit only one branch but wanted to take the more conservative approach for now. I want to solicit opinions from the reviewers on what the desried behavior is here.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.70130% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.58%. Comparing base (fb42cf5) to head (e76a9b8).
⚠️ Report is 40 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/metadata.rs	97.00%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24865      +/-   ##
==========================================
+ Coverage   82.49%   82.58%   +0.09%     
==========================================
  Files        1140     1142       +2     
  Lines      439034   440994    +1960     
  Branches   439034   440994    +1960     
==========================================
+ Hits       362165   364182    +2017     
+ Misses      54962    54813     -149     
- Partials    21907    21999      +92     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[paleolimbot]

Thank you for this fix! I'm not a committer so I can't help with the CI but I'll review from a metadata perspective.

I think the family of functions that are affected here are ones that have >1 "compatible" (castable to a common type) inputs. While there are some extension aware type systems that can handle this (notably: vctrs via vec_ptype2() and vec_cast()), datafusion has no mechanism for that and so probably the best we can do for now is to propagate agressively and rely on engines to write analyzer or optimizer rules to enforce other behaviour.

One other place I've seen this pattern is combining values in a values literal clause. I would personally love if the logic were the same there but that might insert too much scope here.

datafusion/datafusion/expr/src/logical_plan/builder.rs

Lines 309 to 320 in d02bd78

	let metadata = value.metadata(&schema)?;
	if let Some(ref cm) = common_metadata {
	if &metadata != cm {
	return plan_err!(
	"Inconsistent metadata across values list at row {i} column {j}. Was {:?} but found {:?}",
	cm,
	metadata
	);
	}
	} else {
	common_metadata = Some(metadata.clone());
	}

I am curious what @timsaucer would expect to happen to metadata in CASE WHEN since that is a metadata usage I'm less familiar with!

[paleolimbot]

I know you're still working here, just making sure I didn't forget to review!

[paleolimbot]

@james-willis Can you address the merge conflicts? I can bring up this PR in the weekly sync tomorrow to get a second review if it's ready.

[paleolimbot]

Thank you!

This would benefit from a second review to double check my take on metadata merging here, but in general I think this a standalone improvement and fixes a hard to work around bug.

[paleolimbot]

I recently discovered intersect_metadata_for_union(), which silently discards non-matching metadata keys. I like this version better (I want a louder error when the types don't match), but either is OK with me if we want to keep the existing logic for consistency.

[paleolimbot]

Happy to hear another opinion, but I do think this is a bug worth fixing regardless of whether #26145 is a good idea.