Skip to content

Let concat promote a mix of numpy and pandas extension coord dtypes - #11573

Open
dchaudhari7177 wants to merge 1 commit into
pydata:mainfrom
dchaudhari7177:fix/11317-concat-extension-coord-dtype
Open

Let concat promote a mix of numpy and pandas extension coord dtypes#11573
dchaudhari7177 wants to merge 1 commit into
pydata:mainfrom
dchaudhari7177:fix/11317-concat-extension-coord-dtype

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Fixes #11317

The failure

PandasIndex.concat promotes the coordinate dtypes of the indexes it joins with np.result_type (xarray/core/indexes.py:793), which cannot interpret a pandas extension dtype. Under pandas 3 a string index carries a StringDtype, so the reporter's case has one coord at <U1 and the other at StringDtype and raises:

TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type

This is not an exotic mix — it is what you get whenever one dimension coordinate came from a pd.Index (which is how concat is documented to add a new labelled dimension) and another from a plain list.

The fix

By the time the dtypes are promoted, _concat_indexes has already built the joined index, and pandas promoted the dtypes to do so. When any input coord dtype is a pd.api.extensions.ExtensionDtype, use the dtype pandas settled on rather than reimplementing its promotion rules here:

elif any(
    isinstance(dtype, pd.api.extensions.ExtensionDtype)
    for dtype in indexes_coord_dtypes
):
    coord_dtype = new_pd_index.dtype

Two numpy dtypes still go through np.result_type unchanged, so ordinary promotion (<U1 + <U3<U3) is untouched.

I deliberately did not touch xarray.core.dtypes.result_type, which fails the same way on this pair. Making the general promotion path extension-aware is a larger decision about the pandas-3 string transition and would likely collide with in-flight work such as #11474; this keeps the change to the one site the issue is about.

Verified

  • The issue's reproducer now returns a 2-element dim_a with values ['a', 'b'].
  • The xr.Variable form the reporter noted still works is unchanged (<U1).
  • Red before green: with both new tests in place and indexes.py reverted, test_concat_mixed_numpy_and_extension_coord_dtype fails with the original TypeError, and the numpy-path test passes either way — so it is guarding against the new branch swallowing the ordinary case.
  • pytest xarray/tests/test_concat.py xarray/tests/test_indexes.py xarray/tests/test_dataset.py — 680 passed, 59 skipped, 1 xfailed, 1 xpassed.
  • ruff check clean at the pinned v0.15.20.

Environment: pandas 3.0.3, numpy 2.x, Windows.

whats-new.rst entry added under Bug fixes.

PandasIndex.concat promotes the coordinate dtypes of the indexes it joins
with np.result_type, which cannot interpret a pandas extension dtype at
all. Under pandas 3 a string index carries a StringDtype, so joining an
object whose dimension coordinate came from a pd.Index with one whose came
from a plain list raises

    TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type

_concat_indexes has already built the joined index at that point, and
pandas promoted the dtypes to do it. Use the dtype it settled on when any
input is an extension dtype, rather than reimplementing pandas' promotion
rules. Two numpy dtypes still go through np.result_type unchanged.

Fixes pydata#11317.
@welcome

welcome Bot commented Sep 7, 2026

Copy link
Copy Markdown

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

concat fails due to StringDtype introduced by pd.Index

1 participant