Let concat promote a mix of numpy and pandas extension coord dtypes - #11573
Open
dchaudhari7177 wants to merge 1 commit into
Open
Let concat promote a mix of numpy and pandas extension coord dtypes#11573dchaudhari7177 wants to merge 1 commit into
dchaudhari7177 wants to merge 1 commit into
Conversation
PandasIndex.concat promotes the coordinate dtypes of the indexes it joins
with np.result_type, which cannot interpret a pandas extension dtype at
all. Under pandas 3 a string index carries a StringDtype, so joining an
object whose dimension coordinate came from a pd.Index with one whose came
from a plain list raises
TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type
_concat_indexes has already built the joined index at that point, and
pandas promoted the dtypes to do it. Use the dtype it settled on when any
input is an extension dtype, rather than reimplementing pandas' promotion
rules. Two numpy dtypes still go through np.result_type unchanged.
Fixes pydata#11317.
|
Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11317
The failure
PandasIndex.concatpromotes the coordinate dtypes of the indexes it joins withnp.result_type(xarray/core/indexes.py:793), which cannot interpret a pandas extension dtype. Under pandas 3 a string index carries aStringDtype, so the reporter's case has one coord at<U1and the other atStringDtypeand raises:This is not an exotic mix — it is what you get whenever one dimension coordinate came from a
pd.Index(which is howconcatis documented to add a new labelled dimension) and another from a plain list.The fix
By the time the dtypes are promoted,
_concat_indexeshas already built the joined index, and pandas promoted the dtypes to do so. When any input coord dtype is apd.api.extensions.ExtensionDtype, use the dtype pandas settled on rather than reimplementing its promotion rules here:Two numpy dtypes still go through
np.result_typeunchanged, so ordinary promotion (<U1+<U3→<U3) is untouched.I deliberately did not touch
xarray.core.dtypes.result_type, which fails the same way on this pair. Making the general promotion path extension-aware is a larger decision about the pandas-3 string transition and would likely collide with in-flight work such as #11474; this keeps the change to the one site the issue is about.Verified
dim_awith values['a', 'b'].xr.Variableform the reporter noted still works is unchanged (<U1).indexes.pyreverted,test_concat_mixed_numpy_and_extension_coord_dtypefails with the originalTypeError, and the numpy-path test passes either way — so it is guarding against the new branch swallowing the ordinary case.pytest xarray/tests/test_concat.py xarray/tests/test_indexes.py xarray/tests/test_dataset.py— 680 passed, 59 skipped, 1 xfailed, 1 xpassed.ruff checkclean at the pinned v0.15.20.Environment: pandas 3.0.3, numpy 2.x, Windows.
whats-new.rstentry added under Bug fixes.