xr.DataArray.to_series returns a (mutable) view

[dhirschfeld]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd
import xarray as xr

dates = pd.date_range('01-Jan-2019', '01-Jan-2020', name='forward_date')[:-1]
s = pd.Series(np.random.randn(dates.size), dates)
array = xr.DataArray.from_series(s)

In [52]: array
Out[52]: 
<xarray.DataArray (forward_date: 365)>
array([ 0.511149, -1.009054, -0.098397, ...,  0.496596,  1.435346,  0.21206 ])
Coordinates:
  * forward_date  (forward_date) datetime64[ns] 2019-01-01 ... 2019-12-31

In [53]: array.to_series().index.name
Out[53]: 'forward_date'

In [54]: array.to_series().index.name = 'peristent!!!'

In [55]: array.to_series().index.name
Out[55]: 'peristent!!!'

In [56]: array.get_index('forward_date')
Out[56]: 
DatetimeIndex(['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04',
               '2019-01-05', '2019-01-06', '2019-01-07', '2019-01-08',
               '2019-01-09', '2019-01-10',
               ...
               '2019-12-22', '2019-12-23', '2019-12-24', '2019-12-25',
               '2019-12-26', '2019-12-27', '2019-12-28', '2019-12-29',
               '2019-12-30', '2019-12-31'],
              dtype='datetime64[ns]', name='peristent!!!', length=365, freq='D')

Problem description

IMHO Setting index.name on the returned series shouldn't affect the original DataArray - i.e. every call to array.to_series().index.name should return forward_date irrespective of any mutating operations performed on the returned series.

Expected Output

Output of xr.show_versions()

Details

In [48]: xr.show_versions()
Traceback (most recent call last):

  File "<ipython-input-48-6f391305f2fe>", line 1, in <module>
    xr.show_versions()

  File "C:\Users\dhirschf\envs\dev\lib\site-packages\xarray\util\print_versions.py", line 84, in show_versions
    sys_info.extend(netcdf_and_hdf5_versions())

  File "C:\Users\dhirschf\envs\dev\lib\site-packages\xarray\util\print_versions.py", line 75, in netcdf_and_hdf5_versions
    libhdf5_version = h5py.__hdf5libversion__

AttributeError: module 'h5py' has no attribute '__hdf5libversion__'

[shoyer]

I would tend to agree here -- the only data for which we want to use views are (potentially large) numpy arrays.

[dcherian]

@dhirschfeld can you put together a PR to fix this?

[dhirschfeld]

Agreed that we definitely don't want to copy the data! I haven't.put in a PR as I think we're a bit hamstrung by pandas' behaviour here so I'm not quite sure how to fix it (without copying the data).

I'm unlikely to get around to a PR in the near term as I'm swamped (and can put up with deep copying the index for now). From my digging about I think the proper fix should be in pandas such that a shallow copy doesn't return a reference to the original name.

[shoyer]

I don't think we even bother to shallow copy the pandas Index from the DataArray. We should probably do that.

[dhirschfeld]

Unfortunately a shallow copy of a pandas series doesn't shallow copy it's index (I think it should):

>>> series = array.to_series().copy(deep=False)
>>> series.index.name = 'shallow_copy'
>>> array.to_series().index.name
'shallow_copy'    # <--- I'd call this a pandas bug!

[dhirschfeld]

..so you'd need to do:

>>> series = array.to_series().copy(deep=False)
>>> series.index = series.index.copy(deep=False)
>>> series.index.name = 'shallow_copy'
>>> series.index.name
'shallow_copy'
>>> array.to_series().index.name
'forward_date'

[gwgundersen]

I've created PR #3126 to fix this issue.

[sjvrijn]

Since PR #3126 seems to be closed due to performance issues, is this underlying issue still considered an issue, or should it be closed?

[chiruu12]

Still reproduces on main. get_index hands back the stored pandas index, and pandas lets you set .name in place, so a caller can rename the coordinate of the object they got it from:

array = xr.DataArray(np.zeros(2), coords={"x": ["a", "b"]}, dims=["x"])
array.get_index("x").name = "renamed"
array.get_index("x").name   # 'renamed'

Dataset has it too, since get_index is inherited from DataWithCoords.

PR #3126 fixed this in 2019 and stalled on an unanswered question about whether copying would cost anything. It does not. copy(deep=False) on a pandas index is constant time, because the index object is immutable and the copy shares the same underlying array:

      1,000 elements   1.16 us
  1,000,000 elements   1.15 us
 10,000,000 elements   1.19 us

pandas 3.0.5, mean of 10000 calls. xarray already leans on this: the comment at xarray/core/variable.py:3005 says building a new pandas.Index from an old one "is basically free as pandas.Index objects are immutable".

One thing the fix does not cover. Indexes.__getitem__, so ds.indexes["x"], still returns the stored object. xarray/tests/test_indexes.py:630 asserts that identity (indexes["x"] is x_idx), so copying there changes a tested contract and is a call for maintainers rather than something to slip into a bug fix.

to_index() and to_dataframe() were already safe, they go through set_names() which returns a new object.

Opened #11563 with the fix and tests for both DataArray and Dataset.