[spack-stack/2.1] esmpy 8.9.1 does not import

[benkozi]

Description

esmpy import fails on Ursa v2.1 with:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/install/intel-oneapi-compilers/2025.3.1/esmf-8.9.1-otgqi73/lib/python3.11/site-packages/esmpy/__init__.py", line 108, in <module>
    from esmpy.api.esmpymanager import *
  File "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/install/intel-oneapi-compilers/2025.3.1/esmf-8.9.1-otgqi73/lib/python3.11/site-packages/esmpy/api/esmpymanager.py", line 9, in <module>
    from esmpy.interface.cbindings import *
  File "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/install/intel-oneapi-compilers/2025.3.1/esmf-8.9.1-otgqi73/lib/python3.11/site-packages/esmpy/interface/cbindings.py", line 13, in <module>
    from esmpy.interface.loadESMF import _ESMF
  File "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/install/intel-oneapi-compilers/2025.3.1/esmf-8.9.1-otgqi73/lib/python3.11/site-packages/esmpy/interface/loadESMF.py", line 81, in <module>
    _check_version(esmfversion=esmfversion,
  File "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/install/intel-oneapi-compilers/2025.3.1/esmf-8.9.1-otgqi73/lib/python3.11/site-packages/esmpy/interface/loadESMF_helpers.py", line 39, in _check_version
    raise VersionMismatch(f"Cannot use an ESMF development version ({esmfversion}) "
esmpy.util.exceptions.VersionMismatch: Cannot use an ESMF development version (8.9.1 beta snapshot) with an ESMPy release version (8.9.1)

To Reproduce

#!/usr/bin/env bash

set -ue

module use /contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/modules/Core
module load stack-intel-oneapi-compilers/2025.3.1
module load stack-intel-oneapi-mpi/2021.17

module load esmf/8.9.1

which python
python -c "import esmpy"
echo "successfully imported"

Expected behavior

esmpy should import without an ESMF version conflict.

System, compiler, code, ...

• Ursa
• spack-stack v2.1

Additional context

Add any other context about the problem here.

[climbfuji]

@theurich I think this is because we used commit 6a09112 for release 8.9.1 at first, because the final tag wasn't ready. Looking at https://github.com/esmf-org/esmf/commits/v8.9.1/, it seems that commit bd03a24 is that one that fixes this issue, correct?

Is there another workaround instead of updating the ESMF commit in release/2.1 and reinstalling?

I know we recently replaced the commit for esmf@8.9.1 with the actual tag in spack-stack develop, but release/2.1 was cut earlier.

[theurich]

@climbfuji I am afraid that resolving this requires using the correct 8.9.1 release in spack-stack/2.1. It's commit hash bd03a24 (as you say), or just use the actual release tag v8.9.1 (as on spack-stack develop).

I should have paid more attention to this when spack-stack/2.1 was going out, that since we did not have an official v8.9.1 yet for ESMF, it should have contained ESMF 8.9.1 as beta. This way it would all be consistent.

Maybe the best way forward is to push out a patch spack-stack/2.1.1 that uses the correct ESMF v8.9.1 tag.

[climbfuji]

@climbfuji I am afraid that resolving this requires using the correct 8.9.1 release in spack-stack/2.1. It's commit hash bd03a24 (as you say), or just use the actual release tag v8.9.1 (as on spack-stack develop).

I should have paid more attention to this when spack-stack/2.1 was going out, that since we did not have an official v8.9.1 yet for ESMF, it should have contained ESMF 8.9.1 as beta. This way it would all be consistent.

Maybe the best way forward is to push out a patch spack-stack/2.1.1 that uses the correct ESMF v8.9.1 tag.

That's what I feared, thanks for the clarification. @benkozi this will need to wait for a little bit; I think we can push for a 2.1.1 bug fix release in April to give us all a chance finding more issues in 2.1.0 and then fixing them at the same time :-) Will that work for you?

[benkozi]

Yes, it can wait for sure.

One thought as a workaround (a hack really) would be to comment out: https://github.com/esmf-org/esmf/blob/develop/src/addon/esmpy/src/esmpy/interface/loadESMF.py#L81 on the platforms directly until 2.1.1 is rolled out. As far as I know, esmpy is installed using the same esmf hash, so everything should be consistent from the Python perspective. Right now, we're just looking to use the 2.1+ stack on Ursa and Gaea-C6. I could work to test this on Ursa with someone having appropriate perms. Maybe this is too much of an anti-pattern though 😅

[climbfuji]

I am afraid it is [too much of an anti-pattern]. I am sure we can maybe a 2.1.1-rc1 available on a selected platform (Ursa?) soon after we published the 2.1.0 release.

[benkozi]

I've had success building downstream envs with a working esmpy 8.9.1, so I don't think a rush build is necessary. I would not refuse them though! In our case, Gaea-C6 > Ursa.

[theurich]

@benkozi I agree that the hack you suggest at https://github.com/esmf-org/esmf/blob/develop/src/addon/esmpy/src/esmpy/interface/loadESMF.py#L81 should be safe as a work-around for the specific scenario. (Where you technically have a consistent installation, just that during installation ESMPy was told it is 8.9.1, while it is actually from an ESMF beta install.)

I also agree that having this fixed in a spack-stack patch is ultimately the clean way to go.