[CI][Wheels] Get WoA back with PyManager

[khmyznikov]

• Reverting the PR drop building and testing against windows arm64 #14216
• Adding win_arm64 specific section for python version management via PyManager.

First of all, the GitHub team is committed to bringing the Windows 11 ARM image under its ownership. This should eliminate communication delays between the ARM and GitHub teams. I’ll give you the dates as soon as I get them.

Secondly, I’m not satisfied with how the original issue was resolved. It’s unclear how they are preventing it from happening again. That’s why I’m proposing the use of pymanager, the recommended way to manage Python versions on Windows. This should eliminate a recurrence of the original issue.

Lastly, I've removed the combo {VERSION: "3.14", "ABI_VERSION": "py38"} for win_arm64, since there's no official builds of python below 3.11 for WoA.

Thank you in advance for your patience and consideration.

[reaperhulk]

First of all, the GitHub team is committed to bringing the Windows 11 ARM image under its ownership

This is great news. We will be much more interested in merging this and resuming win_arm64 support once this transition has occurred!

[reaperhulk]

Just for reference as we track this: Our desired end state would be being able to treat the WoA builders exactly as we do x86_64 Windows. Bringing it back in-house and making sure there aren't basic image differences/broken core actions will be enormously helpful.

[alex]

Thanks for working on this. Just to hone in on what Paul said, we really don't want to carry a big "if arm64" block :-) And ideally we wouldn't have to carry that much windows specific code (if PyManager is the right thing, maybe setup-python should use it?)

[khmyznikov]

@alex Yeah, I don't like this big chunk neither. I've asked maintainer if we can have pymanager included in the image, that should trim down this fix a lot.

[khmyznikov]

I have good news I can't share yet, but just want to let you know we are not abandoned this and working on full resolution.

[alex]

@khmyznikov FYI, it looks like there's been another regression in win-arm64 + setup-python -- see pyca/bcrypt#1193. I haven't fully diagnosed the cause.

I do see that the image has moved from partner images to the primary image repo, which is an encouraging signal. But to be very direct: repeated breakages like this are precisely the concern we had that led to us not re-adding arm64 to our CI after the original issue was fixed.

[khmyznikov]

@alex Thank you for keeping your finger on the pulse! As a first step, we expect the image to be moved to the main repository very soon. I’m discussing with the team how we can best publicly demonstrate the readiness of new versions. I would be happy to hear any suggestions regarding CI testing.

[alex]

I don't know a ton about what y'all already do, so there's a big risk of suggesting things that y'all already do. That said, here are some things I suspect are not currently done:

• Smoke tests with live actions. When a new image is ready to be pushed, deploy it with some internal ubuntu-22.04-secret-next or whatever label, and then run some real jobs. You can build up a set of workflows that excersise common things: test every supported python version with setup-python on the new image, for example. If those all pass, then it can move on to next gates.
• This is harder, but some sort of shadow/canary builds: pick some real repos+workflows, and when they're triggered, add shadow versions of jobs using the new image and run them. If the new image fails and the old one passed, that's a potential indicator of problems.

[reaperhulk]

Let me add that working rapidly to fix major breakage is also critical. I say this because the basic and total breakage of setup-python for a subset of versions on windows arm64, which we noted 2 days ago, is not fixed and we have no idea when it will be. That's so far beyond unacceptable as to border on farce. Since we've had our entire CI blocked in both PyNaCl and bcrypt due to this we'll be dropping Windows ARM support there today as well.

[khmyznikov]

@reaperhulk please hold. GitHub team reproduces the issue and willing to address it ASAP. The issues was they didn't tested the latest package of the setup python, which was resolved and now will be included into test matrix.

[paveliak]

@reaperhulk Sorry for the troubles you are experiencing with Python on ARM. We are investigating this problem. The workaround is to uninstall Python version that comes cached in the image

    - run: |
        Start-Process -FilePath 'c:\hostedtoolcache\windows\Python\3.14.4\arm64\python-3.14.4-arm64.exe' -ArgumentList @('/uninstall', '/quiet') -Wait
        Remove-Item -Recurse -Force 'c:\hostedtoolcache\windows\Python\3.14.4'
        
    - uses: actions/setup-python@v6.2.0
      with:
        python-version: '3.14t'

In future please open an issue in https://github.com/actions/runner-images and https://github.com/actions/setup-python 🙏

[alex]

To be clear, we're simply dropping Windows+ARM64 from all projects we maintain until we have confidence it will be stable. We're not going to carry workarounds for this.

[reaperhulk]

We've removed support across all our projects so this is no longer an issue for us. It has become clear over the past months that the set of repositories, processes, and procedures that encompass the Windows Arm64 GitHub Actions hosted runner feature lacks basic testing rigor and the organization is not capable of quickly or efficiently addressing those deficiencies. This is not a commentary on any individual contributor, as I'm sure everyone is trying hard within the structural constraints imposed by your employer, but the outcome is that we are unwilling to continue supporting it.

As before, we'll revisit at some point in the future, but for now no PyCA project will support Windows Arm64 wheels.