[BUG] [CI] A/B trial for ARM64 wheels — hardened QEMU (9 auto-retries) vs native `ubuntu-24.04-arm`: run both for weeks, keep the winner (we tried native before, it wasn't ready)

[oberstet]

Motivation

Our ARM64 wheels are built via QEMU aarch64 emulation on x86_64 ubuntu-latest. That path is:

• Slow — the workflow itself notes builds take "45+ minutes due to emulation overhead".
• Flaky — e.g. pypy-3.11-bookworm-manylinux_2_36_aarch64 recently died with exit 139 (SIGSEGV) after 3 retries (Actions run 35626342331), the known PyPy-under-QEMU instability the Dockerfile already tries to tame (PYPY_DISABLE_JIT=1, etc.).

GitHub now offers native ubuntu-24.04-arm / ubuntu-22.04-arm runners, GA 2025-08-07, free for public repositories (autobahn-python qualifies; the label is public-repo-only). Native runners would eliminate the emulation-segfault class entirely and cut build time by ~10×.

Important — this is a re-evaluation, not a first attempt. We already tried GH-hosted ARM64 Linux runners once (≈ 3H 2025) and rejected them: they were not readily available — jobs queued far too long to be usable — and that rejection is the whole reason the QEMU-emulation path was built (many hours of work). What has changed since is the offering itself: those were the Jan-2025 public preview; native only reached GA on Cobalt-100 in Aug-2025. So the prior "it sucked" verdict is real but possibly stale — worth re-testing with a skeptical prior, precisely because being wrong is cheap here and being right retires a large maintenance burden.

But GA + free still ≠ readily available — and availability is exactly what failed last time. On the free shared pool, pickup latency is unknown and varies by time-of-day / day-of-week / demand; if jobs sit queued for hours it is worthless. So rather than assume, we run a head-to-head A/B trial for several weeks and keep whichever path delivers green ARM64 wheels more reliably and with less manual intervention:

• Arm A — hardened QEMU (incumbent): today's emulated path, but with auto-retries raised 3 → 9 so it self-heals without anyone pressing "retry". Remains the release path during the trial.
• Arm B — native (challenger): ubuntu-24.04-arm, Docker/manylinux, no QEMU. Artifact-only during the trial.

If native still isn't ready, we keep (hardened) QEMU — this time with data, not a hunch.

Current state (for reference)

• wheels-arm64.yml: runs-on: ubuntu-latest (x86_64) + QEMU/binfmt + Docker.
• CPython ARM64 → official PyPA manylinux images (manylinux_2_28_aarch64) + musllinux.
• PyPy ARM64 → custom images (FROM pypy:3.11-bookworm / -trixie, i.e. upstream PyPy — we do not build PyPy), because there is no official PyPA PyPy manylinux image.
• QEMU is registered three times (tonistiigi/binfmt --install all → docker/setup-qemu-action → multiarch/qemu-user-static --reset), the last of which likely downgrades the emulator — tracked separately as the interim QEMU-stability fix.

Constraints (must preserve)

• Keep pypy311 — mandatory, upstream-supported. (pypy312 now exists upstream and can be added later as a separate enhancement; no pypy313/314 upstream yet.)
• Build in-container (manylinux_2_28 / musllinux) for portable glibc/musl tags — never build on the bare runner (Ubuntu 24.04 = glibc 2.39).
• Do not build PyPy — keep using upstream published PyPy.
• Releases stay on Arm A (QEMU) until/unless Arm B wins the trial.

Plan (A/B trial, zero-risk)

• Phase 0 — immediate unblock (separate): re-run the flaky QEMU job to clear the current transient failure (tracked in #).
• Phase 0.5 — kill the manual toil now (independent of the trial outcome): raise the emulated job's auto-retries max_attempts: 3 → 9 so it self-heals without manual "retry" clicks, and mark it continue-on-error / non-required (best-effort, Tier-2 per [DOCS] Write down the cross-compilation promise (README + installation docs), with explicit support-status tags #1935) so a still-red emulated job no longer gates merges or paints a red ✗ on unrelated PRs. Bonus signal: if 9 auto-retries still go red, the failure is deterministic (base-image / QEMU drift), pointing to the pin fix in # rather than to transient flakiness.
• Arm A — hardened QEMU (incumbent, remains release path): the existing wheels-arm64.yml with max_attempts: 9; no other change during the trial.
• Arm B — native (challenger, artifact-only): new separate workflow wheels-arm64-native.yml:
  • runs-on: ubuntu-24.04-arm; mirror the existing ARM64 matrix.
  • Build in-container natively — drop QEMU/binfmt steps, --platform linux/arm64, retry wrappers, and the PyPy stability env vars.
  • continue-on-error: true; not a required check; artifact-only (build + smoke-test + upload; not wired into release).
  • Add a schedule: cron trigger (a few times/day) to sample runner availability across the clock, independent of PR cadence.
  • Separate workflow so a queued native job can't hold up Arm A or surface as a pending required check (timeout-minutes counts only running time; an unpicked job can sit queued up to GitHub's ~24h max).
• Measure (~2–4 weeks), both arms in parallel:
  • Time-to-green end-to-end per arm (Arm A includes its retry loops; Arm B includes queue wait).
  • Manual interventions required (Arm A's whole point is to drive this to zero via auto-retry).
  • Arm B pickup latency = job started_at − created_at — the metric that failed last time; the Actions UI shows it per run, pull from the Actions API (dev PC) to chart, and read the cron samples for time-of-day / day-of-week gaps.
  • Success rate per arm; optional diff Arm-A vs Arm-B wheels for the same arch (should be functionally equivalent — a mismatch is worth knowing).
• Decide & cut over: keep the arm that delivers green wheels quicker and with less manual toil. If B wins → point releases at the native artifacts, delete wheels-arm64.yml (or fold into wheels.yml), remove the QEMU machinery. If A wins → keep hardened QEMU as the release path (optionally apply the QEMU/base-image pin from #) and drop Arm B.

Decision criteria (proposed — tune to taste)

Over the ≥2–4-week window, compare the arms and keep the winner. Native (Arm B) supersedes hardened QEMU (Arm A) only if it is clearly better where it matters:

• Availability (Arm B's key risk): ≥ 95% of native jobs picked up within N minutes (proposed N = 10), and no availability gap > M hours (proposed M = 6) in the cron samples — the gate that failed in early 2025.
• Reliability: ≥ 99% green, no infra failures.
• Time-to-green & manual toil: native reaches green faster end-to-end and needs zero manual retries.

If Arm B misses the availability gate → Arm A (hardened QEMU, 9 auto-retries) stays, and since it self-heals without manual clicks that is an acceptable steady state, not a defeat.

Scope / non-goals

• No change to wheel contents or supported Python versions in this issue.
• Adding pypy312 is a separate follow-up (now possible upstream), not part of this migration.
• CPython / macOS / Windows / non-ARM64 builds are untouched.

Impact

• Affected: CI/CD only — new wheels-arm64-native.yml (Arm B), the max_attempts bump in wheels-arm64.yml (Arm A), and release wiring only on cutover.
• Breaking changes: none (both arms are additive/non-blocking; Arm A stays the release path throughout the trial; cutover preserves wheel outputs).
• Expected win if native takes it: ~10× faster ARM64 builds, elimination of QEMU SIGSEGV flakiness, and a large net simplification (delete binfmt/QEMU/retry/stability-hack machinery). If QEMU takes it: zero manual retries via 9 auto-attempts, and the emulated path stays non-blocking.

Notes / risks

• Concurrency: free public-repo runners share a capped concurrency pool; a wide ARM64 matrix may serialize rather than run fully parallel. That's data for the decision, not a blocker (serialized-native still beats emulated).
• Public-repo-only: the ubuntu-24.04-arm label hard-fails in private repos/forks — fine for autobahn, but note it.
• Related: # (interim QEMU stabilization), [DOCS] Write down the cross-compilation promise (README + installation docs), with explicit support-status tags #1935 (wheel-tier governance), Add Windows ARM64 build #1937 (Windows ARM64 — same native-vs-emulation theme).