From 2e4b292aba867651c6aa1fa30f3f34fcb9a88189 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Mon, 10 Aug 2026 13:27:32 -0500 Subject: [PATCH 1/6] CI/docs: gate macOS float32 on Ooura, and say which backend certified what MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The macOS float32 rows have been red on and off since 2026-07-27 and were being read as CI flake. They are not flake: tap/MuTap#31 established that vDSP_fft_zrip at N=2048 returns one of two bit-exact outputs for identical input, drawn once per process, and that the residual suppressor amplifies that draw into a ~97 dB swing on the G.168 §7 tone row. The battery was reporting a real property of a shipped configuration, correctly, for two weeks. Two consequences, both handled here. CI. macOS now gates on the Ooura float32 backend, and vDSP moves to a second, non-gating leg that keeps running the same battery plus a --repeat until-fail:20 pass over the three implicated rows, so the draw rate is recorded rather than sampled once. This stops the flake reddening unrelated PRs without hiding it, and without pre-judging the shipping backend decision that #31 still has to make. The gating job deliberately keeps the name "macOS AppleClang" so branch protection referencing that check keeps working. Both macOS legs now record chip identity and the backend actually built, because #31's draw appears on an M1 VM and does not reproduce on Intel — a bare "macOS" label cannot attribute a result. Docs. docs/itu-compliance.md claimed the float32 battery was "all green on the host CI legs" without naming a backend, which reads as covering a configuration it does not. Spell out that the certified float32 numbers come from Ooura (Linux x2, Windows) and CMSIS Helium + its Ooura fallback (M55) — every target that ships today, all deterministic — and carve out Apple/vDSP as NOT currently certified, pending #31. Also record why a single run cannot certify a bimodal outcome, and that the section's own argument for the precision axis ("if double-passes-implies-float-passes were sound, the tone row could not exist") transposes onto backends unchanged. Verified locally: with -DTAP_DSP_FFT_ACCELERATE=OFF the three implicated rows pass 5/5 repeats each on macOS 15.7.7 / Intel. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- .github/workflows/ci.yml | 60 ++++++++++++++++++++++++++++++++++++++-- docs/itu-compliance.md | 39 +++++++++++++++++++++++++- 2 files changed, 96 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 196cbc6..4b8486d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -12,6 +12,9 @@ jobs: build-and-test: name: ${{ matrix.name }} runs-on: ${{ matrix.os }} + # The vDSP float32 leg is a known-flaky observation post, not a gate — see + # the matrix comment below and tap/MuTap#31. + continue-on-error: ${{ matrix.nonblocking }} strategy: fail-fast: false matrix: @@ -22,18 +25,41 @@ jobs: cxx: g++ werror: ON capi: ON + nonblocking: false - name: Linux Clang os: ubuntu-latest cc: clang cxx: clang++ werror: ON capi: ON - # Warnings stay non-fatal on MSVC until /W4 output has been triaged - # on a Windows runner (same policy as the sibling *Tap repos). + nonblocking: false + # macOS gates on the OOURA float32 backend. Apple's vDSP is the + # platform default and stays exercised by the leg below, but + # vDSP_fft_zrip at N=2048 returns one of two bit-exact outputs for + # identical input, drawn once per process (tap/MuTap#31), and the + # residual suppressor amplifies that into a ~97 dB swing on the + # G.168 §7 tone row. A single run of the float32 battery on vDSP is + # therefore a ~72/28 coin flip rather than a gate, and it has been + # reddening unrelated PRs since 2026-07-27. + # + # This job deliberately keeps the name "macOS AppleClang" so branch + # protection referencing that check keeps working. Restore vDSP as + # the gating backend — and delete the leg below — when #31 resolves. - name: macOS AppleClang os: macos-latest werror: ON capi: ON + nonblocking: false + extra_cmake: -DTAP_DSP_FFT_ACCELERATE=OFF + # Non-gating: keeps the vDSP float32 draw visible (and measures how + # often it bites) without blocking work that has nothing to do with it. + - name: macOS AppleClang (vDSP float32, non-gating) + os: macos-latest + werror: ON + capi: ON + nonblocking: true + # Warnings stay non-fatal on MSVC until /W4 output has been triaged + # on a Windows runner (same policy as the sibling *Tap repos). # capi stays OFF on Windows: tools/capi carries no __declspec(dllexport) # (unlike DspTap's), so an MSVC build would link a DLL exporting nothing # — it would pass without gating anything. Turn this ON in the same @@ -42,11 +68,23 @@ jobs: os: windows-latest werror: OFF capi: OFF + nonblocking: false steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 with: submodules: recursive + # Chip identity matters on Apple: #31's draw was observed on an M1 VM + # and does NOT reproduce on Intel, so a bare "macOS" label is not enough + # to attribute a past result. + - name: Record host identity (macOS) + if: runner.os == 'macOS' + run: | + sysctl -n machdep.cpu.brand_string + sysctl -n hw.model + sw_vers + cc --version | head -2 + - name: Configure env: CC: ${{ matrix.cc }} @@ -56,6 +94,13 @@ jobs: -DCMAKE_BUILD_TYPE=Release -DMUTAP_WERROR=${{ matrix.werror }} -DMUTAP_BUILD_CAPI=${{ matrix.capi }} + ${{ matrix.extra_cmake }} + + # Record the backend actually built, so a log can never be misread about + # which float32 configuration produced its numbers. + - name: Record float32 FFT backend + if: runner.os == 'macOS' + run: grep '^TAP_DSP_FFT_ACCELERATE' build/CMakeCache.txt || echo 'TAP_DSP_FFT_ACCELERATE not in cache (default)' - name: Build run: cmake --build build --config Release -j 4 @@ -63,6 +108,17 @@ jobs: - name: Test run: ctest --test-dir build -C Release --output-on-failure + # A single pass cannot certify a per-process bimodal outcome: on vDSP the + # float32 rows sample a ~72/28 draw. Repeat the implicated rows on this + # host so the log records how often the draw bites, rather than one + # sample of it. Non-gating by virtue of the job's continue-on-error. + - name: Repeat the float32 rows (vDSP draw rate) + if: matrix.nonblocking + run: > + ctest --test-dir build -C Release --output-on-failure + --repeat until-fail:20 + -R 'itu_echo\.EchoStability|g168_adapted\.ToneStability|Float32Parity\.ToneRowWithNarrowbandGuard' + sanitizers: name: ASan + UBSan runs-on: ubuntu-latest diff --git a/docs/itu-compliance.md b/docs/itu-compliance.md index 119b87e..7005402 100644 --- a/docs/itu-compliance.md +++ b/docs/itu-compliance.md @@ -650,7 +650,44 @@ the four suites above (`test_itu_echo`, `test_itu_doubletalk`, `TYPED_TEST` over `` — float is the `/0` leg, double the `/1` leg — and **the whole battery clears every certified gate at float32 with the same margins as double** (64 rows × both precisions, -all green on the host CI legs). +all green on the host CI legs) — with one backend carved out below. + +### Which FFT backend each float32 number was measured on + +Precision is not the only axis the float32 path varies over: the float32 +real FFT has three backends, and "float32" alone does not name a +configuration. The certified numbers above are measured on **Ooura** +(Linux GCC, Linux Clang, Windows MSVC) and on **CMSIS Helium** and its +**Ooura fallback** on the Cortex-M55 leg — which together cover every +deployment target that ships today. Both are deterministic across +processes. + +**Apple/vDSP float32 is NOT currently certified**, and its rows should be +read as pending rather than passing. `TAP_DSP_FFT_ACCELERATE` defaults ON +for Apple, so the macOS host leg runs the battery against Apple's vDSP — +and `vDSP_fft_zrip` at N=2048 has been measured returning one of two +bit-exact outputs for identical input, drawn once per process +(145/55 over 200 processes on macOS 26.5.2 / Xcode 26.6 / Apple M1; not +reproduced on macOS 15.7.7 / AppleClang 17 / Intel, 200/200 identical at +every size 64–8192). The residual suppressor's N=2048 analysis geometry +sits on that draw, and the chain amplifies it into a ~97 dB difference in +the G.168 §7 tone row. Tracked in +[tap/MuTap#31](https://github.com/tap/MuTap/issues/31). + +Two consequences worth stating plainly, because the battery reported this +correctly for two weeks while it was read as CI flake: + +- A row that is run **once** per leg cannot certify a bimodal outcome. On + the vDSP leg the float32 rows sample a ~72/28 draw, so "green" there is + a sample, not a gate. Repetition (`--repeat until-fail:N`) is what makes + a float32 result evidence, and it is now applied on that leg. +- The argument this section makes for the precision axis — *if "double + passes ⇒ float passes" were sound, the tone row could not exist* — + transposes onto backends unchanged. float32-on-Ooura does not certify + float32-on-vDSP any more than double certifies float. + +The Apple float32 rows return to certified status when #31 resolves and +the battery clears them under repetition on that backend. Why both, rather than certify in double and infer float32: the two are not related by a uniform noise floor. The G.168 §7 tone row is the From a10a717b42ac751b3f6bc0870ef9b5bb3295c1e4 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Mon, 10 Aug 2026 20:23:20 -0500 Subject: [PATCH 2/6] Pin the DspTap vDSP alignment fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps submodules/dsptap to da4dd68 (tap/DspTap#9), which root-caused #31: vDSP dispatches on 64-byte buffer alignment and the two paths disagree bit-for-bit, so std::vector's 16-byte-aligned storage left the choice to wherever the heap landed. The wrapper now aligns its split buffers, and DspTap carries fft_alignment_stability as the regression gate. This removes the per-process nondeterminism. It does NOT change the numerics, and the Intel evidence is unambiguous on that point: itu_echo.EchoStability, fs 48000 level -25, macOS 15.7.7 / Intel before the fix amax-amin = 3.27832747136911 vs a 3.0 gate after the fix amax-amin = 3.27832747136911 vs a 3.0 gate Bit-identical. Intel vDSP never dispatched on alignment (200/200 across sizes both before and after), so there was nothing there for the fix to change; that row fails because vDSP and Ooura differ and the chain amplifies the difference, which is the open half of #31. Note this row misses the HOUSE margin of 3.0 dB while still meeting the ITU requirement of 6.0 dB — margin erosion, not a compliance failure, unlike the G.168 tone row's ~97 dB miss on the M1. The macOS gating deliberately stays on Ooura in this commit. With alignment forced, every M1 process now takes what used to be the 140/200 majority path — and nobody has established whether that path is the one that PASSES the compliance rows or the one that fails them. The non-gating vDSP leg plus its repeat step answers that on this push, and the gating decision follows the evidence rather than the other way round. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- submodules/dsptap | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submodules/dsptap b/submodules/dsptap index 9cbfbca..da4dd68 160000 --- a/submodules/dsptap +++ b/submodules/dsptap @@ -1 +1 @@ -Subproject commit 9cbfbca6670bdc929bab2804c713b60dad8b1c0a +Subproject commit da4dd68bab9ec73ba3ad9e1ce4048b9d8fd50d3d From b8df738dae23340cac9eaa654b5c1af72f9da68d Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Mon, 10 Aug 2026 20:45:55 -0500 Subject: [PATCH 3/6] CI: run the repeat step even when the test step failed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repeat step guarded on `matrix.nonblocking` alone, so a failing Test step skipped it — and that is precisely the case it exists to measure. Both macOS vDSP runs on the pinned-fix commit failed at Test and never reached it, losing the repeat data on the one run where it mattered. `always() && matrix.nonblocking` keeps it scoped to the vDSP leg while letting it run after a failure. Worth recording what those two runs already show, since it is a change in the failure signature rather than a repeat of it. With the alignment fix pinned, BOTH runs failed exactly the same two tests: 157 - g168_adapted.ToneStability 179 - Float32Parity.ToneRowWithNarrowbandGuard and itu_echo.EchoStability, which failed on repetition 1 of both earlier M1 runs, now PASSES. Before the fix the failing set wandered run to run; two runs is not proof of determinism, but an identical set twice plus a removed source of per-process variation is what determinism looks like from here, and the repeat step will settle it. Both survivors are the TONAL rows. The row driven by broadband material (CSS) now passes. That is the shape the empty-bin hypothesis predicts: on an on-bin tone nearly every bin holds nothing but float32 rounding noise, and the chain's behaviour depends on which noise it gets — which no backend choice fixes, because Ooura's noise is not more correct than vDSP's, merely different (and measured against a double reference on this material, ~4x less accurate). So the gating stays on Ooura and the compliance doc's "Apple/vDSP not currently certified" carve-out stands. The fix turned an intermittent failure into an honest one; it did not make the shipped configuration pass. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- .github/workflows/ci.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4b8486d..0c51892 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -112,8 +112,11 @@ jobs: # float32 rows sample a ~72/28 draw. Repeat the implicated rows on this # host so the log records how often the draw bites, rather than one # sample of it. Non-gating by virtue of the job's continue-on-error. + # always(), because the whole point of this step is to characterize a + # failure — and a failing Test step above would otherwise skip it, which + # is exactly what happened on the first run that needed it. - name: Repeat the float32 rows (vDSP draw rate) - if: matrix.nonblocking + if: always() && matrix.nonblocking run: > ctest --test-dir build -C Release --output-on-failure --repeat until-fail:20 From db2fe84f5c989621facae422c1f95cfd63ccaa42 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Tue, 11 Aug 2026 06:00:02 -0500 Subject: [PATCH 4/6] =?UTF-8?q?CI:=20gate=20macOS=20on=20vDSP=20=E2=80=94?= =?UTF-8?q?=20the=20backend=20that=20ships?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverses the gating half of the earlier commit on this branch. That change moved the macOS gate to Ooura to stop a flake reddening unrelated PRs; with the flake's root cause now fixed, the remaining red is not noise, and pointing the gate at a backend nobody ships in order to see green would be certifying a configuration that is not the product. Expect this leg RED. Two rows fail on Apple/vDSP, consistently — both runs since the alignment fix pinned: 157 - g168_adapted.ToneStability 179 - Float32Parity.ToneRowWithNarrowbandGuard itu_echo.EchoStability, which failed on repetition 1 of both earlier M1 runs, now passes. This is deliberate, and it is not the old flake: - The per-process nondeterminism was root-caused to vDSP dispatching on 64-byte buffer alignment and fixed in tap/DspTap#9, pinned here. - The two former draws differed by 2.16e-07 peak-normalized — INSIDE the documented 4e-7 bound. So what remains is the chain converting a within-contract backend difference into a compliance-scale swing. - Both surviving failures are TONAL rows; the broadband-driven row passes. On an on-bin tone nearly every bin holds only float32 rounding noise, and against a double reference on that material per-bin relative error exceeds 1e6 for BOTH backends, with Ooura ~4x LESS accurate than vDSP. Switching backends changes which noise the chain reads, not whether it reads noise. main is unprotected, so a red leg is advisory and does not block merges — the reason it can be honest now rather than after the fix. The repeat step stays, with its purpose restated: it no longer measures a draw rate but whether the remaining failures are STABLE. A row failing 20/20 is a deterministic property; one that still wanders means a second source of variation survives and the diagnosis is incomplete. docs/itu-compliance.md now names the two failing rows, says why the distinction between tonal and broadband material is the finding rather than an aside, and states that they return to certified status when the chain stops taking its answer from empty bins — not when a backend is swapped. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- .github/workflows/ci.yml | 68 +++++++++++++++++++--------------------- docs/itu-compliance.md | 60 ++++++++++++++++++++++++----------- 2 files changed, 73 insertions(+), 55 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0c51892..c90781e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -12,9 +12,6 @@ jobs: build-and-test: name: ${{ matrix.name }} runs-on: ${{ matrix.os }} - # The vDSP float32 leg is a known-flaky observation post, not a gate — see - # the matrix comment below and tap/MuTap#31. - continue-on-error: ${{ matrix.nonblocking }} strategy: fail-fast: false matrix: @@ -25,39 +22,37 @@ jobs: cxx: g++ werror: ON capi: ON - nonblocking: false - name: Linux Clang os: ubuntu-latest cc: clang cxx: clang++ werror: ON capi: ON - nonblocking: false - # macOS gates on the OOURA float32 backend. Apple's vDSP is the - # platform default and stays exercised by the leg below, but - # vDSP_fft_zrip at N=2048 returns one of two bit-exact outputs for - # identical input, drawn once per process (tap/MuTap#31), and the - # residual suppressor amplifies that into a ~97 dB swing on the - # G.168 §7 tone row. A single run of the float32 battery on vDSP is - # therefore a ~72/28 coin flip rather than a gate, and it has been - # reddening unrelated PRs since 2026-07-27. + # macOS GATES ON vDSP — the float32 backend that actually ships on + # Apple, and therefore the one whose failures are the ones that + # matter. It is the platform default, so no flag is needed. # - # This job deliberately keeps the name "macOS AppleClang" so branch - # protection referencing that check keeps working. Restore vDSP as - # the gating backend — and delete the leg below — when #31 resolves. + # Expect this leg RED until the tonal fragility in tap/MuTap#31 is + # fixed: g168_adapted.ToneStability and + # Float32Parity.ToneRowWithNarrowbandGuard fail here, consistently + # (both rows, both runs, since the DspTap alignment fix landed). + # That is deliberate. The failure is real — the shipped Apple + # float32 configuration does not currently meet two certified rows — + # and gating on the backend nobody ships in order to see green would + # be reporting on a configuration that is not the product. + # + # NOT the old flake. #31's per-process nondeterminism was + # root-caused to vDSP dispatching on 64-byte buffer alignment and + # fixed in DspTap (pinned here); what remains is the chain + # amplifying a WITHIN-CONTRACT backend difference (the two former + # draws differed by 2.16e-07 peak-normalized, inside the documented + # 4e-7) into a compliance-scale swing on tonal material. No backend + # choice fixes that: measured against a double reference on tonal + # material, Ooura is ~4x LESS accurate than vDSP. - name: macOS AppleClang os: macos-latest werror: ON capi: ON - nonblocking: false - extra_cmake: -DTAP_DSP_FFT_ACCELERATE=OFF - # Non-gating: keeps the vDSP float32 draw visible (and measures how - # often it bites) without blocking work that has nothing to do with it. - - name: macOS AppleClang (vDSP float32, non-gating) - os: macos-latest - werror: ON - capi: ON - nonblocking: true # Warnings stay non-fatal on MSVC until /W4 output has been triaged # on a Windows runner (same policy as the sibling *Tap repos). # capi stays OFF on Windows: tools/capi carries no __declspec(dllexport) @@ -68,7 +63,6 @@ jobs: os: windows-latest werror: OFF capi: OFF - nonblocking: false steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 with: @@ -94,7 +88,6 @@ jobs: -DCMAKE_BUILD_TYPE=Release -DMUTAP_WERROR=${{ matrix.werror }} -DMUTAP_BUILD_CAPI=${{ matrix.capi }} - ${{ matrix.extra_cmake }} # Record the backend actually built, so a log can never be misread about # which float32 configuration produced its numbers. @@ -108,15 +101,18 @@ jobs: - name: Test run: ctest --test-dir build -C Release --output-on-failure - # A single pass cannot certify a per-process bimodal outcome: on vDSP the - # float32 rows sample a ~72/28 draw. Repeat the implicated rows on this - # host so the log records how often the draw bites, rather than one - # sample of it. Non-gating by virtue of the job's continue-on-error. - # always(), because the whole point of this step is to characterize a - # failure — and a failing Test step above would otherwise skip it, which - # is exactly what happened on the first run that needed it. - - name: Repeat the float32 rows (vDSP draw rate) - if: always() && matrix.nonblocking + # These three rows were the per-process bifurcation in #31. Now that the + # alignment fix has removed the draw, repeating them says something + # different but still worth recording: whether the remaining failures are + # STABLE. A row that fails 20/20 is a deterministic property of the + # backend difference; one that still wanders means a second source of + # variation survives and the diagnosis is incomplete. + # + # always(), because the whole point is to characterize a failure — and a + # failing Test step above would otherwise skip it, which is exactly what + # happened on the first run that needed it. + - name: Repeat the float32 rows (stability, not a draw) + if: always() && runner.os == 'macOS' run: > ctest --test-dir build -C Release --output-on-failure --repeat until-fail:20 diff --git a/docs/itu-compliance.md b/docs/itu-compliance.md index 7005402..e045f3c 100644 --- a/docs/itu-compliance.md +++ b/docs/itu-compliance.md @@ -662,32 +662,54 @@ configuration. The certified numbers above are measured on **Ooura** deployment target that ships today. Both are deterministic across processes. -**Apple/vDSP float32 is NOT currently certified**, and its rows should be -read as pending rather than passing. `TAP_DSP_FFT_ACCELERATE` defaults ON -for Apple, so the macOS host leg runs the battery against Apple's vDSP — -and `vDSP_fft_zrip` at N=2048 has been measured returning one of two -bit-exact outputs for identical input, drawn once per process -(145/55 over 200 processes on macOS 26.5.2 / Xcode 26.6 / Apple M1; not -reproduced on macOS 15.7.7 / AppleClang 17 / Intel, 200/200 identical at -every size 64–8192). The residual suppressor's N=2048 analysis geometry -sits on that draw, and the chain amplifies it into a ~97 dB difference in -the G.168 §7 tone row. Tracked in -[tap/MuTap#31](https://github.com/tap/MuTap/issues/31). +**Apple/vDSP float32 is NOT currently certified.** Two rows fail there, and +the macOS CI leg gates on vDSP — the backend that ships on Apple — so this +is visible rather than filed away: + +| row | status on Apple/vDSP | +|---|---| +| `g168_adapted.ToneStability` | **fails** | +| `Float32Parity.ToneRowWithNarrowbandGuard` | **fails** | +| everything else in the battery | passes | + +Both are driven by an **on-bin tone**. The row driven by broadband material +(`itu_echo.EchoStability`) passes. That distinction is the whole +finding, and it is not a backend defect: + +- vDSP's per-process nondeterminism — the original symptom, one of two + bit-exact outputs per process at N=2048 and N=4096 — was root-caused to + vDSP dispatching on 64-byte buffer alignment, and **fixed** in + [tap/DspTap#9](https://github.com/tap/DspTap/pull/9). Those two outputs + differed by 2.16e-07 peak-normalized, i.e. they were **within** the + documented 4e-7 agreement bound of each other the entire time. +- What remains is that the chain converts a within-contract backend + difference into a compliance-scale swing. On an on-bin tone nearly every + bin holds nothing but float32 rounding noise, and measured against a + double-precision reference on that material, per-bin relative error + exceeds 1e6 for **both** backends — with Ooura roughly 4× *less* accurate + than vDSP. So a row whose outcome depends on those bins depends on + rounding noise, and switching backends does not fix it; it only changes + which noise you get. + +Tracked in [tap/MuTap#31](https://github.com/tap/MuTap/issues/31). These +rows return to certified status when the chain no longer takes its answer +from empty bins — not when a backend is swapped. Two consequences worth stating plainly, because the battery reported this correctly for two weeks while it was read as CI flake: -- A row that is run **once** per leg cannot certify a bimodal outcome. On - the vDSP leg the float32 rows sample a ~72/28 draw, so "green" there is - a sample, not a gate. Repetition (`--repeat until-fail:N`) is what makes - a float32 result evidence, and it is now applied on that leg. +- A row that is run **once** per leg cannot certify a bimodal outcome. While + the alignment bug was live, the float32 rows on vDSP sampled a ~70/30 + draw, so a green there was a sample rather than a gate — which is how a + genuine compliance failure read as CI flake for two weeks. Repetition + (`--repeat until-fail:N`) is what makes a float32 result evidence, and it + is applied on the macOS leg. - The argument this section makes for the precision axis — *if "double passes ⇒ float passes" were sound, the tone row could not exist* — transposes onto backends unchanged. float32-on-Ooura does not certify - float32-on-vDSP any more than double certifies float. - -The Apple float32 rows return to certified status when #31 resolves and -the battery clears them under repetition on that backend. + float32-on-vDSP any more than double certifies float. That is why the + backend is named for every number above, and why the macOS gate points at + the backend that ships rather than the one that is convenient. Why both, rather than certify in double and infer float32: the two are not related by a uniform noise floor. The G.168 §7 tone row is the From 6367135887fa1d36698fbfb6969299be9c1eed90 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Tue, 11 Aug 2026 11:49:01 -0500 Subject: [PATCH 5/6] Use Ooura for float32 on Apple, not vDSP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DspTap defaults vDSP ON for Apple; MuTap now turns it back off. Two measured reasons, both from #31. ACCURACY. vDSP dispatches on buffer alignment, and the kernel selected by 64-byte-aligned split buffers is far less accurate on spectra with exactly-empty bins — which is what the G.168 tone row drives the chain with. Median per-bin relative error vs a double reference, Apple M1, N=2048: material vDSP 64-aligned vDSP not-aligned Ooura broadband 1.6e-07 1.2e-07 1.2e-07 tone off-bin 1.3e-06 1.3e-06 6.4e-07 tone on-bin 0.65 1.2e-07 1.1e-07 The gap appears only on exactly-on-bin excitation; any leakage that lifts the empty bins above the noise floor hides it, which is why the peak-normalized fft_backend_parity gate never saw it. Confirmed through the wrapper as compiled (not just the probe driving vDSP directly) and on a second on-bin frequency, with broadband and off-bin material as controls that show no gap. CONTRACT. Apple's own vDSP.h states the routines are "free to rearrange calculations for better performance", that "rounding errors will often be different when operations are rearranged", and that they are "not expected to conform to IEEE 754". Which kernel runs was observed to depend on buffer alignment; nothing documented prevents that changing again. A certified compliance claim cannot rest on it. Note this also rules out the tempting fix of forcing the OTHER alignment: it works today and depends on undocumented dispatch to keep working. This corrects the reading in the previous commit on this branch, which gated macOS on vDSP and described the tone-row failures as the chain being fragile to a within-contract difference. The A/B settled it the other way: the rows pass on the non-64-aligned kernel and on Ooura, so they were correctly refusing a degraded spectrum rather than over-reacting to legitimate rounding. Set as a non-FORCE cache entry, so -DTAP_DSP_FFT_ACCELERATE=ON still wins for anyone measuring it. Not proposed as a DspTap change: vDSP remains the right default there for consumers who want the ~3x and can tolerate the latitude. The macOS CI leg now gates on the product's own default rather than overriding it, so the certified configuration and the built one cannot drift apart. Verified locally (macOS 15.7.7, Intel): default configure selects Ooura, -DTAP_DSP_FFT_ACCELERATE=ON still selects vDSP, and the three rows implicated in #31 pass 3/3 repetitions each on the new default. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- .github/workflows/ci.yml | 27 +++----------- CMakeLists.txt | 30 ++++++++++++++- docs/itu-compliance.md | 80 +++++++++++++++++++++------------------- 3 files changed, 77 insertions(+), 60 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c90781e..0fbdcaa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -28,27 +28,12 @@ jobs: cxx: clang++ werror: ON capi: ON - # macOS GATES ON vDSP — the float32 backend that actually ships on - # Apple, and therefore the one whose failures are the ones that - # matter. It is the platform default, so no flag is needed. - # - # Expect this leg RED until the tonal fragility in tap/MuTap#31 is - # fixed: g168_adapted.ToneStability and - # Float32Parity.ToneRowWithNarrowbandGuard fail here, consistently - # (both rows, both runs, since the DspTap alignment fix landed). - # That is deliberate. The failure is real — the shipped Apple - # float32 configuration does not currently meet two certified rows — - # and gating on the backend nobody ships in order to see green would - # be reporting on a configuration that is not the product. - # - # NOT the old flake. #31's per-process nondeterminism was - # root-caused to vDSP dispatching on 64-byte buffer alignment and - # fixed in DspTap (pinned here); what remains is the chain - # amplifying a WITHIN-CONTRACT backend difference (the two former - # draws differed by 2.16e-07 peak-normalized, inside the documented - # 4e-7) into a compliance-scale swing on tonal material. No backend - # choice fixes that: measured against a double reference on tonal - # material, Ooura is ~4x LESS accurate than vDSP. + # macOS gates on the float32 backend MuTap actually uses, which is + # OOURA — the root CMakeLists turns DspTap's Apple vDSP default back + # off, with the measurements behind that decision recorded there and + # in docs/itu-compliance.md (tap/MuTap#31). No flag needed here: the + # gate follows the product's default rather than overriding it, so + # this leg cannot drift away from what ships. - name: macOS AppleClang os: macos-latest werror: ON diff --git a/CMakeLists.txt b/CMakeLists.txt index cccdd94..0c15d65 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -6,10 +6,36 @@ project(MuTap VERSION 0.1.0 LANGUAGES C CXX) # Apple vDSP float32 backends, formerly vendored here (include/mutap/fft.h + # third_party/{ooura,cmsis-dsp}). Now consumed from the tap/dsptap submodule as # tap::dsp, which carries the same numeric contract and selects the per-target -# backend itself (CMSIS on the bare-metal M55, vDSP on Apple, Ooura elsewhere). -# Override with -DTAP_DSP_FFT_CMSIS=OFF / -DTAP_DSP_FFT_ACCELERATE=OFF. +# backend itself (CMSIS on the bare-metal M55, Ooura elsewhere). +# Override with -DTAP_DSP_FFT_CMSIS=OFF / -DTAP_DSP_FFT_ACCELERATE=ON. +# +# APPLE float32 USES OOURA, NOT vDSP — DspTap defaults vDSP ON for Apple and +# MuTap deliberately turns it back off. Two measured reasons (tap/MuTap#31): +# +# 1. Accuracy. On spectra with exactly-empty bins — an on-bin tone, which is +# what the G.168 tone rows drive the chain with — the vDSP kernel selected +# by 64-byte buffer alignment is far less accurate than Ooura. Measured +# against a double-precision reference on Apple M1, MEDIAN per-bin relative +# error at N=2048: vDSP 0.65, Ooura 1.1e-07. Broadband and off-bin material +# show no such gap, which is why the peak-normalized parity gate never saw +# it. The compliance tone rows fail on that kernel and pass on Ooura. +# +# 2. Contract. Apple's own vDSP.h states the routines are "free to rearrange +# calculations for better performance", that "rounding errors will often be +# different when operations are rearranged", and that they are "not expected +# to conform to IEEE 754". Which kernel runs has been observed to depend on +# buffer alignment, and nothing documented prevents that from changing +# again. A certified compliance claim cannot rest on it. +# +# This is a MuTap policy choice, not a DspTap defect: vDSP remains the right +# default for consumers who want the ~3x and can tolerate that latitude. Setting +# it here (not FORCE) so an explicit -DTAP_DSP_FFT_ACCELERATE=ON still wins for +# anyone who wants to measure it. # ------------------------------------------------------------------------------ +set(TAP_DSP_FFT_ACCELERATE OFF CACHE BOOL + "Route the float32 FFT through Apple vDSP (MuTap defaults this OFF; see #31)") + add_subdirectory(submodules/dsptap) # ------------------------------------------------------------------------------ diff --git a/docs/itu-compliance.md b/docs/itu-compliance.md index e045f3c..b58a5d9 100644 --- a/docs/itu-compliance.md +++ b/docs/itu-compliance.md @@ -657,43 +657,49 @@ all green on the host CI legs) — with one backend carved out below. Precision is not the only axis the float32 path varies over: the float32 real FFT has three backends, and "float32" alone does not name a configuration. The certified numbers above are measured on **Ooura** -(Linux GCC, Linux Clang, Windows MSVC) and on **CMSIS Helium** and its -**Ooura fallback** on the Cortex-M55 leg — which together cover every -deployment target that ships today. Both are deterministic across -processes. - -**Apple/vDSP float32 is NOT currently certified.** Two rows fail there, and -the macOS CI leg gates on vDSP — the backend that ships on Apple — so this -is visible rather than filed away: - -| row | status on Apple/vDSP | -|---|---| -| `g168_adapted.ToneStability` | **fails** | -| `Float32Parity.ToneRowWithNarrowbandGuard` | **fails** | -| everything else in the battery | passes | - -Both are driven by an **on-bin tone**. The row driven by broadband material -(`itu_echo.EchoStability`) passes. That distinction is the whole -finding, and it is not a backend defect: - -- vDSP's per-process nondeterminism — the original symptom, one of two - bit-exact outputs per process at N=2048 and N=4096 — was root-caused to - vDSP dispatching on 64-byte buffer alignment, and **fixed** in - [tap/DspTap#9](https://github.com/tap/DspTap/pull/9). Those two outputs - differed by 2.16e-07 peak-normalized, i.e. they were **within** the - documented 4e-7 agreement bound of each other the entire time. -- What remains is that the chain converts a within-contract backend - difference into a compliance-scale swing. On an on-bin tone nearly every - bin holds nothing but float32 rounding noise, and measured against a - double-precision reference on that material, per-bin relative error - exceeds 1e6 for **both** backends — with Ooura roughly 4× *less* accurate - than vDSP. So a row whose outcome depends on those bins depends on - rounding noise, and switching backends does not fix it; it only changes - which noise you get. - -Tracked in [tap/MuTap#31](https://github.com/tap/MuTap/issues/31). These -rows return to certified status when the chain no longer takes its answer -from empty bins — not when a backend is swapped. +(Linux GCC, Linux Clang, Windows MSVC, **and macOS on both architectures** — +see below) and on **CMSIS Helium** and its **Ooura fallback** on the +Cortex-M55 leg — which together cover every deployment target that ships +today. All are deterministic across processes, which is the property that +makes a single run of a row count as evidence at all. + +**On Apple, MuTap uses Ooura for float32, not vDSP** — DspTap defaults vDSP +ON for Apple and MuTap's root `CMakeLists.txt` turns it back off. The +certified Apple numbers are therefore Ooura numbers, and the macOS CI leg +gates on that same default, so the certified configuration and the built +one cannot drift apart. + +Two measured reasons, both from [tap/MuTap#31](https://github.com/tap/MuTap/issues/31): + +**Accuracy on sparse spectra.** vDSP dispatches on buffer alignment, and +the kernel selected by 64-byte-aligned split-complex buffers is far less +accurate on spectra with exactly-empty bins — which is precisely what the +G.168 §7 tone row drives the chain with. Median per-bin relative error +against a double-precision reference, Apple M1: + +| material | vDSP (64-aligned) | vDSP (not aligned) | Ooura | +|---|---|---|---| +| broadband | 1.6e-07 | 1.2e-07 | 1.2e-07 | +| tone, off-bin | 1.3e-06 | 1.3e-06 | 6.4e-07 | +| **tone, on-bin** | **0.65** | 1.2e-07 | 1.1e-07 | + +The gap appears only on exactly-on-bin excitation; any leakage that lifts +the empty bins above the noise floor hides it, which is why the +peak-normalized `fft_backend_parity` gate never saw it. On that kernel +`g168_adapted.ToneStability` and +`Float32Parity.ToneRowWithNarrowbandGuard` fail; on Ooura they pass. + +**Contract.** Apple's `vDSP.h` states the routines are "free to rearrange +calculations for better performance", that "rounding errors will often be +different when operations are rearranged", and that they are "not expected +to conform to IEEE 754". Which kernel runs was observed to depend on buffer +alignment, and nothing documented prevents that from changing again. A +certified compliance claim cannot rest on it. + +This is a MuTap policy choice rather than a DspTap defect — vDSP stays the +right default for consumers who want the ~3× and can tolerate that +latitude. Anyone wanting to measure it can still build with +`-DTAP_DSP_FFT_ACCELERATE=ON`. Two consequences worth stating plainly, because the battery reported this correctly for two weeks while it was read as CI flake: From 4e95efcee7bf966f945c58fd3f6a20ec1855088d Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Tue, 11 Aug 2026 12:44:55 -0500 Subject: [PATCH 6/6] Re-pin DspTap to the accurate-kernel placement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps submodules/dsptap to ff6fc6c (tap/DspTap#10), which places the vDSP split buffers 32 bytes past a 64-byte boundary so the accurate kernel is selected, and adds fft_tonal_accuracy to assert that property rather than trust the constant. MuTap builds TAP_DSP_FFT_ACCELERATE=OFF, so this changes nothing about what MuTap computes — the float32 path here is Ooura either way. It is worth pinning regardless: the previous pin (da4dd68) forced the LESS accurate kernel on any consumer that does use vDSP, and leaving a submodule pinned at a known-worse revision is a trap for whoever unpins it next. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh --- submodules/dsptap | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submodules/dsptap b/submodules/dsptap index da4dd68..ff6fc6c 160000 --- a/submodules/dsptap +++ b/submodules/dsptap @@ -1 +1 @@ -Subproject commit da4dd68bab9ec73ba3ad9e1ce4048b9d8fd50d3d +Subproject commit ff6fc6c5067f675316fd07a05e9c078dbc23b1bd