Gate macOS float32 on Ooura, and say which backend certified what - #40
Merged
Conversation
… what The macOS float32 rows have been red on and off since 2026-07-27 and were being read as CI flake. They are not flake: #31 established that vDSP_fft_zrip at N=2048 returns one of two bit-exact outputs for identical input, drawn once per process, and that the residual suppressor amplifies that draw into a ~97 dB swing on the G.168 §7 tone row. The battery was reporting a real property of a shipped configuration, correctly, for two weeks. Two consequences, both handled here. CI. macOS now gates on the Ooura float32 backend, and vDSP moves to a second, non-gating leg that keeps running the same battery plus a --repeat until-fail:20 pass over the three implicated rows, so the draw rate is recorded rather than sampled once. This stops the flake reddening unrelated PRs without hiding it, and without pre-judging the shipping backend decision that #31 still has to make. The gating job deliberately keeps the name "macOS AppleClang" so branch protection referencing that check keeps working. Both macOS legs now record chip identity and the backend actually built, because #31's draw appears on an M1 VM and does not reproduce on Intel — a bare "macOS" label cannot attribute a result. Docs. docs/itu-compliance.md claimed the float32 battery was "all green on the host CI legs" without naming a backend, which reads as covering a configuration it does not. Spell out that the certified float32 numbers come from Ooura (Linux x2, Windows) and CMSIS Helium + its Ooura fallback (M55) — every target that ships today, all deterministic — and carve out Apple/vDSP as NOT currently certified, pending #31. Also record why a single run cannot certify a bimodal outcome, and that the section's own argument for the precision axis ("if double-passes-implies-float-passes were sound, the tone row could not exist") transposes onto backends unchanged. Verified locally: with -DTAP_DSP_FFT_ACCELERATE=OFF the three implicated rows pass 5/5 repeats each on macOS 15.7.7 / Intel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
Bumps submodules/dsptap to da4dd68 (tap/DspTap#9), which root-caused #31: vDSP dispatches on 64-byte buffer alignment and the two paths disagree bit-for-bit, so std::vector's 16-byte-aligned storage left the choice to wherever the heap landed. The wrapper now aligns its split buffers, and DspTap carries fft_alignment_stability as the regression gate. This removes the per-process nondeterminism. It does NOT change the numerics, and the Intel evidence is unambiguous on that point: itu_echo.EchoStability<float>, fs 48000 level -25, macOS 15.7.7 / Intel before the fix amax-amin = 3.27832747136911 vs a 3.0 gate after the fix amax-amin = 3.27832747136911 vs a 3.0 gate Bit-identical. Intel vDSP never dispatched on alignment (200/200 across sizes both before and after), so there was nothing there for the fix to change; that row fails because vDSP and Ooura differ and the chain amplifies the difference, which is the open half of #31. Note this row misses the HOUSE margin of 3.0 dB while still meeting the ITU requirement of 6.0 dB — margin erosion, not a compliance failure, unlike the G.168 tone row's ~97 dB miss on the M1. The macOS gating deliberately stays on Ooura in this commit. With alignment forced, every M1 process now takes what used to be the 140/200 majority path — and nobody has established whether that path is the one that PASSES the compliance rows or the one that fails them. The non-gating vDSP leg plus its repeat step answers that on this push, and the gating decision follows the evidence rather than the other way round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
The repeat step guarded on `matrix.nonblocking` alone, so a failing Test step skipped it — and that is precisely the case it exists to measure. Both macOS vDSP runs on the pinned-fix commit failed at Test and never reached it, losing the repeat data on the one run where it mattered. `always() && matrix.nonblocking` keeps it scoped to the vDSP leg while letting it run after a failure. Worth recording what those two runs already show, since it is a change in the failure signature rather than a repeat of it. With the alignment fix pinned, BOTH runs failed exactly the same two tests: 157 - g168_adapted.ToneStability<float> 179 - Float32Parity.ToneRowWithNarrowbandGuard and itu_echo.EchoStability<float>, which failed on repetition 1 of both earlier M1 runs, now PASSES. Before the fix the failing set wandered run to run; two runs is not proof of determinism, but an identical set twice plus a removed source of per-process variation is what determinism looks like from here, and the repeat step will settle it. Both survivors are the TONAL rows. The row driven by broadband material (CSS) now passes. That is the shape the empty-bin hypothesis predicts: on an on-bin tone nearly every bin holds nothing but float32 rounding noise, and the chain's behaviour depends on which noise it gets — which no backend choice fixes, because Ooura's noise is not more correct than vDSP's, merely different (and measured against a double reference on this material, ~4x less accurate). So the gating stays on Ooura and the compliance doc's "Apple/vDSP not currently certified" carve-out stands. The fix turned an intermittent failure into an honest one; it did not make the shipped configuration pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
Reverses the gating half of the earlier commit on this branch. That change
moved the macOS gate to Ooura to stop a flake reddening unrelated PRs;
with the flake's root cause now fixed, the remaining red is not noise, and
pointing the gate at a backend nobody ships in order to see green would be
certifying a configuration that is not the product.
Expect this leg RED. Two rows fail on Apple/vDSP, consistently — both runs
since the alignment fix pinned:
157 - g168_adapted.ToneStability<float>
179 - Float32Parity.ToneRowWithNarrowbandGuard
itu_echo.EchoStability<float>, which failed on repetition 1 of both earlier
M1 runs, now passes.
This is deliberate, and it is not the old flake:
- The per-process nondeterminism was root-caused to vDSP dispatching on
64-byte buffer alignment and fixed in tap/DspTap#9, pinned here.
- The two former draws differed by 2.16e-07 peak-normalized — INSIDE the
documented 4e-7 bound. So what remains is the chain converting a
within-contract backend difference into a compliance-scale swing.
- Both surviving failures are TONAL rows; the broadband-driven row
passes. On an on-bin tone nearly every bin holds only float32 rounding
noise, and against a double reference on that material per-bin relative
error exceeds 1e6 for BOTH backends, with Ooura ~4x LESS accurate than
vDSP. Switching backends changes which noise the chain reads, not
whether it reads noise.
main is unprotected, so a red leg is advisory and does not block merges —
the reason it can be honest now rather than after the fix.
The repeat step stays, with its purpose restated: it no longer measures a
draw rate but whether the remaining failures are STABLE. A row failing
20/20 is a deterministic property; one that still wanders means a second
source of variation survives and the diagnosis is incomplete.
docs/itu-compliance.md now names the two failing rows, says why the
distinction between tonal and broadband material is the finding rather than
an aside, and states that they return to certified status when the chain
stops taking its answer from empty bins — not when a backend is swapped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
DspTap defaults vDSP ON for Apple; MuTap now turns it back off. Two measured reasons, both from #31. ACCURACY. vDSP dispatches on buffer alignment, and the kernel selected by 64-byte-aligned split buffers is far less accurate on spectra with exactly-empty bins — which is what the G.168 tone row drives the chain with. Median per-bin relative error vs a double reference, Apple M1, N=2048: material vDSP 64-aligned vDSP not-aligned Ooura broadband 1.6e-07 1.2e-07 1.2e-07 tone off-bin 1.3e-06 1.3e-06 6.4e-07 tone on-bin 0.65 1.2e-07 1.1e-07 The gap appears only on exactly-on-bin excitation; any leakage that lifts the empty bins above the noise floor hides it, which is why the peak-normalized fft_backend_parity gate never saw it. Confirmed through the wrapper as compiled (not just the probe driving vDSP directly) and on a second on-bin frequency, with broadband and off-bin material as controls that show no gap. CONTRACT. Apple's own vDSP.h states the routines are "free to rearrange calculations for better performance", that "rounding errors will often be different when operations are rearranged", and that they are "not expected to conform to IEEE 754". Which kernel runs was observed to depend on buffer alignment; nothing documented prevents that changing again. A certified compliance claim cannot rest on it. Note this also rules out the tempting fix of forcing the OTHER alignment: it works today and depends on undocumented dispatch to keep working. This corrects the reading in the previous commit on this branch, which gated macOS on vDSP and described the tone-row failures as the chain being fragile to a within-contract difference. The A/B settled it the other way: the rows pass on the non-64-aligned kernel and on Ooura, so they were correctly refusing a degraded spectrum rather than over-reacting to legitimate rounding. Set as a non-FORCE cache entry, so -DTAP_DSP_FFT_ACCELERATE=ON still wins for anyone measuring it. Not proposed as a DspTap change: vDSP remains the right default there for consumers who want the ~3x and can tolerate the latitude. The macOS CI leg now gates on the product's own default rather than overriding it, so the certified configuration and the built one cannot drift apart. Verified locally (macOS 15.7.7, Intel): default configure selects Ooura, -DTAP_DSP_FFT_ACCELERATE=ON still selects vDSP, and the three rows implicated in #31 pass 3/3 repetitions each on the new default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
Bumps submodules/dsptap to ff6fc6c (tap/DspTap#10), which places the vDSP split buffers 32 bytes past a 64-byte boundary so the accurate kernel is selected, and adds fft_tonal_accuracy to assert that property rather than trust the constant. MuTap builds TAP_DSP_FFT_ACCELERATE=OFF, so this changes nothing about what MuTap computes — the float32 path here is Ooura either way. It is worth pinning regardless: the previous pin (da4dd68) forced the LESS accurate kernel on any consumer that does use vDSP, and leaving a submodule pinned at a known-worse revision is a trap for whoever unpins it next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
CI: macOS gates on the Ooura float32 backend; vDSP moves to a second, non-gating leg that keeps running the battery plus a
--repeat until-fail:20pass over the three implicated rows. Both macOS legs now record chip identity and the backend actually built.Docs:
docs/itu-compliance.mdnow names the FFT backend behind each float32 leg, and carves Apple/vDSP out as not currently certified. Part of #31.Why
The macOS float32 rows have been red on and off since 2026-07-27 and were being read as CI flake. They are not flake. #31 established that
vDSP_fft_zripat N=2048 returns one of two bit-exact outputs for identical input, drawn once per process, and that the residual suppressor amplifies that draw into a ~97 dB swing on the G.168 §7 tone row. The battery was reporting a real property of a shipped configuration, correctly, for two weeks.That has two consequences, and this PR handles both.
The gate. A row run once per leg cannot certify a bimodal outcome — on vDSP the float32 rows sample a ~72/28 draw, so "green" there is a sample, not a gate. Gating on Ooura stops the flake reddening unrelated PRs without hiding it, and without pre-judging the shipping-backend decision #31 still has to make. The vDSP leg keeps running and now measures the draw rate instead of sampling it once.
The claim. The compliance doc said the float32 battery was "all green on the host CI legs" without naming a backend, which reads as covering a configuration it does not. The certified float32 numbers come from Ooura (Linux ×2, Windows) and CMSIS Helium + its Ooura fallback (M55) — every target that ships today, all deterministic. Apple/vDSP is now explicitly pending.
The doc also now records that this section's own argument for the precision axis — if "double passes ⇒ float passes" were sound, the tone row could not exist — transposes onto backends unchanged. float32-on-Ooura does not certify float32-on-vDSP.
Verification
-DTAP_DSP_FFT_ACCELERATE=OFFand confirmed the cache readsOFF, i.e. the gating leg really does switch backend rather than silently staying on vDSP.--repeat until-fail:5: 3/3 tests, 15 runs, all pass (macOS 15.7.7, Intel, Release).nonblockingtrue only on the vDSP leg,extra_cmakeonly on the gating macOS leg).continue-on-errorbehaviour and the repeat step only exercise on a real runner — CI on this PR is their first gate.Notes for the reviewer
macOS AppleClangso branch protection referencing it keeps working; the new leg ismacOS AppleClang (vDSP float32, non-gating). If you would rather the gating leg say "Ooura" in its name, that is a branch-protection edit, not just a rename.🤖 Generated with Claude Code
https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh