diag: the float32 FFT probe that found #31's root cause - #41
Open
tap wants to merge 6 commits into
Open
Conversation
…nd why #31 established the mechanism — vDSP_fft_zrip at N=2048 returns one of two bit-exact outputs for identical input, drawn once per process — but left three questions open that between them decide what to actually do about it. This adds the measurements, and a dispatchable workflow to run them on the runner where the split occurs. 1a. WHICH SIZES. Only N=512 and N=2048 have ever been probed. The sweep now covers every power of two from 64 to 8192, so a consumer can tell whether its geometry is exposed at all — the suppressor's N=2048 is not the only geometry in the family. 1b. HOW FAR APART. The draws were hashed but never compared, and this is the fork in the road. A gap >= 1e-4 relative means vDSP violates its own documented 4e-7 contract and this is Apple's bug, fixed by turning the backend off. A gap at ~1 ULP means the contract holds and OUR chain bifurcates on a legitimate backend difference — which no threshold change addresses and which would recur on any future toolchain. The diff mode reports per-bin relative error and classifies the result against both boundaries. Note this is in tension with the perturbation study earlier in #31, which found the failing row survives 1e-5 injection and needs 1e-4 to break; that tension is exactly what the measurement resolves. 1c. WHAT SELECTS THE DRAW. The cross-process sweep cannot separate "fixed at process start" from "follows the heap address" — both are constant within a process. The align mode moves the split-complex buffers and the interleaved input to deliberate byte offsets within ONE process, holding the FFTSetup fixed, and then holds the buffers fixed while recreating the setup. If the hash tracks the offset, over-aligning m_rp/m_ip in DspTap's wrapper fixes this while keeping vDSP and its ~3x speedup — the only candidate outcome that costs nothing. The workflow is macOS-only and deliberately separate from ci.yml: it is temporary, it should be dispatchable on demand, and deleting it when #31 closes should be one file rather than a surgical edit. Validated locally on macOS 15.7.7 / AppleClang 17 / Intel i9, where the probe reports vDSP DETERMINISTIC — 200/200 identical at every size 64-8192, including 2048, with the align probe showing no movement across ten buffer offsets or eight fresh setups. That is a useful negative: the split is not universal to vDSP, so it is specific to the newer stack (M1 and/or macOS 26 / Xcode 26), and a local Intel Mac cannot substitute for the runner. The align and diff paths were exercised against synthesized draws to confirm they classify a 1-ULP gap and a 1e-3 gap correctly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
… size
Measured on this Intel Mac while validating the probe, and it changes what
the diagnostic should be asking.
The verdict line claimed a max per-bin relative gap of >=1e-4 meant "the
backend violates its contract outright". That was overstated, because the
contract and the chain are stated in DIFFERENT metrics, and on real data
they disagree by ~1700x:
vDSP vs Ooura, N=2048, same host, same input
peak-normalized absolute 1.57e-07 WITHIN the documented 4e-7
per-bin relative 2.68e-04 PAST the 1e-4 at which #31's
perturbation study broke the row
Both numbers describe the same difference. fft_backend_parity asserts the
first (5e-6 * peak) and "agrees to <4e-7 relative" was measured as the
first; the chain is sensitive to the second. So the honest reading is not
that Apple is broken, nor that our chain is spontaneously fragile — it is
that the contract is stated in a metric that does not bound what this
consumer needs. Report both, and say that plainly when they disagree.
This also resolves the tension flagged when the sweep was added: the
perturbation study found the row needs >=1e-4 per-bin relative to break,
and the real vDSP-vs-Ooura difference is 2.68e-04 per-bin relative. The
study's model predicted this correctly; it was being compared against the
wrong metric.
The new 1b' step makes the cross-backend diff a first-class measurement at
every size. Unlike the draw-vs-draw diff it exists on every host, not only
one that bifurcates — on Intel, where vDSP is fully deterministic across
200 processes, the cross-backend gap is still there and still past the
chain's breaking point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
Formatting only, no behaviour change. The file is compiled directly by the diagnostic workflow rather than through CMake, so nothing but the style gate reads it — and the style gate is what caught it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
tap
changed the title
Pins the DspTap alignment fix here, and adds the round-2 experiment that should have been run before the fix was called a fix. The alignment change made vDSP deterministic by forcing every process onto the 64-byte-aligned kernel. Nobody checked whether that is the kernel on which the compliance rows PASS. The 140/60 split was measured on FFT output hashes, not on test outcomes, so "the majority path" says nothing about which path is good. If the tone rows pass on the other one, the fix selected the losing kernel and the current account of #31 is wrong. Three builds of one tree, same host, same run: aligned as shipped unaligned the fix deliberately defeated (+4 bytes past the boundary), so every process takes the non-64-aligned kernel ooura backend control Each runs the two failing tonal rows plus the broadband row that passes, 5 repetitions each. A row that fails 5/5 in one variant and passes 5/5 in another is decisive. A row that wanders WITHIN a variant means a source of variation survived the fix and the diagnosis is incomplete — which is the outcome I most want to know about, because the claim that a ~1.7 ULP FFT difference produces a 97 dB swing currently rests on two post-fix runs and nothing else. The patch that defeats the fix asserts its site appears exactly once. Without that, a failed patch would silently build the same thing twice and report it as a clean A/B — the most plausible way this experiment could lie. Also re-runs the 200-process determinism sweep with the fix pinned. The earlier sweep proved the bifurcation BEFORE the fix; this is the same measurement after it, on the same host, and it is the in-situ check that the wrapper is now invariant rather than merely passing DspTap's own tests. Verified locally: the patch applies (assert passes, one site), the patched tree compiles clean with -Wall -Wextra, and the submodule restores to a clean checkout afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
Round 2 established that the compliance rows PASS on the non-64-aligned kernel and FAIL on the aligned one — which means the alignment fix, by forcing every process onto the aligned path, took the pass rate from ~30% to 0%. That result does not by itself justify flipping the constant. Two kernels that differ by ~1.7 ULP can both be within contract, in which case picking the one whose rounding the gates happen to like is choosing a lucky coin, and the chain is still the thing that is broken. But if the aligned kernel is materially LESS accurate — a real defect in Apple's aligned path — then avoiding it is ordinary engineering. The `truth` mode drives vDSP directly at both alignments within one process and compares each against a double-precision transform of the same input (error ~1e-15 at these sizes, so a legitimate stand-in for truth against float32's ~1e-7). Two materials, since the failing rows are tonal and the existing parity gate is broadband. Locally on Intel both alignments report identical figures, which is the expected null — Intel vDSP does not dispatch on alignment — so this only proves the probe works. The M1 is the measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
…aterials Two checks before acting on the round-2 result, because I have moved early twice on this issue already. 1. THROUGH THE WRAPPER. The earlier accuracy numbers came from the probe driving vDSP directly with its own buffers, so "the aligned kernel is less accurate" could in principle have been an artefact of how the probe places them. The wrapper is now measured as compiled — built once stock (64-aligned) and once with the alignment defeated — which is the code path MuTap actually runs. 2. MORE MATERIALS. The finding rested on one on-bin tone at bin n/16. Now: broadband and an off-bin tone as controls (every bin carries energy, so both kernels should look identical), a second on-bin tone at n/8, and a sparse three-tone spectrum so a result cannot be pinned on the single-tone special case. The prediction, if the round-2 reading is right: the two kernels agree on broadband and off-bin material and diverge by orders of magnitude in the p50 column on the sparse ones. If instead they diverge everywhere, or nowhere, the reading is wrong and the remedy question reopens. p50 is the column that matters and is worth saying why: on sparse material max_rel saturates (a near-empty bin divided into anything is enormous, for every backend) and abs/peak hides the damage (it normalizes by the loudest bin, so an error confined to quiet bins vanishes). The median bin says whether MOST of the spectrum is right, and it is the only one of the three that separated the kernels. Local Intel run: all three rows identical on every material, which is the expected null — Intel vDSP does not dispatch on alignment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Extends
tests/fft_determinism_probe.cppand adds a dispatchable macOS-only workflow that runs it. Diagnostic for #31; both files are marked for deletion when it closes.This has now run and produced answers — they are below, and they located the root cause. The fix is in tap/DspTap#9.
Results, from run 31419956480
Apple M1 (Virtual) / VirtualMac2,1, macOS 26.5.2 (25F84), Apple clang 21.0.0. 200 processes per backend per size.
1a — which sizes bifurcate
N=4096 was not previously known to be affected. The two N=2048 hashes match the ones in #31's original report.
1b — how far apart the draws are
At N=2048: 83.2% of bins differ; max absolute difference 1.31e-06, which is 2.16e-07 peak-normalized — inside the documented 4e-7 bound. So the chain has been bifurcating on a difference the FFT is explicitly permitted to have. That is the serious reading, not the reassuring one.
1c — what selects the draw
rpmod 64591c8a27e6078cd761cb8eac91ae7d41vDSP dispatches on 64-byte alignment, and
std::vector's 16-byte-aligned storage left the choice to the heap. Recreating theFFTSetupwith buffers fixed changes nothing. Root cause; fixed in DspTap#9 by aligning the buffers.Why the probe is shaped this way
The cross-process sweep alone cannot separate "the draw is fixed at process start" from "the draw follows the buffer address" — both are constant within a process. Only moving the buffers inside one process, with the setup held fixed, distinguishes them, which is why
alignexists and why it was the measurement that broke the case open.The
diffmode reports two metrics deliberately, because they disagree by ~1700× on this material and the whole argument in #31 turns on which is meant: peak-normalized absolute (whatfft_backend_parityasserts and what the 4e-7 bound was measured as) and per-bin relative (what the chain is sensitive to, and what #31's perturbation study injected).Verification
alignshowing no movement. A useful negative — the dispatch is specific to the newer stack, so a local Intel Mac cannot substitute for the runner.diffhas no natural case on a deterministic host, so it was exercised against synthesized draws: a 1-ULP perturbation classifies as epsilon (1.19e-07), a 1e-3 perturbation as past the breaking point.align's replication of the wrapper is confirmed faithful by its offset-0 hash matching the wrapper's own output hash exactly.Notes for the reviewer
diag/**andworkflow_dispatch, so deleting it when macOS float32 flake: Apple vDSP's real FFT is non-deterministic across processes at N=2048 #31 closes is one file.dsptappin picks up DspTap#9.🤖 Generated with Claude Code
https://claude.ai/code/session_01FVTJEw3TW1sMS1nq26rGmh