The R2 serialize lock added in #914 suppresses a real, reproduced corruption: with the lock off (LAMBDA_VM_GPU_SERIALIZE_R2=0), real-block proves (block 25368371, epoch 2^22, 32 GB RTX 5090, 5 GB VRAM ballast) intermittently complete at full speed and fail verification with "Composition Polynomial verification failed" on one random table — no OOM logged, nothing in the prover output. Observed 2 events in ~65 lock-off completions; 51/51 clean with the lock on; lock cost measured ≈0 on this workload.
What is established about the mechanism:
- NOT retained-pool reuse: corruption persists with
LAMBDA_VM_MEMPOOL_RELEASE_MB=0, so the reuse window is cuMemAllocAsync's short-range stream-ordered recycling.
- The R2 device-only chain (eval → decompose → slab LDE) is single-stream per table with stream-ordered temp frees — the race must involve the pool's cross-stream reuse machinery against a reader the driver has no dependency for. The one flagged unordered cross-stream read is the LogUp aux build's
trace_dev access (ordered only by a pageable-D2H fallback, crypto/math-cuda/src/lde.rs ~608).
- Per-slice event tracking is not a viable fix (proves blow a 300 s timeout vs a 67 s baseline when re-enabled).
- Reproduction is box-sensitive: a third, identically-specced box gave 0 events in 69 lock-off completions with an identical failure mix.
Suggested continuation, in value order:
- Run the reproducing regime with the
LAMBDA_VM_GPU_XCHECK=1 diagnostics from the round-4 residency branch — the per-table in-prover check plus post-mortem names the corrupted stage on the first caught event, replacing arm statistics entirely.
- On an 80 GB card: rerun with the pool cross-stream-reuse kill switch (attributes
REUSE_ALLOW_OPPORTUNISTIC / REUSE_ALLOW_INTERNAL_DEPENDENCIES / REUSE_FOLLOW_EVENT_DEPENDENCIES zeroed — the patch exists; it does not fit in 32 GB, zero proves complete).
- With the site named, replace the global lock with a targeted drain/event and drop
LAMBDA_VM_GPU_SERIALIZE_R2.
A corrupt proof blob, the full run matrix, the harness, and both experiment patches are archived (see the #914 review thread for the summary).
The R2 serialize lock added in #914 suppresses a real, reproduced corruption: with the lock off (
LAMBDA_VM_GPU_SERIALIZE_R2=0), real-block proves (block 25368371, epoch 2^22, 32 GB RTX 5090, 5 GB VRAM ballast) intermittently complete at full speed and fail verification with "Composition Polynomial verification failed" on one random table — no OOM logged, nothing in the prover output. Observed 2 events in ~65 lock-off completions; 51/51 clean with the lock on; lock cost measured ≈0 on this workload.What is established about the mechanism:
LAMBDA_VM_MEMPOOL_RELEASE_MB=0, so the reuse window is cuMemAllocAsync's short-range stream-ordered recycling.trace_devaccess (ordered only by a pageable-D2H fallback,crypto/math-cuda/src/lde.rs~608).Suggested continuation, in value order:
LAMBDA_VM_GPU_XCHECK=1diagnostics from the round-4 residency branch — the per-table in-prover check plus post-mortem names the corrupted stage on the first caught event, replacing arm statistics entirely.REUSE_ALLOW_OPPORTUNISTIC/REUSE_ALLOW_INTERNAL_DEPENDENCIES/REUSE_FOLLOW_EVENT_DEPENDENCIESzeroed — the patch exists; it does not fit in 32 GB, zero proves complete).LAMBDA_VM_GPU_SERIALIZE_R2.A corrupt proof blob, the full run matrix, the harness, and both experiment patches are archived (see the #914 review thread for the summary).