feat(prover): chunk the accelerator tables and project them for storage - #919
Open
Oppen wants to merge 2 commits into
Open
feat(prover): chunk the accelerator tables and project them for storage#919Oppen wants to merge 2 commits into
Oppen wants to merge 2 commits into
Conversation
KECCAK, KECCAK_RND, ECSM, ECDAS, HINT and COMMIT had no max_rows entry: each one padded its whole op list into a single table, so a keccak- or ECSM-heavy epoch built one table whose height is proportional to guest data. At epoch_size_log2=23 that is a single multi-GB allocation no storage mode can stream. KECCAK, KECCAK_RND, ECSM and ECDAS were also absent from auto_storage's projection entirely, so the Ram/Disk decision under-projected exactly the programs most likely to need Disk. Every one of these chips evaluates row-locally (no `main(1, ..)` reference), so their rows split across tables the way the core chips' do, and the buses they drive are a multiset argument that does not care which table a row sits in. KECCAK_RND splits on whole permutations, since one call is 24 contiguous rows. The limits follow the existing effective-width model, and `accelerator_max_rows_track_effective_width` pins each width to the AIR so a new column or bus cannot leave a limit stale. Breaking: the six tables move out of FIXED_TABLE_COUNT (11 -> 5) into TableCounts, which changes the sub-proof layout and the statement absorbed into the transcript (tags bumped to V4/V3). Prover and verifier must be deployed together; earlier binaries cannot verify these proofs.
…fied - Drift tests only ran guests with zero keccak/ECSM calls, so the new projection formulas were exercised at count 0 only. `count_table_lengths_matches_keccak_trace` runs a three-permutation guest, making KECCAK and KECCAK_RND non-empty. - The chunking test stopped at trace shape; a split table's buses only matter once proved. It now proves and verifies the multi-chunk trace. - The ECDAS per-call bound backs a storage projection, so exceeding it must fail in release too: debug_assert -> assert with the offending count. - Revert CONTINUATION_GLOBAL_TAG to V2: the global statement absorbs no table counts, so this change does not alter it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The six data-proportional accelerator chips — KECCAK, KECCAK_RND, ECSM, ECDAS, HINT,
COMMIT — get
max_rowslimits and chunking, and all of them are projected inauto_storage.Why
Split out of the #874 review, where the DMA table was flagged for having no
max_rowsentry. The gap is the whole accelerator family, not DMA:
n.next_power_of_two().max(4)),so table height is proportional to guest data with no ceiling. At
epoch_size_log2 = 23a keccak- or ECSM-heavy epoch is a single multi-GB table — and a single table is the one
thing no storage mode can stream:
disk-spillbounds memory per chunk, and there wasonly ever one chunk.
auto_storage::table_specsatall, so the Ram/Disk estimate under-projected precisely the programs most likely to
need Disk. (COMMIT was already projected.)
Why chunking is sound here
Every one of these
ConstraintSets is row-local — none referencesmain(1, ..)— so noconstraint spans a chunk boundary, and the buses they drive are a LogUp multiset argument
that is indifferent to which table a row sits in. KECCAK_RND splits on whole permutations:
one call is 24 contiguous rows (
ROUNDS_PER_OP), and its limit divides by that.Limits
Same effective-width model as the core chips (
main_cols + 3 × buses, MEMW's 127 @ 2^19 asthe baseline):
accelerator_max_rows_track_effective_widthpins each width to the AIR, so adding a columnor a bus to one of these chips fails a test instead of silently leaving its limit stale.
MaxRowsConfig::small()deliberately keeps the production values for these six: shrinkingthem to 2^5 splits a committed output or one ECSM ladder into dozens of sub-proofs and
slows every test that uses it. The chunking test shrinks them itself.
Projection
TableLengthsgainskeccak/keccak_rnd/ecsm/ecdas/hintpadded-row counts;count_table_lengthscounts the ecalls andtable_specsincludes all of them plusKECCAK_RC. ECDAS is an upper bound (
ecdas::MAX_STEPS_PER_ECSM, ≤2 ladder steps per scalarbit), in the same spirit as the existing LT/MUL/DVRM/BRANCH bounds — the drift test asserts
>=for it and exact equality for the other four.Breaking
The six tables move out of
FIXED_TABLE_COUNT(11 → 5) intoTableCounts. That changesthe sub-proof layout and the statement absorbed into the transcript, so the domain tags are
bumped (
STATEMENT_V3→V4, continuation epoch/globalV2→V3). Prover and verifiermust ship together; earlier binaries cannot verify these proofs. #874 and #876 also touch
FIXED_TABLE_COUNT, so whichever lands second rebases.Validation
make test-prover: 560 passed, 0 failed, 24 ignored.make lint: clean (all four clippy passes, includingdebug-checks,disk-spill,cuda).accelerator_chunking_tests: effective widths pinned; a 3-call keccak guest at onecall per chunk produces one KECCAK and one KECCAK_RND chunk per call, each padded to 32
rows; default limits keep a small program at one chunk per accelerator.
make testadditionally fails on this machine in-p executor --libwithdyld: symbol not found in flat namespace '__end'— a link/env failure in a crate thisPR does not touch (running the prebuilt
executortest binary directly reproduces it,and the binary contains no prover code). Not investigated here.
Not in scope
Epoch sizing is still cycle-count-only (
resume_with_limit), so an epoch's row budget canstill be blown with no cycle-count change. Bounding that needs the executor to stop an
epoch on a weighted row budget, which breaks the "intermediate epoch = exactly 2^k cycles"
invariant (
continuation.rs:1327) — deliberately left out.