refactor: delete the batched proof format, per-table only - #973
Merged
Conversation
The batched multi-Merkle-tree proof format is retired, and the device MMCS existed only for it: `MmcsGroupHasher` was the device twin of the host's `StreamingMmcsBuilder`, and `build_mmcs_tree_on_device` built the one tree per epoch that the per-table prover does not have. Removed: `src/mmcs.rs`, its module declaration, the five backend handles and their loads in `src/device.rs`, the five kernels in `kernels/keccak.cu` (`mmcs_states_init`, `mmcs_absorb_row_pair_row_major`, `mmcs_absorb_row_pair_ext3_slabs`, `mmcs_states_finalize`, `keccak_mmcs_level`), and `tests/mmcs_tree_parity.rs`, whose subject was device-vs-host root parity for that tree. `kernels/blake3.cu` had no MMCS twin to remove. `merkle_gather_paths` and `keccak_launch_cfg` are hash- and format-agnostic and stay: the per-table path is their caller.
…own module The batched proof format is retired, so every test whose SUBJECT is that format goes. Deletions, by file: - `aggregator_tests.rs` (2,664 lines, 10 tests) — the module is typed end to end on `BatchedMultiProof`. Its GLOBAL leg is not: the cross-epoch global memory proof is per-table under either format, so `RealGlobal`, `real_global`, `GlobalTableArenas`, `global_verifier_program`, `global_arena_words` and `the_global_verifier_leg_runs_and_rejects_tampers` move verbatim into `per_table_aggregator_tests.rs`. One substitution: the fixture bundle now comes from `prove_continuation` instead of `prove_continuation_batched` — identical signature, and `real_global` already took a plain `ContinuationProof`. - `group_leaf_tests.rs` (295 lines, 3 tests) — the mixed-height group leaf. - `epoch_tests.rs` — the batched harness and its 7 tests, plus `the_real_block_base_epoch_ab`, whose subject was per-table VS batched and which cannot be a one-armed A/B. That test had absorbed `real_epoch_with`'s doc block (no blank line between them); deleting it returns the doc to `real_epoch_with`. - `wrap_tests.rs` — the batched wrap section, its 3 tests and the P2 batched block driver. That driver's doc had likewise absorbed `the_real_blocks_page_census`'s; the census doc is reattached. - `epoch_verify_tests.rs` — the batched leg-input builders. - `machine_tests.rs` — the three batched LFM epoch/wire tests. - `algebraic_commit.rs` — `the_mixed_group_leaf_equals_the_hosts_group_hash`. A group leaf over several matrices is the mixed-height construction; the one-matrix case it degenerates to is `the_emitted_leaf_and_parent_equal_the_hosts_backend`'s subject and stays gated. `StarkHash::Batched` is the batch-commitment leaf backend, not the format, and is untouched. Two tests are EDITED rather than deleted, because their per-table half is the claim and the batched half was an arm of it: - `blake3_chip_tests::chunked_blake3_proves_and_verifies` loses its batched arm; the per-table prove/verify and the host-chain digest equality stay. - `algebraic_transcript::phase_a_absorbs_derive_the_hosts_shared_pair` loses the shape histogram, both sides of it. What the gate is actually for survives intact: that a preprocessed root reaching the transcript as program TEXT (`append_const_bytes`) and as proof-carried cells (`RootCells::constant` + `absorb_misaligned`) derive the same challenge, i.e. that `bytes_to_cell` and `commitment_to_digest` agree. That property is Phase A's on either format and is invisible under a byte hash.
…ry points `batched_epoch.rs` (the shape histogram and the one-shared-query-set challenge spine) and `batched_epoch_verify.rs` (the mixed-height MMCS walk, the query mix, the batched FRI leg and the standalone terminal check) emit a verifier for a proof type that no longer exists. Their per-table counterparts are `epoch.rs` (`fork_table` + `emit_table_challenges`) and `epoch_verify.rs` + `sub_proof.rs`, both untouched. `proof.rs` loses `lfm_prove_batched`, `BatchedLfmProof`, `lfm_verify_batched` and `verify_against_batched`. `lfm_prove` / `lfm_verify` / `verify_against_artifacts` / `verify_against` were already the default everywhere and are unchanged; `expected_public_balance` and `decide_lfm_residency` are shared with the per-table path and stay. `aggregation_wrap_options` stays: blowup 4 / 110 queries with the FRI terminal at 2^8 is the wrap-to-aggregator preset, priced on the aggregator's per-query Merkle-walk volume, and is a property of the wrap layer rather than of a proof format. It has no caller in-tree at this commit. Note `epoch_verify.rs::group_leaf_felts` is a DIFFERENT function from the deleted `batched_epoch_verify.rs::group_leaf_felts` — same name, per-table subject, many callers — and is untouched.
`EpochProofBody` and `EpochProofFormat` collapse to the per-table variant, so `EpochProof.proof` is now a plain `MultiProof<F, E, ()>`. This is a WIRE FORMAT change: the rkyv-archived enum discriminant leaves `EpochProof`, so bundles do not round-trip across this commit. Accepted — one format end to end is the point. What goes with it: `prove_continuation_batched`, the `prove_continuation_with_format` indirection and `prove_epoch`'s `format` parameter (the per-table arm is now the whole function), `verify_epoch`'s batched arm, `BatchedEpochProofRef` and its `materialize` (the archived-side deserialization only the batched verifier needed), and `EpochProofView::batched_proof`. `per_table_proof` returns `MultiProofView` rather than `Option`, which drops the panic in `ArchivedContinuationProof::epoch_proof` and the `.expect` at its one other call site (`lfm::epoch_tests`): with one format there is no bundle shape the per-table arena can be handed and refuse. `num_sub_proofs` reads `proof.proofs.len()` directly. The GLOBAL memory proof and the cross-epoch binding are untouched — they were per-table under either format, which is why `verify_l2g_commitment_binding_view` needs no change. Tests: `the_batched_continuation_matches_the_per_table_l2g_roots` and `a_tampered_batched_continuation_is_rejected` go. Their per-table halves are already gated — `test_split_verify_rejects_tampered_register_fini` for the `reg_fini` arm, `lfm::epoch_tests` for the corrupted L2G root — and `corrupt_epoch_l2g_root_for_tests` keeps that caller. The `proof_diff` diagnostic loses its format match and diffs the one body.
… FRI The batched multi-Merkle-tree format is retired. Removed: `batched/proof.rs`, `batched/prover.rs`, `batched/round4.rs`, `batched/verifier.rs`, `fri/batched.rs` (height combination, the batched commit phase and the shared challenge derivation), and the two suites whose subject they are — `tests/batched_mmcs_soundness_tests.rs` and `tests/batched_prover_tests.rs`. The standalone-terminal binding and the two-class epsilon_C split go with round 4; the shape-binding argument goes with the format. `prover::multi_prove` and `verifier::multi_verify` are untouched and were already the default. `batched/shape.rs` and `fri/mmcs.rs` STAY for now, and only because production code outside this crate still reads them: `prover/src/lfm/registry.rs` takes a `&RoundShape` in `pinned_prep_widths`, and `prover/src/lfm/commit.rs`'s `PrepRoundBuilder` is a `StreamingMmcsBuilder`. Both die with the preprocessed round root in the registry commit, which is sequenced after the pin regeneration. `batched/mod.rs` is reduced to the surviving module and the doc links into the deleted ones are gone. Two rationale paragraphs lose their second caller rather than their claim: `prover::compute_composition_parts` and `verifier::step_2_verify_claimed_composition_polynomial` still exist for the reason their signatures show, so the docs now say what they are instead of who else called them.
…poch proof Collapsing `EpochProofBody` to a plain `MultiProof` changed the archived layout of `ContinuationGuestInput`: a v2 blob carries an enum discriminant at an offset a reader built after that commit does not expect. The version field is the DESIGNED guard for exactly that — its own doc records v1 to v2 as "32-bit offsets and are incompatible" — and leaving it at 2 meant the guard did not move with the format it guards. The visible cost was 17 failures in the `--lib` suite, all at `proof_fixture.rs:231` (`fixture blob must validate`). `proof_fixture::load_or_generate` returns the cached blob whenever `prefix_is_readable` passes, and that checks only the magic and the version, so a blob written by a pre-collapse build was handed back instead of regenerated and rkyv's validator rejected it downstream. At 3 the prefix check misses, the cache prints "not readable by this build — regenerating" and rewrites itself atomically, so it self-heals on every branch and every box with no manual eviction. The real reason is not the cache. `recursion_archive_bytes` exists so a wrong-format blob halts with a legible error "instead of surfacing whatever bytecheck makes of old-format bytes" — and the guest calls it before its unsafe in-place read. At version 2 a v2 blob passed that check and reached `rkyv::access`; at 3 it is refused where the refusal is legible. `cache_format_key` is deliberately NOT changed, though folding the format in would also have fixed the 17 (it keys `machine_tests::fixture_cache`'s filename). Its job is the axes that move the bytes WITHOUT moving the version — the statement domain tag and the commitment hash, both pin-level. A change that moves the version is already separated by the prefix check, so keying on it too would state one fact on two axes that can later drift apart. No literal 2 existed anywhere: every site reads the constant.
MauroToscano
force-pushed
the
pt/delete-batched
branch
from
September 8, 2026 16:54
b433674 to
0556f34
Compare
`verify_against_artifacts`'s "What it does NOT do yet" paragraph explained the gap by pointing at the other proof format: the preprocessed round was a commitment "only a verifier reading a `BatchedMultiProof` can use (`stark::batched::verifier`)", unchecked "until the machine switches paths". With that format gone from this branch there is no such proof, no such verifier, and no second path to switch to, so the sentence explained a situation that no longer exists — and a reader was told `prep_root` is plumbing ahead of a consumer that is never coming. The claim reads as current-state, so leaving it standing is worse than a dangling name: it misdescribes why a verification does not cover a field. The clause now states the standing fact — `prep_root` is a second commitment over the same preprocessed matrices and nothing reads it — and the trailing sentence stops promising a future consumer. The rest of the paragraph is untouched: it describes `prep_root`, which is still here, and goes with it when the registry's preprocessed round root goes.
MauroToscano
marked this pull request as ready for review
September 8, 2026 19:08
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Removes the batched multi-Merkle-tree proof format from this lane, leaving the
per-table prover as the only format on this branch, end to end.
This is one of two drafts held side by side rather than a retirement. The batched format
stays open as #951 and is preserved in full at
archive/batched-format-pre-deletion@ 59fc77d. "Where a new proof format attaches"below is the map of the seams it used, written for whoever adds the next one.
+610 / −15,776 over 35 files, six signed commits, one area each.
crypto/starkbatched/{proof,prover,round4,verifier}.rs,fri/batched.rs, and the two suites whose subject they are (batched_mmcs_soundness_tests.rs,batched_prover_tests.rs)crypto/math-cudasrc/mmcs.rs, its five backend handles, the five MMCS kernels inkernels/keccak.cu, andtests/mmcs_tree_parity.rsprover/src/lfmbatched_epoch.rs,batched_epoch_verify.rs,group_leaf_tests.rs,aggregator_tests.rs, the batched entry points inproof.rs, and the batched tests across five filesprover/src/continuation.rsEpochProofFormat/EpochProofBodycollapse to a plainMultiProoffieldTwo files stay alive on purpose and die in the follow-up:
batched/shape.rsandfri/mmcs.rs. Production code outsidestarkstill reads them —prover/src/lfm/registry.rstakes a&RoundShapeinpinned_prep_widths, andprover/src/lfm/commit.rs'sPrepRoundBuilderis aStreamingMmcsBuilder. Both go withthe preprocessed round root in the registry commit, which is sequenced after the pin
regeneration.
The wire format moves
EpochProof.proofis aMultiProofwhere it was anEpochProofBodyenum, so everyarchived bundle carries a discriminant a reader built after this change does not expect.
RECURSION_INPUT_VERSIONgoes 2 → 3 with it.That bump is the point, not a cache-busting trick. The version field is the DESIGNED
guard for an incompatible encoding — its own doc records v1 → v2 as "32-bit offsets and
are incompatible" — and this branch moved the encoding without it at first. The visible
cost was 17 failures at
proof_fixture.rs:231, becauseload_or_generatereturns thecached blob whenever
prefix_is_readablepasses and that checks only magic and version.The invisible cost was the real one:
recursion_archive_bytesexists so a wrong-formatblob halts legibly "instead of surfacing whatever bytecheck makes of old-format bytes",
and the guest calls it before its unsafe in-place read. At version 2 a stale blob passed
that check and reached
rkyv::access.cache_format_keyis deliberately NOT changed, though folding the format in would alsohave made the 17 green. Its documented job is the axes that move the bytes WITHOUT moving
the version — the statement domain tag and the commitment hash, both pin-level. A
version-moving change is already separated by the prefix check, so keying on it twice
would state one fact on two axes that can drift apart. Keying the cache would have turned
the tests green while leaving the legibility hole exactly where it was.
The global leg moves rather than dying
aggregator_tests.rsis typed end to end onBatchedMultiProof, but its GLOBAL leg isnot: the cross-epoch global memory proof is per-table under either format.
RealGlobal,real_global,GlobalTableArenas,global_verifier_program,global_arena_wordsandthe_global_verifier_leg_runs_and_rejects_tampersmove verbatim intoper_table_aggregator_tests.rs, non-ignored, with one substitution: the fixture bundlecomes from
prove_continuationinstead ofprove_continuation_batched. Identicalsignatures, and
real_globalalready took a plainContinuationProof.Why the rebase kept that file deleted, and what was checked first
The RPX pin edits
aggregator_tests.rs, so the rebase hit a modify/delete. Resolving itto KEEP DELETED is exactly the kind of resolution that looks like dropping someone else's
fix, so it was verified rather than assumed. The pin's entire diff to that file is one
pattern applied seven times —
build_artifacts(...)→build_artifacts_with_hasher(..., BLOCK_HASHER)at :1682, :1880, :1928, :2198, :2423,:2458, :2588, plus the import at :39. None of the seven lands inside the six ranges
extracted into
per_table_aggregator_tests.rs; all are infixture_leg,fixture_wraps,fixture_aggregate,the_aggregate_leg_census_matches_the_closed_formandthe_real_block_aggregates_end_to_end, every one deleted. And the extracted file needsnone of them: it contains no
build_artifactscall at all, builds atWrapHash::production()and namesBLOCK_HASHERat bothexecutesites, so it alreadysatisfies the classification rule by executing rather than proving. Keeping the deletion
drops no pin fix.
The other rebase conflicts resolve in the PIN's favour wherever the enclosing item
survives. In particular
wrap_tests.rstakes the pin's import block — it dropsTestPermutationand widensbuild_artifactsto{build_artifacts, build_artifacts_with_hasher}, both servingthe_census_agrees_with_the_traces_the_prover_builds,which survives — and keeps the pin's module-doc rewrite explaining why that test reads its
tenant from
artifacts.hasher: the socket chip's width is tenant-dependent, so adefaulted
HasherKindis an out-of-bounds index rather than a disagreement.Two tests edited rather than deleted
Their batched half was an arm of a claim that is not batched.
blake3_chip_tests::chunked_blake3_proves_and_verifiesloses its batched arm only; theper-table prove/verify and the host-chain digest equality stay.
algebraic_transcript::phase_a_absorbs_derive_the_hosts_shared_pairloses the shapehistogram, both sides of it, and survives the format it was written beside. What the gate
is actually for is format-independent: that a preprocessed root reaching the transcript as
program TEXT (
append_const_bytes) and as proof-carried cells (RootCells::constant+absorb_misaligned) derives the same challenge, i.e. thatbytes_to_cellandcommitment_to_digestagree. That is Phase A's property under either format, it isinvisible under a byte hash, and it is load-bearing precisely because the block path is
now algebraic. Deleting it whole would have removed a live gate to tidy a format.
One intended coverage loss
algebraic_commit::the_mixed_group_leaf_equals_the_hosts_group_hashis deleted. Itssubject is a leaf over SEVERAL concatenated matrices, which only the mixed-height MMCS
constructs, so the property has no per-table meaning. The one-matrix case it degenerates
to stays gated by
the_emitted_leaf_and_parent_equal_the_hosts_backendin the same file.group_leaf_tests.rscovered the same construction and goes with it.Bycatch, not churn
Two doc blocks were misattached before this branch, each by a missing blank line between a
doc comment and its item, and each is returned to its owner by deleting the batched item
that had absorbed it:
real_epoch_withinepoch_tests.rs(absorbed bythe_real_block_base_epoch_ab) andthe_real_blocks_page_censusinwrap_tests.rs(absorbed by the batched block driver).
Where a new proof format attaches
The batched format is not being erased — it stays open as #951 and as
archive/batched-format-pre-deletion@ 59fc77d, the last tip carrying all fourteenbatched and mixed-height files. This section is the map that branch is a reference
implementation of: every seam a second format used, and what this PR leaves standing
there. A third format's cost is dominated by finding these, not by editing them.
EpochProofFormat(the choice),EpochProofBody(the payload),EpochProofView::batched_proof,BatchedEpochProofRefEpochProof.proofis a plainMultiProofprove_continuation_batched→prove_continuation_with_format→prove_epoch(format)and itsmatchprove_continuationonlyverify_epoch's second arm, replaying and verifying the carved prooflfm_prove_batched,verify_against_batched,BatchedLfmProof— deliberately siblings oflfm_prove/verify_againstrather than anOption, because the wire type differedstark::batched::{prover,verifier}besidestark::prover::multi_prove/stark::verifier::multi_verifyfri/batched.rs(height combination, batched commit phase, shared challenge derivation) andfri/mmcs.rs(mixed-height MMCS) besidefri::commit_phase_from_evaluations/fri::query_phasefri/batched.rsdeleted;fri/mmcs.rsstill standinglfm/batched_epoch.rs(challenge spine) andlfm/batched_epoch_verify.rs(walk, query mix, FRI leg) besidelfm/epoch.rs,lfm/epoch_verify.rs,lfm/sub_proof.rsstark::batched::shape—RoundShape,EpochShape,PinnedPrep,CarvedMainlfm/commit.rs::PrepRoundBuilderandregistry.rs'sprep_root/pinned_prep_widthsmath_cuda::mmcsand its five keccak kernels besidemerkle.rsRECURSION_INPUT_VERSION, which MUST move whenever the archived encoding changesTwo things are worth knowing before reading that table as a to-do list.
The container enum is probably the wrong seam for a low-degree-test swap. The batched
format needed a container variant because it changed the SHAPE of an epoch proof: one
mixed-height-MMCS proof for the whole epoch instead of one per table. A format that
replaces the proximity test while keeping one proof per table changes what lives INSIDE a
StarkProof— the commit phase, the query phase, the FRI payload and its emitter legs —and needs no container variant at all. That work lands in the
friseam and the emitterseam, both of which this PR leaves structurally intact on the per-table side.
The wire version is the guard, and it is easy to forget. Any format change that moves
the archived encoding must bump
RECURSION_INPUT_VERSION. This PR is itself the workedexample: the collapse moved the encoding, the bump was initially missed, and the failure
surfaced far away as a stale fixture blob rather than as a format error.
Evidence
Gated at this head:
An earlier gate of the same six commits on the pre-pin tip read 1102 / 1 / 43; its single
failure was
epoch_tests::the_closure_rejects_a_moved_index_or_output, which is also inthe list below. The rebase onto the RPX pin is what brings the other seven.
All 8 failures are the pin's, none are this branch's. #967 lands with ten known-red
tests; this branch deletes two of them because they are batched
(
epoch_tests::the_assembled_carved_batched_epoch_verifier_runsandepoch_tests::the_batched_query_census_matches_the_closed_form), leaving the other eightuntouched:
epoch_tests::the_closure_rejects_a_moved_index_or_output,fri_tests::the_emitted_permutation_count_meets_the_pinned_prediction,fri_tests::the_fri_join_adds_no_second_point_derivation,machine_tests::register_derivation_cost,machine_tests::the_register_derivation_matches_production,machine_tests::the_register_derivation_proves_and_verifies,machine_tests::transcript_replay_cell_counts, andper_table_census_tests::the_blake3_tenant_socket_matches_the_record. See #967 for whyeach is red.
The arithmetic closes from both ends. #967's gate reads 1119 / 10 / 47 = 1176. This branch
removes 30 tests, of which 2 were failing, 4 were ignored and 24 were passing, giving
1095 / 8 / 43 = 1146 — and 1146 is what
cargo test -- --listreports at this headindependently.
One counting note, because two different numbers in this area are easy to conflate. A
#[test]ATTRIBUTE count is not a count of what RUNS: modules behinddebug-checks,disk-spillandcudaare compiled out of a default build.crypto/stark/srccarries 307attributes at this head and the binary runs 285, and grepping
#[ignore]overprover/srcover-counts the ignored total by one for the same reason. Every count quotedabove is from a gate run rather than from a grep.