Skip to content

refactor: delete the batched proof format, per-table only - #973

Merged
MauroToscano merged 7 commits into
per-table-gpufrom
pt/delete-batched
Sep 8, 2026
Merged

refactor: delete the batched proof format, per-table only#973
MauroToscano merged 7 commits into
per-table-gpufrom
pt/delete-batched

Conversation

@MauroToscano

Copy link
Copy Markdown
Contributor

Draft. Removes the batched multi-Merkle-tree proof format from this lane, leaving the
per-table prover as the only format on this branch, end to end.

This is one of two drafts held side by side rather than a retirement. The batched format
stays open as #951 and is preserved in full at
archive/batched-format-pre-deletion @ 59fc77d. "Where a new proof format attaches"
below is the map of the seams it used, written for whoever adds the next one.

+610 / −15,776 over 35 files, six signed commits, one area each.

area +/− what goes
crypto/stark +47 / −7,477 batched/{proof,prover,round4,verifier}.rs, fri/batched.rs, and the two suites whose subject they are (batched_mmcs_soundness_tests.rs, batched_prover_tests.rs)
crypto/math-cuda +1 / −730 src/mmcs.rs, its five backend handles, the five MMCS kernels in kernels/keccak.cu, and tests/mmcs_tree_parity.rs
prover/src/lfm +513 / −7,208 batched_epoch.rs, batched_epoch_verify.rs, group_leaf_tests.rs, aggregator_tests.rs, the batched entry points in proof.rs, and the batched tests across five files
prover/src/continuation.rs +49 / −361 EpochProofFormat / EpochProofBody collapse to a plain MultiProof field

Two files stay alive on purpose and die in the follow-up: batched/shape.rs and
fri/mmcs.rs. Production code outside stark still reads them —
prover/src/lfm/registry.rs takes a &RoundShape in pinned_prep_widths, and
prover/src/lfm/commit.rs's PrepRoundBuilder is a StreamingMmcsBuilder. Both go with
the preprocessed round root in the registry commit, which is sequenced after the pin
regeneration.

The wire format moves

EpochProof.proof is a MultiProof where it was an EpochProofBody enum, so every
archived bundle carries a discriminant a reader built after this change does not expect.
RECURSION_INPUT_VERSION goes 2 → 3 with it.

That bump is the point, not a cache-busting trick. The version field is the DESIGNED
guard for an incompatible encoding — its own doc records v1 → v2 as "32-bit offsets and
are incompatible" — and this branch moved the encoding without it at first. The visible
cost was 17 failures at proof_fixture.rs:231, because load_or_generate returns the
cached blob whenever prefix_is_readable passes and that checks only magic and version.
The invisible cost was the real one: recursion_archive_bytes exists so a wrong-format
blob halts legibly "instead of surfacing whatever bytecheck makes of old-format bytes",
and the guest calls it before its unsafe in-place read. At version 2 a stale blob passed
that check and reached rkyv::access.

cache_format_key is deliberately NOT changed, though folding the format in would also
have made the 17 green. Its documented job is the axes that move the bytes WITHOUT moving
the version — the statement domain tag and the commitment hash, both pin-level. A
version-moving change is already separated by the prefix check, so keying on it twice
would state one fact on two axes that can drift apart. Keying the cache would have turned
the tests green while leaving the legibility hole exactly where it was.

The global leg moves rather than dying

aggregator_tests.rs is typed end to end on BatchedMultiProof, but its GLOBAL leg is
not: the cross-epoch global memory proof is per-table under either format. RealGlobal,
real_global, GlobalTableArenas, global_verifier_program, global_arena_words and
the_global_verifier_leg_runs_and_rejects_tampers move verbatim into
per_table_aggregator_tests.rs, non-ignored, with one substitution: the fixture bundle
comes from prove_continuation instead of prove_continuation_batched. Identical
signatures, and real_global already took a plain ContinuationProof.

Why the rebase kept that file deleted, and what was checked first

The RPX pin edits aggregator_tests.rs, so the rebase hit a modify/delete. Resolving it
to KEEP DELETED is exactly the kind of resolution that looks like dropping someone else's
fix, so it was verified rather than assumed. The pin's entire diff to that file is one
pattern applied seven times — build_artifacts(...)
build_artifacts_with_hasher(..., BLOCK_HASHER) at :1682, :1880, :1928, :2198, :2423,
:2458, :2588, plus the import at :39. None of the seven lands inside the six ranges
extracted into per_table_aggregator_tests.rs; all are in fixture_leg, fixture_wraps,
fixture_aggregate, the_aggregate_leg_census_matches_the_closed_form and
the_real_block_aggregates_end_to_end, every one deleted. And the extracted file needs
none of them: it contains no build_artifacts call at all, builds at
WrapHash::production() and names BLOCK_HASHER at both execute sites, so it already
satisfies the classification rule by executing rather than proving. Keeping the deletion
drops no pin fix.

The other rebase conflicts resolve in the PIN's favour wherever the enclosing item
survives. In particular wrap_tests.rs takes the pin's import block — it drops
TestPermutation and widens build_artifacts to {build_artifacts, build_artifacts_with_hasher}, both serving the_census_agrees_with_the_traces_the_prover_builds,
which survives — and keeps the pin's module-doc rewrite explaining why that test reads its
tenant from artifacts.hasher: the socket chip's width is tenant-dependent, so a
defaulted HasherKind is an out-of-bounds index rather than a disagreement.

Two tests edited rather than deleted

Their batched half was an arm of a claim that is not batched.

blake3_chip_tests::chunked_blake3_proves_and_verifies loses its batched arm only; the
per-table prove/verify and the host-chain digest equality stay.

algebraic_transcript::phase_a_absorbs_derive_the_hosts_shared_pair loses the shape
histogram, both sides of it, and survives the format it was written beside. What the gate
is actually for is format-independent: that a preprocessed root reaching the transcript as
program TEXT (append_const_bytes) and as proof-carried cells (RootCells::constant +
absorb_misaligned) derives the same challenge, i.e. that bytes_to_cell and
commitment_to_digest agree. That is Phase A's property under either format, it is
invisible under a byte hash, and it is load-bearing precisely because the block path is
now algebraic. Deleting it whole would have removed a live gate to tidy a format.

One intended coverage loss

algebraic_commit::the_mixed_group_leaf_equals_the_hosts_group_hash is deleted. Its
subject is a leaf over SEVERAL concatenated matrices, which only the mixed-height MMCS
constructs, so the property has no per-table meaning. The one-matrix case it degenerates
to stays gated by the_emitted_leaf_and_parent_equal_the_hosts_backend in the same file.
group_leaf_tests.rs covered the same construction and goes with it.

Bycatch, not churn

Two doc blocks were misattached before this branch, each by a missing blank line between a
doc comment and its item, and each is returned to its owner by deleting the batched item
that had absorbed it: real_epoch_with in epoch_tests.rs (absorbed by
the_real_block_base_epoch_ab) and the_real_blocks_page_census in wrap_tests.rs
(absorbed by the batched block driver).

Where a new proof format attaches

The batched format is not being erased — it stays open as #951 and as
archive/batched-format-pre-deletion @ 59fc77d, the last tip carrying all fourteen
batched and mixed-height files. This section is the map that branch is a reference
implementation of: every seam a second format used, and what this PR leaves standing
there. A third format's cost is dominated by finding these, not by editing them.

seam what the batched format put there after this PR
epoch container + dispatch EpochProofFormat (the choice), EpochProofBody (the payload), EpochProofView::batched_proof, BatchedEpochProofRef collapsed — EpochProof.proof is a plain MultiProof
prove entry prove_continuation_batchedprove_continuation_with_formatprove_epoch(format) and its match collapsed — prove_continuation only
verify entry verify_epoch's second arm, replaying and verifying the carved proof collapsed — one straight-line arm
LFM prove/verify lfm_prove_batched, verify_against_batched, BatchedLfmProof — deliberately siblings of lfm_prove/verify_against rather than an Option, because the wire type differed deleted; the per-table pair is unchanged
host prover/verifier stark::batched::{prover,verifier} beside stark::prover::multi_prove / stark::verifier::multi_verify deleted
low-degree test + commitment fri/batched.rs (height combination, batched commit phase, shared challenge derivation) and fri/mmcs.rs (mixed-height MMCS) beside fri::commit_phase_from_evaluations / fri::query_phase fri/batched.rs deleted; fri/mmcs.rs still standing
recursion emitters lfm/batched_epoch.rs (challenge spine) and lfm/batched_epoch_verify.rs (walk, query mix, FRI leg) beside lfm/epoch.rs, lfm/epoch_verify.rs, lfm/sub_proof.rs deleted
shape description stark::batched::shapeRoundShape, EpochShape, PinnedPrep, CarvedMain still standing
preprocessed round root lfm/commit.rs::PrepRoundBuilder and registry.rs's prep_root / pinned_prep_widths still standing
device math_cuda::mmcs and its five keccak kernels beside merkle.rs deleted
wire version RECURSION_INPUT_VERSION, which MUST move whenever the archived encoding changes

Two things are worth knowing before reading that table as a to-do list.

The container enum is probably the wrong seam for a low-degree-test swap. The batched
format needed a container variant because it changed the SHAPE of an epoch proof: one
mixed-height-MMCS proof for the whole epoch instead of one per table. A format that
replaces the proximity test while keeping one proof per table changes what lives INSIDE a
StarkProof — the commit phase, the query phase, the FRI payload and its emitter legs —
and needs no container variant at all. That work lands in the fri seam and the emitter
seam, both of which this PR leaves structurally intact on the per-table side.

The wire version is the guard, and it is easy to forget. Any format change that moves
the archived encoding must bump RECURSION_INPUT_VERSION. This PR is itself the worked
example: the collapse moved the encoding, the bump was initially missed, and the failure
surfaced far away as a stale fixture blob rather than as a format error.

Evidence

Gated at this head:

cargo check --workspace --all-targets                 exit 0
make lint                                             exit 0   (all five feature passes,
                                                                including cuda)
cargo test --release -p stark
  285 passed; 0 failed; 3 ignored   (lib binary + three integration binaries)
cargo test --release -p lambda-vm-prover --lib
  1095 passed; 8 failed; 43 ignored; 0 measured; 0 filtered out; finished in 2108.80s
machine_tests::registry_drift_*                       6/6 green

An earlier gate of the same six commits on the pre-pin tip read 1102 / 1 / 43; its single
failure was epoch_tests::the_closure_rejects_a_moved_index_or_output, which is also in
the list below. The rebase onto the RPX pin is what brings the other seven.

All 8 failures are the pin's, none are this branch's. #967 lands with ten known-red
tests; this branch deletes two of them because they are batched
(epoch_tests::the_assembled_carved_batched_epoch_verifier_runs and
epoch_tests::the_batched_query_census_matches_the_closed_form), leaving the other eight
untouched: epoch_tests::the_closure_rejects_a_moved_index_or_output,
fri_tests::the_emitted_permutation_count_meets_the_pinned_prediction,
fri_tests::the_fri_join_adds_no_second_point_derivation,
machine_tests::register_derivation_cost,
machine_tests::the_register_derivation_matches_production,
machine_tests::the_register_derivation_proves_and_verifies,
machine_tests::transcript_replay_cell_counts, and
per_table_census_tests::the_blake3_tenant_socket_matches_the_record. See #967 for why
each is red.

The arithmetic closes from both ends. #967's gate reads 1119 / 10 / 47 = 1176. This branch
removes 30 tests, of which 2 were failing, 4 were ignored and 24 were passing, giving
1095 / 8 / 43 = 1146 — and 1146 is what cargo test -- --list reports at this head
independently.

One counting note, because two different numbers in this area are easy to conflate. A
#[test] ATTRIBUTE count is not a count of what RUNS: modules behind debug-checks,
disk-spill and cuda are compiled out of a default build. crypto/stark/src carries 307
attributes at this head and the binary runs 285, and grepping #[ignore] over
prover/src over-counts the ignored total by one for the same reason. Every count quoted
above is from a gate run rather than from a grep.

The batched multi-Merkle-tree proof format is retired, and the device MMCS
existed only for it: `MmcsGroupHasher` was the device twin of the host's
`StreamingMmcsBuilder`, and `build_mmcs_tree_on_device` built the one tree
per epoch that the per-table prover does not have.

Removed: `src/mmcs.rs`, its module declaration, the five backend handles and
their loads in `src/device.rs`, the five kernels in `kernels/keccak.cu`
(`mmcs_states_init`, `mmcs_absorb_row_pair_row_major`,
`mmcs_absorb_row_pair_ext3_slabs`, `mmcs_states_finalize`,
`keccak_mmcs_level`), and `tests/mmcs_tree_parity.rs`, whose subject was
device-vs-host root parity for that tree.

`kernels/blake3.cu` had no MMCS twin to remove. `merkle_gather_paths` and
`keccak_launch_cfg` are hash- and format-agnostic and stay: the per-table
path is their caller.
…own module

The batched proof format is retired, so every test whose SUBJECT is that
format goes. Deletions, by file:

- `aggregator_tests.rs` (2,664 lines, 10 tests) — the module is typed end to
  end on `BatchedMultiProof`. Its GLOBAL leg is not: the cross-epoch global
  memory proof is per-table under either format, so `RealGlobal`,
  `real_global`, `GlobalTableArenas`, `global_verifier_program`,
  `global_arena_words` and `the_global_verifier_leg_runs_and_rejects_tampers`
  move verbatim into `per_table_aggregator_tests.rs`. One substitution: the
  fixture bundle now comes from `prove_continuation` instead of
  `prove_continuation_batched` — identical signature, and `real_global` already
  took a plain `ContinuationProof`.
- `group_leaf_tests.rs` (295 lines, 3 tests) — the mixed-height group leaf.
- `epoch_tests.rs` — the batched harness and its 7 tests, plus
  `the_real_block_base_epoch_ab`, whose subject was per-table VS batched and
  which cannot be a one-armed A/B. That test had absorbed `real_epoch_with`'s
  doc block (no blank line between them); deleting it returns the doc to
  `real_epoch_with`.
- `wrap_tests.rs` — the batched wrap section, its 3 tests and the P2 batched
  block driver. That driver's doc had likewise absorbed
  `the_real_blocks_page_census`'s; the census doc is reattached.
- `epoch_verify_tests.rs` — the batched leg-input builders.
- `machine_tests.rs` — the three batched LFM epoch/wire tests.
- `algebraic_commit.rs` — `the_mixed_group_leaf_equals_the_hosts_group_hash`.
  A group leaf over several matrices is the mixed-height construction; the
  one-matrix case it degenerates to is `the_emitted_leaf_and_parent_equal_the_hosts_backend`'s
  subject and stays gated. `StarkHash::Batched` is the batch-commitment leaf
  backend, not the format, and is untouched.

Two tests are EDITED rather than deleted, because their per-table half is the
claim and the batched half was an arm of it:

- `blake3_chip_tests::chunked_blake3_proves_and_verifies` loses its batched
  arm; the per-table prove/verify and the host-chain digest equality stay.
- `algebraic_transcript::phase_a_absorbs_derive_the_hosts_shared_pair` loses
  the shape histogram, both sides of it. What the gate is actually for
  survives intact: that a preprocessed root reaching the transcript as program
  TEXT (`append_const_bytes`) and as proof-carried cells
  (`RootCells::constant` + `absorb_misaligned`) derive the same challenge,
  i.e. that `bytes_to_cell` and `commitment_to_digest` agree. That property is
  Phase A's on either format and is invisible under a byte hash.
…ry points

`batched_epoch.rs` (the shape histogram and the one-shared-query-set challenge
spine) and `batched_epoch_verify.rs` (the mixed-height MMCS walk, the query
mix, the batched FRI leg and the standalone terminal check) emit a verifier for
a proof type that no longer exists. Their per-table counterparts are
`epoch.rs` (`fork_table` + `emit_table_challenges`) and `epoch_verify.rs` +
`sub_proof.rs`, both untouched.

`proof.rs` loses `lfm_prove_batched`, `BatchedLfmProof`, `lfm_verify_batched`
and `verify_against_batched`. `lfm_prove` / `lfm_verify` /
`verify_against_artifacts` / `verify_against` were already the default
everywhere and are unchanged; `expected_public_balance` and
`decide_lfm_residency` are shared with the per-table path and stay.

`aggregation_wrap_options` stays: blowup 4 / 110 queries with the FRI terminal
at 2^8 is the wrap-to-aggregator preset, priced on the aggregator's per-query
Merkle-walk volume, and is a property of the wrap layer rather than of a proof
format. It has no caller in-tree at this commit.

Note `epoch_verify.rs::group_leaf_felts` is a DIFFERENT function from the
deleted `batched_epoch_verify.rs::group_leaf_felts` — same name, per-table
subject, many callers — and is untouched.
`EpochProofBody` and `EpochProofFormat` collapse to the per-table variant, so
`EpochProof.proof` is now a plain `MultiProof<F, E, ()>`. This is a WIRE FORMAT
change: the rkyv-archived enum discriminant leaves `EpochProof`, so bundles do
not round-trip across this commit. Accepted — one format end to end is the
point.

What goes with it: `prove_continuation_batched`, the
`prove_continuation_with_format` indirection and `prove_epoch`'s `format`
parameter (the per-table arm is now the whole function), `verify_epoch`'s
batched arm, `BatchedEpochProofRef` and its `materialize` (the archived-side
deserialization only the batched verifier needed), and
`EpochProofView::batched_proof`.

`per_table_proof` returns `MultiProofView` rather than `Option`, which drops
the panic in `ArchivedContinuationProof::epoch_proof` and the `.expect` at its
one other call site (`lfm::epoch_tests`): with one format there is no bundle
shape the per-table arena can be handed and refuse. `num_sub_proofs` reads
`proof.proofs.len()` directly.

The GLOBAL memory proof and the cross-epoch binding are untouched — they were
per-table under either format, which is why `verify_l2g_commitment_binding_view`
needs no change.

Tests: `the_batched_continuation_matches_the_per_table_l2g_roots` and
`a_tampered_batched_continuation_is_rejected` go. Their per-table halves are
already gated — `test_split_verify_rejects_tampered_register_fini` for the
`reg_fini` arm, `lfm::epoch_tests` for the corrupted L2G root — and
`corrupt_epoch_l2g_root_for_tests` keeps that caller. The `proof_diff`
diagnostic loses its format match and diffs the one body.
… FRI

The batched multi-Merkle-tree format is retired. Removed: `batched/proof.rs`,
`batched/prover.rs`, `batched/round4.rs`, `batched/verifier.rs`,
`fri/batched.rs` (height combination, the batched commit phase and the shared
challenge derivation), and the two suites whose subject they are —
`tests/batched_mmcs_soundness_tests.rs` and `tests/batched_prover_tests.rs`.

The standalone-terminal binding and the two-class epsilon_C split go with round 4;
the shape-binding argument goes with the format. `prover::multi_prove` and
`verifier::multi_verify` are untouched and were already the default.

`batched/shape.rs` and `fri/mmcs.rs` STAY for now, and only because production
code outside this crate still reads them: `prover/src/lfm/registry.rs` takes a
`&RoundShape` in `pinned_prep_widths`, and `prover/src/lfm/commit.rs`'s
`PrepRoundBuilder` is a `StreamingMmcsBuilder`. Both die with the preprocessed
round root in the registry commit, which is sequenced after the pin
regeneration. `batched/mod.rs` is reduced to the surviving module and the doc
links into the deleted ones are gone.

Two rationale paragraphs lose their second caller rather than their claim:
`prover::compute_composition_parts` and
`verifier::step_2_verify_claimed_composition_polynomial` still exist for the
reason their signatures show, so the docs now say what they are instead of who
else called them.
…poch proof

Collapsing `EpochProofBody` to a plain `MultiProof` changed the archived
layout of `ContinuationGuestInput`: a v2 blob carries an enum discriminant at
an offset a reader built after that commit does not expect. The version field
is the DESIGNED guard for exactly that — its own doc records v1 to v2 as
"32-bit offsets and are incompatible" — and leaving it at 2 meant the guard did
not move with the format it guards.

The visible cost was 17 failures in the `--lib` suite, all at
`proof_fixture.rs:231` (`fixture blob must validate`).
`proof_fixture::load_or_generate` returns the cached blob whenever
`prefix_is_readable` passes, and that checks only the magic and the version, so
a blob written by a pre-collapse build was handed back instead of regenerated
and rkyv's validator rejected it downstream. At 3 the prefix check misses, the
cache prints "not readable by this build — regenerating" and rewrites itself
atomically, so it self-heals on every branch and every box with no manual
eviction.

The real reason is not the cache. `recursion_archive_bytes` exists so a
wrong-format blob halts with a legible error "instead of surfacing whatever
bytecheck makes of old-format bytes" — and the guest calls it before its unsafe
in-place read. At version 2 a v2 blob passed that check and reached
`rkyv::access`; at 3 it is refused where the refusal is legible.

`cache_format_key` is deliberately NOT changed, though folding the format in
would also have fixed the 17 (it keys `machine_tests::fixture_cache`'s
filename). Its job is the axes that move the bytes WITHOUT moving the version —
the statement domain tag and the commitment hash, both pin-level. A change that
moves the version is already separated by the prefix check, so keying on it too
would state one fact on two axes that can later drift apart.

No literal 2 existed anywhere: every site reads the constant.
`verify_against_artifacts`'s "What it does NOT do yet" paragraph explained the
gap by pointing at the other proof format: the preprocessed round was a
commitment "only a verifier reading a `BatchedMultiProof` can use
(`stark::batched::verifier`)", unchecked "until the machine switches paths".

With that format gone from this branch there is no such proof, no such
verifier, and no second path to switch to, so the sentence explained a
situation that no longer exists — and a reader was told `prep_root` is
plumbing ahead of a consumer that is never coming. The claim reads as
current-state, so leaving it standing is worse than a dangling name: it
misdescribes why a verification does not cover a field.

The clause now states the standing fact — `prep_root` is a second commitment
over the same preprocessed matrices and nothing reads it — and the trailing
sentence stops promising a future consumer. The rest of the paragraph is
untouched: it describes `prep_root`, which is still here, and goes with it
when the registry's preprocessed round root goes.
@MauroToscano
MauroToscano marked this pull request as ready for review September 8, 2026 19:08
@MauroToscano
MauroToscano merged commit 99d25bc into per-table-gpu Sep 8, 2026
11 of 14 checks passed
@MauroToscano
MauroToscano deleted the pt/delete-batched branch September 8, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant