Multiproof optimisations

[ajhavlin]

Description

Batch proof pruning: switch to CoSet multi-proof

This PR optimizes multiproofs within Merkle_Tree. Inspired by the CoSet strategy described by Chiesa and Yogev, the proof that is transmit is compressed to the optimal set of digests by applying deduplication and excluding nodes along any of the individual computation paths.

Motivation and old strategy (front-incremental / prefix encoding)

• The proof carried the full auth path for each leaf in an ordered leaf-indexed list.
• Compression compares each auth path against the previous auth path in the ordered leaf-indexed list, reusing the previous path's nodes until the two paths diverge (we call reused nodes the "prefix" and the remainder of the path is the "suffix").

For example, verifying the set ${\ I, J, L\ }$ with the prefix strategy in the following tree:

             A              d = 0
           /   \
          B     C           d = 1
        / \    /  \
       D  E   F    H        d = 2
     ... / \ / \ ....
        I  J L  M           d = 3

proceeds as follows:

1. Compute the full auth path for leaf $I$: $\textsf{Copath}(I) = {\ C, D, J\ }$.
2. Set prefix_length = $0$, suffix = $\textsf{Copath}(I)$.
3. Compute the full auth path for leaf $J$: $\textsf{Copath}(J) = {\ C, D, I\ }$.
4. Compare against $\textsf{Copath}(I)$; the two overlap at $C, D$ so prefix_length = $2$, suffix = ${\ I\ }$.
5. Compute the full auth path for leaf $L$: $\textsf{Copath}(L) = {\ B, H, M\ }$.
6. No overlap with $\textsf{Copath}(J)$ so prefix_length = $0$, suffix = $\textsf{Copath}(J)$.

New strategy: CoSet

Prefix encoding was transmitting redundant digests.

Consider a new strategy where, for a batch $I$ of leaf indexes, we define the on-path sets $A_j$ as the union across single-openings of parent nodes at depth $j$, and define the minimal copath $B_j^$ as the nodes at depth $j$ that must be transmit for the verifier's checks, given that the verifier can compute $A_j$ ($B_j^ = \textsf{siblings}(A_j) \setminus A_j$).

Taking the example above: $\textsf{CoSet} = {\ D, H, M\ }$, split as leaf_copath $= {\ M\ }$ and inner_copath $= {\ D, H\ }$.

Both the prover and verifier can independently recompute $A_j$ for all depths from leaf_indexes and tree_height alone, using compute_on_path. Therefore the proof needed is:

(tree_height: usize, leaf_copath: Vec<LeafDigest>, inner_copath: Vec<InnerDigest>, leaf_indexes: Vec<usize>)

Digests in inner_copath are emitted in canonical order: depth $1$ ascending, depth $2$ ascending, …, depth $d-2$ ascending. The verifier reconstructs the same ordering, consumes the iterator, and rejects on any count mismatch.

Security. The prover now only has the ability to choose the digests that it transmits. Therefore, any malicious choice of digests will reject so long as collision-resistance holds.

Proof size. Exactly $|B_\text{leaf}^| + |\bigcup_j B_j^|$ digests. This is at most $|I| \cdot (d - \log_2|I|) + |I| - 1$ digests in the most spread apart choice of $I$, and is strictly smaller than the front-incremental scheme whenever opened leaves share on-path ancestors.

Implementation

• CoPath.inner_copath is a flat Vec<P::InnerDigest>, traversed from leaf level to root. Copath.leaf_copath is built in the same manner but left explicit to separate between the LeafHash and TwoToOneHash domains. Additional proof supplements are only tree_height and leaf_indexes.
• MerkleTree::generate_multi_proof is the single entry point for batch proof generation.
• CoPath::verify consumes inner_copath via an iterator; copath_iter.next().is_some() asserts exact exhaustion.
• inner_levels is the sole authoritative map in verify and the recompute helpers.
• CoPath implements the batch proof logic directly; helpers (ingest_leaves, expected_leaf_coset, validate_leaf_copath, recompute_bottom_parents, recompute_inner_layers) are module-private.

Test suite

field_mt_tests covers three categories:

• Correctness: generate proofs for single-leaf and full-batch cases and assert verification passes.
• Rejection: tampered leaf copath, missing or providing excess inner digests, wrong tree height, empty batch, duplicate indices, and mismatched leaf ordering all fail verification.
• Structural count: assert that the exact inner_copath.len() is met for the structures sharing a parent, having divergent paths, full subtrees, spread leaves, and all leaves.

Benchmark Results

Benchmark comparisons for the two Merkle tree batching strategies were ran across various tree depths and opening structures.

Key outcomes:

• Reduced proof size consistently (upwards of a 90% reduction) compared to the prefix encoding, closely matching the derived upper bound: $$|\textsf{CoSet}(I)| \leq |I| \cdot (d - \log_2|I|) + (|I|-1),$$where $I$ is the opening set, $d$ the depth, and $|\textsf{CoSet}(I)|$ the number of co-path nodes.
• Both prover and verifier time remain almost identical since at the core we are still computing the on-path sets for each opening and applying a deduplication and sorting step.

Limitations

• Digest size dominates proof size.
• The canonical ordering requires a sequential scan from depth $1$ to $d-1$; random access to the $k$-th copath node is $O(k)$. This is not a problem since verification naturally scans the copath in order.

Security / correctness

Formal definitions of $A_j$ (on-path) and $B_j^*$ (minimal copath), the inductive reconstruction argument, and the collision-resistance binding are in Section. 29.2 of Building Cryptographic Proofs from Hash Functions (Chiesa and Yogev, 2024).

References

1. Chiesa, A. and Yogev, E., 2024. Building Cryptographic Proofs from Hash Functions. Section 29. Available at: https://github.com/hash-based-snargs-book/hash-based-snargs-book

---

Before we can merge this PR, please make sure that all the following items have been checked off. If any of the checklist items are not applicable, please leave them but write a little note why.

•  Targeted PR against correct branch (main)
•  Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
•  Wrote unit tests
•  Updated relevant documentation in the code
•  Added a relevant changelog entry to the Pending section in CHANGELOG.md
•  Re-reviewed Files changed in the Github PR explorer

[z-tech]

Little roadmap here about verification

[z-tech]

Maybe think about just one function for the time being that it's not clear what benefit this provides (and this purely readability).