Skip to content

Flat merkle tree - #174

Merged
dryajov merged 20 commits into
mainfrom
feat/merkle-tree-parallelization
Aug 6, 2026
Merged

dryajov merged 20 commits into
mainfrom
feat/merkle-tree-parallelization

Conversation

@dryajov

@dryajov dryajov commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds a flat merkle tree layout on disk, instead of the serialized proof. Proofs are now regenerated on request.

@greptile-apps

greptile-apps Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

Greptile Summary

The PR replaces serialized Merkle-proof persistence with a flat on-disk Merkle-tree layout and regenerates proofs during reads.

  • Adds flat tree indexing, node metadata, persistence, validation, and cleanup.
  • Integrates tree-node movement with temporary-overlay finalization.
  • Adds lifecycle barriers and status transitions around concurrent writes, deletion, and finalization.
  • Updates retrieval, erasure coding, REST error handling, and repository tests.

Confidence Score: 4/5

The PR is not yet safe to merge because internal-node loss still breaks stored-proof retrieval and a failed finalization restoration can strand an overlay permanently.

Missing internal flat nodes are still propagated as retrieval failures before stored proofs can be selected, while a failed move followed by a failed status restoration leaves the temporary overlay Finalizing even though deletion rejects that state.

Files Needing Attention: archivist/stores/repostore/treeops.nim, archivist/stores/repostore/store.nim, archivist/stores/repostore/overlays/overlays.nim

Important Files Changed

Filename Overview
archivist/stores/repostore/treeops.nim Adds flat-tree node persistence, validation, shape resolution, proof regeneration, and tree-node cleanup.
archivist/stores/repostore/overlays/overlays.nim Extends overlay finalization and deletion to move and remove flat-tree records while coordinating writers through lifecycle states.
archivist/stores/repostore/operations.nim Atomically derives and stores flat-tree metadata alongside leaf and block metadata and adds write barriers for lifecycle transitions.
archivist/stores/repostore/store.nim Regenerates proofs from flat nodes for completed manifest-backed datasets while retaining stored proofs for unresolved tree shapes.
archivist/merkletree/flatindex.nim Introduces validated conversions between logical Merkle-tree coordinates and flat storage indices.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Store leaves and proofs] --> B[Derive flat internal nodes]
  B --> C[Persist leaf, block, overlay, and tree metadata]
  C --> D[Finalize temporary overlay]
  D --> E[Attach manifest and mark Completed]
  E --> F[Resolve tree shape on retrieval]
  F --> G[Load flat sibling nodes]
  G --> H[Regenerate and verify proofs]
Loading

Reviews (8): Last reviewed commit: "fix: adapt flat tree serving to Option-p..." | Re-trigger Greptile

Comment thread archivist/stores/repostore/treeops.nim Resolved
@dryajov
dryajov force-pushed the feat/merkle-tree-parallelization branch from 1012f69 to 6ab2172 Compare August 4, 2026 00:03
@dryajov dryajov changed the title Feat/merkle tree parallelization Flat merkle tree Aug 4, 2026
@dryajov
dryajov force-pushed the feat/merkle-tree-parallelization branch from 8d688ce to 11d5102 Compare August 5, 2026 20:05
@dryajov
dryajov marked this pull request as draft August 5, 2026 20:07
@dryajov
dryajov force-pushed the feat/merkle-tree-parallelization branch from 11d5102 to 95177cf Compare August 5, 2026 20:20
@dryajov
dryajov marked this pull request as ready for review August 5, 2026 20:20
@dryajov
dryajov force-pushed the feat/merkle-tree-parallelization branch from 95177cf to 7d70071 Compare August 5, 2026 23:15
@dryajov
dryajov force-pushed the feat/merkle-tree-parallelization branch from 7d70071 to 925467e Compare August 6, 2026 01:27
@dryajov
dryajov merged commit 45f4925 into main Aug 6, 2026
37 checks passed
@dryajov
dryajov deleted the feat/merkle-tree-parallelization branch August 6, 2026 04:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant