Skip to content

feat: export pending remote updates for persistence - #1220

Merged
zxch3n merged 5 commits into
mainfrom
lody/export-pending-updates
Oct 22, 2026
Merged

zxch3n merged 5 commits into
mainfrom
lody/export-pending-updates

Conversation

@zxch3n

@zxch3n zxch3n commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Imported remote updates can wait for missing causal dependencies outside the applied oplog. Ordinary snapshot/update exports omit those operations, so saving only a snapshot can lose already-received updates across a restart.

Adds Rust LoroDoc::export_pending_updates() -> Result<Option<Vec<u8>>, LoroEncodeError> and WASM/pure TypeScript exportPendingUpdates(): Uint8Array | undefined. The sidecar uses existing FastUpdates, removes applied prefixes and duplicate overlaps, preserves counter gaps and cross-peer dependencies, and restores through ordinary import after a matching snapshot. Export neither consumes pending operations nor commits local transactions, advances applied versions, or emits events.

Rustdoc/JSDoc and the TypeScript README explain consistent checkpoints, clearing an obsolete sidecar on an empty result, retaining the previous checkpoint on an error, and received-data durability versus complete synchronization. Changesets are included. Transport cursor policy and the reported Lody traffic incident remain separate work.

Review baseline: main 136b944f2501b16eb82ef1c51541627958199d75; current head: d5f9ba31f2c3bc1d6f3ecab44bdd9b9cd2c6e01d. Published Rust 1.16.2 commit af130d97d8b110add13283fc5bfc81d97a9a609f is an ancestor, retaining the released #1103–#1108 fixes. The existing TypeScript import normalization remains unchanged; dependency preservation is confined to the new exporter.

Release verification found and fixed two performance issues:

  • TypeScript pending blocks now keep only the key-table entries they reference, avoiding quadratic output from fragmented inputs sharing a large dictionary. A 1,024-block real-Rust fixture shrank from 4,309,190 to 87,574 exported bytes, matching the WASM encoding. Unknown future payloads conservatively retain their original table.
  • Lazy ordered Map reads now decode directly into their ordered cache, scalar values bypass heap-frame setup, and each read reuses one postcard decoder cursor. Collections retain the iterative decoder and deep-value safety. No wire-format change or untrusted allocation reservation.

Validation on the final source:

  • Rebuilt release WASM: 523 passed, 2 skipped, 2 todo; five entrypoint declarations, CommonJS, typecheck, Deno (4), and Bun (4) passed.
  • Pure TypeScript: 633 passed / 3 skipped; build, typecheck, lint, format passed. One initial timing-only test failed under concurrent load and passed unchanged on rerun; the threshold was not changed.
  • Rust: internal suite 473 passed / 4 ignored, related public suites 74 passed / 1 ignored. Pending coverage includes shallow snapshots, before-shallow-root rejection, duplicate/overlapping ranges, and uncommitted local transactions.
  • WASM/TypeScript interop: 128 missing-update combinations and 30 overlap/local-transaction cases. Published 1.16.3 compatibility: 32 cases / 643 assertions.
  • Independent review additionally exercised 20,000-level values, all scalar kinds, pending restoration, caller-buffer mutation, and encoding error paths.
  • cargo fmt passed. Strict Clippy retains three errors independently reproduced on exact main (question_mark in loro.rs, two doc_lazy_continuation errors in query_by_len.rs). No all-checks-green claim is made. Long fuzzing and the browser matrix were not repeated.

Performance preflight passed after fixing the measured cold-read regression. Final comparison used the same Rust 1.98.1 and Node 22.23.4 as the published-runtime comparison, with 175 independent processes over seven paired rounds. For the 19 MB snapshot workload, published 1.16.4 took 13.183 ms and the candidate 12.547 ms (paired ratio 0.963, empirical 95% interval 0.920–1.015); first Map read was 4.901 vs 4.839 ms. Update import, snapshot export and ordinary checkpoint showed no material regression. Small-map first-read retained an approximately 22 µs difference, and a local-edit microbenchmark approximately 90 ns per commit; the latter was already present on main and was neutral against exact main. These measurements do not prove zero difference for every workload. The deep-value safety tests remain enabled.

Downstream compatibility preflight also passed: four complete, unified pnpm dependency graphs (published 1.16.3, published 1.16.4, and both candidate builds) each passed the original Repo Streams/WebSocket suite, 48/48, using its default config and real feature flags. A temporary alias-based harness failure did not reproduce with unified graphs; no production logic or test threshold was weakened.

Final-head GitHub Test All run 39744491760 passed, and this PR has merged. Official release workflows have published Rust loro/loro-internal/loro-common/loro-kv-store 1.17.0 and npm loro-crdt 1.17.0, loro-crdt-map 1.17.0 and loro.js 0.4.0. Registry tarballs have been downloaded and their checksums/SRI verified; the released WASM API has been called successfully. Release PRs: Rust #1221, JavaScript #1181.

Related approved work: FFI #17, website #54, merged Lody investigation #1815. FFI #17 has merged with the actual crates.io loro 1.17.0 dependency and locked registry checksums; default Rust, generated Python and real Swift tests passed without path overrides. FFI distribution publication is separate.

Add read-only Rust, WASM and pure TypeScript APIs to preserve imported
updates waiting for causal dependencies alongside an ordinary snapshot.

Model: gpt-6.1-sol
Restore main's applied-prefix slicing during TypeScript imports so exact
history validation keeps accepting ordinary sync traffic. Adapt internal
fixtures to the current JSON Result and conflicting-operation checks.

Cover shallow snapshot sidecar recovery and explicit rejection of updates
based before a shallow root in the existing Rust and TypeScript suites.

Model: gpt-6.1-sol
@zxch3n
zxch3n force-pushed the lody/export-pending-updates branch from c48e4c7 to ed71753 Compare October 22, 2026 05:26
@zxch3n
zxch3n marked this pull request as ready for review October 22, 2026 06:30
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

WASM Size Report

  • Original size: 3511.39 KB
  • Gzipped size: 1173.71 KB
  • Brotli size: 821.94 KB

Build pending sidecars from each record's referenced keys instead of
repeating the complete shared input dictionary for every change. Reuse
value-key remapping for nested data and preserve opaque future tables.

Cover real Rust multi-change inputs, linear output bounds, local drafts,
cross-runtime recovery and conservative future-property handling.

Model: gpt-6.1-sol
Decode snapshot-backed Map values directly into the ordered lazy cache and
return scalar values before entering the collection frame loop. Keep nested
values on the existing iterative reader without reserving untrusted lengths.
Cover the existing layout, metadata, duplicate keys, malformed lengths and
truncation, plus deep-value replacement and error cleanup on a small stack.
See context/deep-value-stack-safety.md.

Model: gpt-6.1-sol
Share one postcard deserializer across root keys and iterative value frames
instead of recreating and finalizing a decoder for each scalar. Preserve the
existing wire layout, remaining metadata, malformed-input errors, and deep
value cleanup. See context/deep-value-stack-safety.md.

Model: gpt-6.1-sol
@zxch3n
zxch3n merged commit 1cf1dc5 into main Oct 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant