Repository navigation
feat: export pending remote updates for persistence - #1220
Merged
Merged
Conversation
Add read-only Rust, WASM and pure TypeScript APIs to preserve imported updates waiting for causal dependencies alongside an ordinary snapshot. Model: gpt-6.1-sol
Restore main's applied-prefix slicing during TypeScript imports so exact history validation keeps accepting ordinary sync traffic. Adapt internal fixtures to the current JSON Result and conflicting-operation checks. Cover shallow snapshot sidecar recovery and explicit rejection of updates based before a shallow root in the existing Rust and TypeScript suites. Model: gpt-6.1-sol
zxch3n
force-pushed
the
lody/export-pending-updates
branch
from
October 22, 2026 05:26
c48e4c7 to
ed71753
Compare
zxch3n
marked this pull request as ready for review
October 22, 2026 06:30
Contributor
WASM Size Report
|
Build pending sidecars from each record's referenced keys instead of repeating the complete shared input dictionary for every change. Reuse value-key remapping for nested data and preserve opaque future tables. Cover real Rust multi-change inputs, linear output bounds, local drafts, cross-runtime recovery and conservative future-property handling. Model: gpt-6.1-sol
Decode snapshot-backed Map values directly into the ordered lazy cache and return scalar values before entering the collection frame loop. Keep nested values on the existing iterative reader without reserving untrusted lengths. Cover the existing layout, metadata, duplicate keys, malformed lengths and truncation, plus deep-value replacement and error cleanup on a small stack. See context/deep-value-stack-safety.md. Model: gpt-6.1-sol
Share one postcard deserializer across root keys and iterative value frames instead of recreating and finalizing a decoder for each scalar. Preserve the existing wire layout, remaining metadata, malformed-input errors, and deep value cleanup. See context/deep-value-stack-safety.md. Model: gpt-6.1-sol
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Imported remote updates can wait for missing causal dependencies outside the applied oplog. Ordinary snapshot/update exports omit those operations, so saving only a snapshot can lose already-received updates across a restart.
Adds Rust
LoroDoc::export_pending_updates() -> Result<Option<Vec<u8>>, LoroEncodeError>and WASM/pure TypeScriptexportPendingUpdates(): Uint8Array | undefined. The sidecar uses existing FastUpdates, removes applied prefixes and duplicate overlaps, preserves counter gaps and cross-peer dependencies, and restores through ordinary import after a matching snapshot. Export neither consumes pending operations nor commits local transactions, advances applied versions, or emits events.Rustdoc/JSDoc and the TypeScript README explain consistent checkpoints, clearing an obsolete sidecar on an empty result, retaining the previous checkpoint on an error, and received-data durability versus complete synchronization. Changesets are included. Transport cursor policy and the reported Lody traffic incident remain separate work.
Review baseline: main
136b944f2501b16eb82ef1c51541627958199d75; current head:d5f9ba31f2c3bc1d6f3ecab44bdd9b9cd2c6e01d. Published Rust 1.16.2 commitaf130d97d8b110add13283fc5bfc81d97a9a609fis an ancestor, retaining the released #1103–#1108 fixes. The existing TypeScript import normalization remains unchanged; dependency preservation is confined to the new exporter.Release verification found and fixed two performance issues:
Validation on the final source:
cargo fmtpassed. Strict Clippy retains three errors independently reproduced on exact main (question_markinloro.rs, twodoc_lazy_continuationerrors inquery_by_len.rs). No all-checks-green claim is made. Long fuzzing and the browser matrix were not repeated.Performance preflight passed after fixing the measured cold-read regression. Final comparison used the same Rust 1.98.1 and Node 22.23.4 as the published-runtime comparison, with 175 independent processes over seven paired rounds. For the 19 MB snapshot workload, published 1.16.4 took 13.183 ms and the candidate 12.547 ms (paired ratio 0.963, empirical 95% interval 0.920–1.015); first Map read was 4.901 vs 4.839 ms. Update import, snapshot export and ordinary checkpoint showed no material regression. Small-map first-read retained an approximately 22 µs difference, and a local-edit microbenchmark approximately 90 ns per commit; the latter was already present on main and was neutral against exact main. These measurements do not prove zero difference for every workload. The deep-value safety tests remain enabled.
Downstream compatibility preflight also passed: four complete, unified pnpm dependency graphs (published 1.16.3, published 1.16.4, and both candidate builds) each passed the original Repo Streams/WebSocket suite, 48/48, using its default config and real feature flags. A temporary alias-based harness failure did not reproduce with unified graphs; no production logic or test threshold was weakened.
Final-head GitHub Test All run 39744491760 passed, and this PR has merged. Official release workflows have published Rust
loro/loro-internal/loro-common/loro-kv-store1.17.0 and npmloro-crdt1.17.0,loro-crdt-map1.17.0 andloro.js0.4.0. Registry tarballs have been downloaded and their checksums/SRI verified; the released WASM API has been called successfully. Release PRs: Rust #1221, JavaScript #1181.Related approved work: FFI #17, website #54, merged Lody investigation #1815. FFI #17 has merged with the actual crates.io
loro1.17.0 dependency and locked registry checksums; default Rust, generated Python and real Swift tests passed without path overrides. FFI distribution publication is separate.