Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions pages/docs/api/js.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -596,7 +596,8 @@ type ExportMode =
**⚠️ Important Notes:**

- **Shallow snapshots**: Cannot import updates from before the shallow start point. Peers can only sync if they have versions after this point.
- **Auto-commit**: The document automatically commits pending operations before export.
- **Auto-commit**: The document automatically commits local uncommitted operations before export. Remote pending imports still require their missing causal dependencies.
- **Remote pending updates**: Snapshot and update exports exclude imported operations waiting for missing causal dependencies. See [`exportPendingUpdates()`](#LoroDoc.exportPendingUpdates) to persist them separately.
- **Performance**: Export new snapshots periodically to reduce import times for new peers.

**Examples:**
Expand Down Expand Up @@ -627,6 +628,35 @@ const shallowSnapshot = doc.export({

</Indent>

<Method id="LoroDoc.exportPendingUpdates">
```typescript no_run
exportPendingUpdates(): Uint8Array | undefined
```
</Method>
<Indent>
**Availability:** [`loro-crdt@1.17.0`](https://www.npmjs.com/package/loro-crdt/v/1.17.0) (WASM) and [`loro.js@0.4.0`](https://www.npmjs.com/package/loro.js/v/0.4.0) (pure TypeScript), and later versions.

Exports imported remote operations that are waiting for missing causal predecessors. These are [pending imports](/docs/concepts/import_status#pending-operations), distinct from uncommitted local edits. Ordinary snapshot and update exports do not include them.

**Returns:** `undefined` when no remote operations remain pending; otherwise, binary updates in the existing FastUpdates format. Pass these bytes to ordinary [`import()`](#LoroDoc.import) to restore them. The bytes preserve the pending operations, but need not match the original received blobs byte for byte.

**Behavior and limits:**

- Read-only: does not consume the pending queue, commit a local transaction, emit events, or advance the applied version vector (`appliedVV`) or frontiers.
- Persist a normal snapshot and pending bytes as one consistent checkpoint. Prevent document mutations/imports between the two exports; the calls are not atomic. When the result is `undefined`, clear any older stored pending record. See the [persistence example](/docs/tutorial/persistence#persisting-pending-imports).
- Restore with `doc.import(snapshot)`, then `doc.import(pending)` if pending bytes exist. Operations still wait for their actual missing predecessors; the API cannot fetch or reconstruct predecessors never received from a remote peer.
- Exporting pending bytes, or receiving `undefined`, does not prove the document is synced. Transport cursors, durable storage, and removal of original updates remain the application's responsibility.

The corresponding Rust method is:

```rust
LoroDoc::export_pending_updates(&self) -> Result<Option<Vec<u8>>, LoroEncodeError>
```

`Ok(None)` corresponds to `undefined`; an export error must not be treated as an empty queue. See [`LoroDoc::export_pending_updates` in Rust 1.17.0](https://docs.rs/loro/1.17.0/loro/struct.LoroDoc.html#method.export_pending_updates). The separate [FFI binding source](https://github.com/loro-dev/loro-ffi/pull/17) exposes the corresponding method; check the release of the language binding you use. A core release does not publish downstream binding packages.

</Indent>

<Method id="LoroDoc.import">
```typescript no_run
import(data: Uint8Array): ImportStatus
Expand All @@ -639,7 +669,7 @@ Imports updates or snapshots into the document. Returns an `ImportStatus` descri

- `data` - Binary data or another LoroDoc to import from

**⚠️ Important:** LoroDoc will automatically commits pending operations before import. If the doc is in detached mode, the imported operations are recorded into OpLog but not applied to DocState until you call `attach()`, see [Attached vs Detached States](/docs/concepts/attached_detached) adn [OpLog and DocState](/docs/concepts/oplog_docstate).
**⚠️ Important:** LoroDoc automatically commits local uncommitted operations before import. Remote pending operations can apply only when their missing dependencies arrive. If the doc is in detached mode, the imported operations are recorded into OpLog but not applied to DocState until you call `attach()`, see [Attached vs Detached States](/docs/concepts/attached_detached) and [OpLog and DocState](/docs/concepts/oplog_docstate).

**Example:**

Expand Down
49 changes: 36 additions & 13 deletions pages/docs/concepts/import_status.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,44 +8,67 @@

```ts twoslash
interface ImportStatus {
success: PeerVersionRange; // What was applied
pending?: PeerVersionRange; // What needs dependencies
success: Map<PeerID, CounterSpan>; // What was integrated into history
pending: Map<PeerID, CounterSpan> | null; // What still needs dependencies
}

interface PeerVersionRange {
[peerId: `${number}`]: {
start: number; // Inclusive
end: number; // Exclusive (e.g., 0-50 = ops 0-49)
};
}
type PeerID = `${number}`;
type CounterSpan = {
start: number; // Inclusive
end: number; // Exclusive (e.g., 0-50 = ops 0-49)
};
```

Both fields are present in the WASM and pure TypeScript implementations.
`pending` is `null` when the import reports no pending operations. These ranges
are JavaScript `Map` values: use `.size`, `.get(peerId)`, or iteration;
`Object.keys()` does not enumerate Map entries.

`ImportStatus` reports an import result, not a complete query of the document's
pending queue; `pending === null` does not guarantee that no earlier imports
remain pending. Use [`exportPendingUpdates()`](/docs/api/js#LoroDoc.exportPendingUpdates)
to export all currently received but unintegrated operations; `undefined` means
that queue is empty, not that the document is synced.


## Pending Operations

Operations become pending when they depend on missing operations (causal dependencies).
These are imported remote operations, not local edits waiting for `commit()`.
Normal snapshot and update exports exclude them. The
[`exportPendingUpdates()`](/docs/api/js#LoroDoc.exportPendingUpdates) API exports
them separately for persistence; see the [checkpoint and recovery example](/docs/tutorial/persistence#persisting-pending-imports).
Restoring those bytes with ordinary `import()` preserves the pending operations,
but they apply only after their actual missing dependencies arrive.

### Common Scenario: Out-of-Order Delivery

```ts twoslash
import { LoroDoc } from "loro-crdt";
import { expect } from "expect";
// ---cut---
const doc = new LoroDoc();

// Peer A creates ops 0-4, then 5-10
const peerA = new LoroDoc();
peerA.getText("text").insert(0, "Hello");
const update1 = peerA.export({ mode: "update" });
const afterUpdate1 = peerA.oplogVersion();

peerA.getText("text").insert(5, " World");
const update2 = peerA.export({ mode: "update", from: peerA.version() });
const update2 = peerA.export({ mode: "update", from: afterUpdate1 });

// If update2 arrives first:
const status = doc.import(update2);
console.log(status.pending); // Ops 10-19 pending, need 0-9
expect(status.success).toBeInstanceOf(Map);
expect(status.pending).toBeInstanceOf(Map);
expect(status.pending?.size).toBeGreaterThan(0); // Ops 5-10 need ops 0-4.
expect(doc.getText("text").toString()).toBe("");

// Import missing dependencies:
doc.import(update1); // Both updates now applied
const completed = doc.import(update1); // Both updates now applied
expect(completed.pending).toBeNull();
expect(doc.getText("text").toString()).toBe("Hello World");
```

## Handling Pending Operations
Expand All @@ -60,7 +83,7 @@ async function handleImport(
) {
const status = doc.import(update);

if (status.pending) {
if (status.pending !== null && status.pending.size > 0) {
const missing = await fetchMissing(doc.version());
doc.import(missing);
}
Expand All @@ -78,7 +101,7 @@ const update = new Uint8Array();
// ---cut---
const doc = new LoroDoc();
const status = doc.import(update);
if (status.pending) {
if (status.pending !== null && status.pending.size > 0) {
console.warn("Operations pending:", status.pending);
// Fetch missing updates
}
Expand Down
11 changes: 7 additions & 4 deletions pages/docs/tutorial/encoding.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ Loro introduces three encoding modes to meet the needs for different use cases:

- **Updates Encoding**: Encodes all or a specified range of operations, mainly
used for delta updates of documents.
- **Snapshot Encoding**: Encodes the entire document, including both all the
operations and the current document state.
- **Snapshot Encoding**: Encodes the document's integrated operation history and
current state. Imported operations waiting for missing dependencies are excluded;
see [pending-import persistence](./persistence#persisting-pending-imports).
- **Shallow Snapshot Encoding**: Loro 1.0 provides a new encoding format for
discarding useless historical operations. It is a special snapshot that
encodes the most recent historical operations, the starting state of the
Expand Down Expand Up @@ -87,8 +88,10 @@ doc1.import(bytes);
console.log(doc1.toJSON()); // { text: "hello" }
```

Snapshot Encoding comprehensively captures a document's current state and its
historical operations. This mode can quickly obtain the latest document state.
Snapshot Encoding captures a document's current state and integrated operation
history for fast loading. Like update exports, it excludes imported operations
waiting for missing causal dependencies. See [pending-import persistence](./persistence#persisting-pending-imports)
to preserve those operations across restarts.

## Shallow Snapshot Encoding

Expand Down
6 changes: 6 additions & 0 deletions pages/docs/tutorial/get_started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,9 @@ Loro is a pure library and does not handle network protocols or storage mechanis
To save the document, use `doc.export({mode: "snapshot"})` to get its binary
form. To open it again, use `doc.import(data)` to load this binary data.

Snapshots exclude imported operations waiting for missing dependencies. See
[pending-import persistence](./persistence#persisting-pending-imports) to preserve them across restarts.

```ts twoslash
import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt";
import { expect } from "expect";
Expand Down Expand Up @@ -348,6 +351,9 @@ const update1 = doc.export({ mode: "update", from: lastSavedVersion });
If updates accumulate, exporting a new snapshot can quicken import times and
decrease the overall size of the exported data.

Before discarding the original updates, durably store everything needed for
recovery, including pending imports; see [persistence](./persistence#persisting-pending-imports).

You can store the binary data exported from Loro wherever you prefer.

## Sync
Expand Down
95 changes: 93 additions & 2 deletions pages/docs/tutorial/persistence.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ operations. Here's how it works:
3. Implement a debounce or throttle mechanism to trigger snapshot saves after a
certain number of edits.

Snapshots contain integrated operation history and state, but exclude imported operations
waiting for missing causal dependencies. If your application receives such
updates, retain their original bytes or use the
[pending-import checkpoint](#persisting-pending-imports) described below.

This method simplifies initial application development but has a drawback: user
edits are not immediately saved. Let's explore how to quickly save each user
edit while minimizing resource consumption.
Expand All @@ -24,12 +29,98 @@ To achieve both quick saves and resource efficiency:
- Loro validates the checksum embedded in every snapshot/update header during import, so corrupted binary payloads (for example, from bit flips in queues or object storage) are rejected before they touch your document state.
- When loading a document, import the snapshot and all related updates to get
the latest version.
- After importing, export a new snapshot to replace the old one and remove
imported updates for faster future loading.
- After importing, export a new snapshot for faster future loading. Remove
original updates only after durable storage contains everything needed to
recover them. A snapshot alone does not preserve pending imports: retain their
original bytes, or persist the snapshot together with pending bytes as below.
- If your `LoroDoc` has grown large and older history can be safely recycled,
use `Shallow Snapshot Encoding` to reduce snapshot size. You can archive the
history before the shallow snapshot's start version in cold storage.

### Persisting Pending Imports

[`exportPendingUpdates()`](/docs/api/js#LoroDoc.exportPendingUpdates) is available
since `loro-crdt@1.17.0` (WASM) and `loro.js@0.4.0` (pure TypeScript). The example
below imports the WASM package; use `"loro.js"` for the pure TypeScript implementation.
Here, pending means remote operations already imported but waiting for causal
predecessors, not local edits waiting for `commit()`.

Capture a normal snapshot and the pending bytes without allowing document
mutations or imports between the two calls. Save both as one consistent
checkpoint, using a storage transaction or a versioned generation. Snapshot
export commits local edits as usual; pending export itself is read-only and does
not consume the queue, commit edits, or advance the applied version vector.
Replace the previous checkpoint only after both exports succeed. If the pending
result is `undefined`, delete the previous pending record or explicitly store its
absence; skipping the write would leave stale bytes behind. An export error is
not an empty queue.

This self-contained example delivers B before its dependency A, simulates a
restart, then imports the real A. The `saved` object represents one checkpoint;
an application must write it to durable storage.

```ts no_run
import { LoroDoc } from "loro-crdt";

function assert(condition: unknown, message: string): asserts condition {
if (!condition) throw new Error(message);
}

const source = new LoroDoc();
source.getText("text").insert(0, "base|");
const base = source.export({ mode: "snapshot" });
const beforeA = source.oplogVersion();
source.getText("text").insert(5, "A");
const updateA = source.export({ mode: "update", from: beforeA });
const beforeB = source.oplogVersion();
source.getText("text").insert(6, "B");
const updateB = source.export({ mode: "update", from: beforeB });

const receiver = new LoroDoc();
receiver.import(base);
const status = receiver.import(updateB); // A has not arrived.
assert(status.pending instanceof Map && status.pending.size > 0, "B is pending");
assert(receiver.getText("text").toString() === "base|", "B cannot apply yet");

// Do not interleave document mutations/imports between these exports.
let saved = {
snapshot: receiver.export({ mode: "snapshot" }),
pending: receiver.exportPendingUpdates(),
};
assert(saved.pending !== undefined, "Checkpoint must preserve B");
assert(receiver.exportPendingUpdates() !== undefined, "Export does not consume B");
// Durably replace BOTH fields in one storage transaction/generation here.

const snapshotOnly = new LoroDoc();
snapshotOnly.import(saved.snapshot);
assert(snapshotOnly.exportPendingUpdates() === undefined, "Snapshot excludes B");

const restored = new LoroDoc();
restored.import(saved.snapshot);
restored.import(saved.pending); // Ordinary import; no special restore API.
assert(restored.getText("text").toString() === "base|", "B still needs A");
assert(restored.exportPendingUpdates() !== undefined, "B survives restart");

restored.import(updateA); // The actual missing predecessor finally arrives.
assert(restored.getText("text").toString() === "base|AB", "A unblocks B");
assert(restored.exportPendingUpdates() === undefined, "No pending operations remain");

saved = {
snapshot: restored.export({ mode: "snapshot" }),
pending: restored.exportPendingUpdates(),
};
assert(saved.pending === undefined, "Clear the older pending record when saving");
// Durably replace the checkpoint, including removing the old pending field.
```

The snapshot and pending bytes restore what was received; they cannot supply
predecessors that the remote peer never sent. Even an empty pending queue does
not prove synchronization is complete. Advancing a durable receive cursor or
discarding original update bytes requires that everything needed for recovery is
durably stored first, whether as original updates or a snapshot plus pending bytes.
This API neither defines nor advances a transport cursor; your storage and
transport protocol must define that boundary.

For collaboration, the binary data generated by snapshot/updates can be
transmitted through any medium, such as WebRTC, WebSocket, or HTTP.

Expand Down
Loading
Loading