docs: record desktop traffic and pending persistence findings

.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md

@@ -0,0 +1,88 @@
+# Desktop network amplification: snapshot traffic and reconnect boundaries
+
+Status: proposed
+Translation: current
+
+[中文](2026-10-22-desktop-network-amplification.zh.md)
+
+## Abstract
+
+Source analysis identifies snapshot-transfer amplification and an idle recovery loop that can repeatedly download full snapshots without retry delays when a missing dependency lies beyond the bootstrap page. These mechanisms reproduce with the exact locked packages but do not establish the cause of a reported overnight traffic total. Ordinary network reconnects retain their cursor, and idle prefetch does not retry itself without another event. Lody also prevents automatic app suspension by default, independently of running tasks, so screen-off does not imply synchronization stopped. Attribute actual request/response bytes before choosing a fix; no product behavior changes in this investigation.
+
+## Scope and evidence
+
+Inspected Lody `57bb469c2a31d43b4e9c830017504bc498ee2b5e`, `loro-repo` 0.23.3, `streams-crdt` 0.19.1, `streams-client` 0.12.1 and real Loro 1.16.3. Downloaded npm tarball SHA-512 values match this checkout's lockfile. Experiments use synthetic content, in-memory HTTP boundaries and explicit completion signals; none measures a user's network or the deployed service. The installed app's revision, upload/download split and traffic counter scope remain unknown.
+
+## Full snapshot upload amplification
+
+[`createCliStreamsTransport`](../../../../apps/cli/src/lib/loro/streams-transport.ts) enables snapshot uploads with a five-second debounce. Plaintext foreground renderer transport also permits uploads. The locked SDK's `transport-snapshot.ts` and `transport-internal.ts` compare the current stream tail offset with the remote snapshot offset; the default minimum difference is only 100 KiB. Passing that gate exports the entire document snapshot, compresses it, and PUTs it. This is an absolute tail threshold, independent of the existing snapshot's size.
+
+The timer is scheduled after a successful local append. Remote imports alone do not schedule it. Repeated appends reset the debounce; unchanged offsets and a tail below the threshold do not upload. The manager rechecks the remote snapshot offset and remote-writer observations can suppress uploads. It is therefore incorrect to describe this as an unconditional upload every five seconds.
+
+A deterministic experiment invokes the exact copied SDK manager with real Loro and its bundled zstd, replacing only HEAD/PUT and the debounce wait. Eight synthetic 128 KiB writes produce 1,049,574 bytes of framed increments and eight full compressed snapshot bodies totaling 115,431,823 bytes. The initial snapshot is 16,777,540 bytes raw and 13,938,153 bytes compressed. Zero tail, a tail below 100 KiB and an unchanged post-upload cursor each produce no upload. These are measured fixture encodings and body sizes, not measured wire traffic or a production upload rate.
+
+A separate manual-clock experiment verifies debounce reset, no periodic idle upload, suppression after a changed remote version and preservation after an equal-version echo. The threshold counts all stream-tail bytes since the remote snapshot, even though only a successful local append schedules the timer.
+
+For scale only: 100 MB of successful full-snapshot transfers every 36 seconds totals 80 GB in eight hours. Real sizing must use compressed request/response bytes, not document memory or raw snapshot size.
+
+## Disposable cloud prefetch repeats bootstrap
+
+[`runEagerSyncWorkerTask`](../../../../packages/components/src/providers/eager-sync-worker-task.ts) imports its own cached snapshot, then constructs a fresh cloud transport without a matching remote cursor. The SDK creates an empty cursor store and bootstraps again; the cached document's version does not make this bootstrap request incremental. A synthetic real-adapter experiment starts with 50,000 characters and adds one character for each of two subsequent rounds. All three rounds request `/bootstrap` and receive full snapshots, even though the later replicas already hold the preceding state.
+
+This applies to the cloud worker path. Dual-mode desktop routes its own-machine prefetch through the local transport; remote-machine sessions use cloud, while cloud-only desktop uses cloud for all sessions. Automatic candidates are limited to 20, execution is serial, and the coordinator skips joined/materialized sessions and covered activity checkpoints. Sidebar/visibility/offline gates pause work. New activity or another eligible scheduling event is required; the 15-second freshness value is not an unconditional polling period.
+
+There is an additional large-document failure path. [`writeEagerSyncSnapshot`](../../../../packages/components/src/providers/eager-sync-snapshot-cache.ts) refuses a raw Loro snapshot larger than 128 MiB. The worker then returns `skipped`, and [`background-sync-coordinator.ts`](../../../../packages/components/src/providers/background-sync-coordinator.ts) records a completed activity checkpoint only for `synced`. Any session metadata event reseeds the bounded candidate list, so another session's metadata changes can requeue the same oversized document even when its own activity timestamp is unchanged. A deterministic coordinator experiment keeps both session timestamps fixed, returns `skipped`, and observes three prefetches of the large session after startup and two other-session metadata events. This is not a single integrated oversized-cloud download test; it combines a cache rejection proven in source, a real-adapter repeated-bootstrap test, and a coordinator scheduling test. The 128 MiB cache limit is raw document encoding, not compressed network size.
+
+The [serial worker decision](../../implemented/architecture/2026-09-12-session-prefetch-worker.md) deliberately avoids sharing the foreground cursor, and the [bounded window](../../implemented/architecture/2026-09-16-bounded-session-prefetch-window.md) limits scope. Any optimization must persist a worker's own snapshot and corresponding cursor together; copying another replica's cursor can skip missing data.
+
+## Connection errors and traffic are different evidence
+
+Chromium defines `ERR_CONNECTION_CLOSED` as a TCP FIN and `ERR_PROXY_CONNECTION_FAILED` as failure to resolve or connect to the proxy itself, explicitly excluding an HTTP CONNECT-stage failure. See [Chromium's error definitions](https://chromium.googlesource.com/chromium/src/+/main/net/base/net_error_list.h). A group of affected API, presence and control streams supports investigating a shared proxy path; it does not identify who closed the earlier connections or prove repeated large transfers.
+
+In the locked SDK, same-room `rejoin()` retains its cursor and ordinary live-read failures back off. A synthetic test bootstraps at offset 100000, saves offset 100123 in an injected in-memory cursor store, then injects a network failure. The next SSE request uses offset 100123 and the bootstrap count remains one; disk durability is not tested. Missing cursors, server 410 responses and incomplete-import recovery are separate bootstrap paths. The historical [read-loop spin](2026-10-19-streams-read-loop-spin.md) can stop making requests and must not be used as evidence of high network volume.
+
+Renderer Chromium networking and daemon Node networking also have different proxy owners. [`system-proxy-env.ts`](../../../../apps/electron/src/main/services/system-proxy-env.ts) fills absent child-process proxy variables from Chromium's resolved system proxy at launch. Existing environment values win. This child environment is a startup snapshot, while the renderer uses Chromium's proxy configuration; disabling the environment fallback does not disable the renderer proxy.
+
+## Idle and sleep boundaries
+
+[`CliService`](../../../../apps/electron/src/main/services/cli-service.ts) defaults `preventSleepEnabled` to true, applies a persisted override if present, and starts `prevent-app-suspension` without checking whether a task or agent is running. [Electron documents](https://www.electronjs.org/docs/latest/api/power-save-blocker) that this keeps the system active while allowing the display to turn off. [`desktop-log.ts`](../../../../apps/electron/src/main/desktop-log.ts) records suspend/resume but does not pause Streams or refresh proxies. Renderer recovery uses visibility/online events and a 60-second health check; these do not establish that the remote service or selected proxy is reachable.
+
+The prefetch coordinator has no autonomous retry after `skipped`. A manual-clock experiment finds zero remaining timers after that outcome and no retry during unchanged idle conditions; hidden-to-visible resume reseeds once. A real Repo/Flock experiment also confirms that a same-value metadata write emits no event, while a real PR or read-receipt change on another session can requeue the skipped document. Presence heartbeats use a separate ephemeral channel; unchanged PR polls do not write metadata. Therefore no-task operation weakens the repeated-upload hypothesis, and ordinary idle polling alone is not evidence of repeated full prefetch.
+
+## Idle incomplete-import recovery can bypass backoff
+
+In exact `streams-crdt` 0.19.1, `recoverLiveIncompleteRemoteApply` calls `bootstrapState`. If bootstrap is partial, that method calls `catchupOrGone`; when catch-up reaches the tail with missing CRDT dependencies, it calls `bootstrapState` again. The retry remains inside the recovery call and never reaches its outer backoff. The SSE inactivity timer is cleared on entry to the error handler, before recovery. Successful individual requests do not impose a bound on the entire recursive recovery.
+
+A bounded experiment runs the real SDK read loop, real Loro adapter and bundled zstd with an injected already-joined read-only state and synthetic client responses. The host intentionally omits one causal predecessor and places the dependent update beyond the first bootstrap page. With zero local appends it returns three identical 3,485,325-byte compressed snapshots (10,455,975 bytes total), zero cursor saves and zero backoff timers before the experiment stops the fourth attempt. The control places that same incomplete tail in a complete bootstrap response: one snapshot is read, then backoff is scheduled. This tests the SDK recovery algorithm, not real HTTP, desktop sleep or production stream corruption.
+
+Partial bootstrap is a supported server behavior, not an invented response shape. The inspected [server at `470ce12`](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/rr/src/protocol.rs#L4187) bounds the retained update page, defaults to a 64 MiB tail budget, and streams the snapshot separately. The [protocol](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/specs/ds-extensions.md#L377) requires ordinary GET continuation after a partial bootstrap. The experiment uses a smaller synthetic page; under the inspected default the bad tail must lie beyond the much larger initial page. The deployed server revision/configuration and the user's missing dependencies remain unverified.
+
+A second, separate gap accepts a regressing response offset when the current cursor still equals the request offset (`resolveNextOffsetAfterConcurrentAdvance`). A real-SDK synthetic SSE experiment alternates an empty control returning offset zero with existing updates: six reads replay 786,729 framed bytes without local writes or document growth. This requires the remote response to return an older offset; there is no evidence that the user's server or proxy did so. It is not a consequence of sleep alone.
+
+## Proposed containment and available logs
+
+### Persist imported pending updates as a separate payload
+
+Further source analysis and real Loro 1.16.3 experiments confirm a separate persistence gap: ordinary snapshot and full-history update exports omit remotely imported updates parked for missing causal dependencies. With an existing `base|` document, receiving B before its predecessor A leaves B pending. Without restarting, supplying A produces `base|AB`. Restoring only the ordinary snapshot and then supplying A produces `base|A`; restoring the snapshot plus saved original B bytes and then supplying A produces `base|AB`. This concerns imported remote pending changes, not an uncommitted local transaction.
+
+The separate [Loro implementation](https://github.com/loro-dev/loro/pull/1220) has been rebased onto core main `136b944f2501b16eb82ef1c51541627958199d75`, including the published Rust 1.16.2 fixes, and now provides `exportPendingUpdates(): Uint8Array | undefined` and Rust `export_pending_updates() -> Result<Option<Vec<u8>>, LoroEncodeError>`. It re-encodes the decoded pending changes with the existing updates format rather than retaining original input bytes. Empty pending returns undefined/None; nonempty bytes restore through ordinary `import`. Export does not consume pending, commit local edits, advance applied versions or emit events. Behavior tests cover partial applied ranges, multiple peers, unknown Lamport values and duplicate/overlapping entries, including counter gaps and restoration after missing dependencies arrive. The optional-byte API is also implemented in `loro-dev/loro-ffi`, with real Swift/Python restoration and typed-error tests. Its review Git pin must include those published fixes for portable validation; a package release must still align with a released core version. Rustdoc, JSDoc and `loro-dev/loro-docs` explain consistent checkpoints and their limits. These development changes are unpublished and do not alter this Lody checkout.
+
+Later Repo/Streams integration must recover the ordinary snapshot and pending payload together. An empty payload must clear obsolete stored pending data. The proposed integration should explicitly warn about persistent pending-encoding errors and retain the previous checkpoint rather than silently remaining stuck. A checkpoint that means "received bytes are durable" is valid only after all corresponding applied and pending data are recoverable; it must remain distinct from applied-version/completeness and must not authorize a complete remote snapshot or log compaction. Current Streams 0.19.1 deliberately withholds durable cursor saves while imports remain incomplete, so the export gap alone does not prove the reporter lost data. Some Repo storage/native paths already retain raw pending journals; the existing Lody Promise document export path does not make ordinary snapshot/update export include pending. Adding a Loro API alone does not change that storage boundary or fix the SDK recovery recursion. Permanently absent remote predecessors still require actual missing data or a usable replacement snapshot; retaining pending cannot manufacture them.
+
+The proposed SDK repair has two layers: prohibit another bootstrap from a bootstrap's own catch-up after repeated incomplete apply, and bound each room's full-recovery attempts and cumulative download bytes. An unchanged snapshot/tail/missing-dependency fingerprint must not trigger unlimited full reads. A 30-second delay alone is insufficient for large snapshots. Preserve the local document, pending imports, unsent edits and safe cursor; isolate only the affected room. Remote tail changes can supply missing predecessors even without a new snapshot, so low-cost checkpoint observation may permit a bounded new attempt. Wake, token refresh and reconnect alone must not reset the budget. Exhaustion needs an explicit state and recovery action; budget policy remains a proposal, not implemented behavior.
+
+The selected proposed attempt limit is **one additional full recovery per unresolved missing-dependency episode**, not one HTTP request and not one recovery per reconnect. Normal continuation pages belong to that single recovery. The current outer network policy defaults to unlimited attempts with a 30-second delay cap; the internal bootstrap recursion has no attempt counter at all. A new episode gate must survive internal self-restart, rejoin and replacement of a room session in the same runtime. Merely throwing a terminal error is insufficient because the current read loop self-restarts. Reset the allowance only after missing dependencies are actually resolved and safe catch-up completes, or an explicit repair/retry action grants another attempt; a changed remote offset alone is not proof of repair. Preserving this limit across app restarts would additionally require a durable fault marker and is a separate persistence choice. The user endorsed the one-recovery direction; no code change has been implemented.
+
+Acceptance should measure transferred fixture bytes through eight hours of fake time, preserve normal cross-page convergence, recover when a missing predecessor or newer snapshot arrives, prevent cursor advancement across incomplete data, and keep independent rooms working. The experiment directly injects an already-joined room and does not establish that the initial join's hard deadline is absent.
+
+Current logging can reveal the first SSE failure but is not a download ledger. The SDK emits `sse flush chain error` with `CRDT apply incomplete` / `missing deps`. [`installStreamsDiagnostics`](../../../../apps/cli/src/lib/loro/doc.ts) captures `[streams-crdt]` console errors in the CLI file log under `[loro-streams-diag]`; the file sink retains debug by default. Recursive `bootstrapState` / `catchupOrGone` do not log each successful download. `live incomplete apply recovery failed` is emitted only after recovery returns with an error; a long-poll path can enter recursion without the initial SSE log. An absence of these strings cannot exclude the loop, while one incomplete-import error does not establish repeated downloads or their volume. Enabling trace does not create missing request/byte diagnostics.
+
+Cloud desktop Nightly shares the CLI data root: default `~/.lody/logs/` (Windows `%USERPROFILE%\.lody\logs`), subject to `LODY_DATA_DIR`. Daily files use `YYYY-MM-DD.log`, rotate around 20 MB and retain seven days, including compressed rotations. Desktop logs include build/channel and power `suspend` / `resume`. The in-app Help → Report bug → select machine → Share logs path collects today and yesterday, but limits each day's tail to 4 MiB; older events may require the corresponding rotated files. These are current-source findings; the reporter's complete Nightly build number and bundled SDK still need verification.
+
+## Investigation decision and limits
+
+Prioritize per-method/path transferred body totals: request bytes for snapshot PUT/POST and response bytes for bootstrap GET/live reads, including other origins; record stream identity, offsets, recovery attempts, reconnect counts, byte totals and explicit budget-stop reasons without tokens or document content. Separate uploads from downloads and external interfaces from loopback. An error-console count has neither bytes nor retry timing. Current routine diagnostics retain warnings/errors, not a complete byte ledger.
+
+Automatic updates have timed checks, and managed runtimes reuse installed versions and coalesce same-version downloads; no tight application-owned download loop was found. Hidden public-browser views remain alive, so page-owned media/requests are a conditional alternative. Local preview HTTP/WebSocket traffic may appear in some process counters. Neither is established as the source of the reported total.
+
+Candidate fixes remain proposed: bounded incomplete-import recovery that reaches backoff, rejection of cursor regression, size-aware snapshot thresholds or a minimum upload interval, replica-bound prefetch checkpoints, and bounded reconsideration of oversized/cache-rejected documents. A skipped download must not be labeled durable or synced. Raising thresholds trades upload cost for more tail replay; removing prefetch trades bandwidth for cold-open latency. First attribute the actual bytes; for an idle report prioritize repeated bootstrap/continuation requests and `apply_incomplete` evidence. No source fix, production capture, deployment or full application check was performed.