From e9095331f2dbb82d6a1e60ef8cbd819348265dee Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Thu, 22 Oct 2026 12:49:50 +0800 Subject: [PATCH 1/2] docs: record desktop traffic and pending persistence findings Model: gpt-6.1-sol --- ...026-10-22-desktop-network-amplification.md | 88 +++++++++++++++++++ ...-10-22-desktop-network-amplification.zh.md | 88 +++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 .agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md create mode 100644 .agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md diff --git a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md new file mode 100644 index 0000000000..e6cb74a8dc --- /dev/null +++ b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md @@ -0,0 +1,88 @@ +# Desktop network amplification: snapshot traffic and reconnect boundaries + +Status: proposed +Translation: current + +[中文](2026-10-22-desktop-network-amplification.zh.md) + +## Abstract + +Source analysis identifies snapshot-transfer amplification and an idle recovery loop that can repeatedly download full snapshots without retry delays when a missing dependency lies beyond the bootstrap page. These mechanisms reproduce with the exact locked packages but do not establish the cause of a reported overnight traffic total. Ordinary network reconnects retain their cursor, and idle prefetch does not retry itself without another event. Lody also prevents automatic app suspension by default, independently of running tasks, so screen-off does not imply synchronization stopped. Attribute actual request/response bytes before choosing a fix; no product behavior changes in this investigation. + +## Scope and evidence + +Inspected Lody `57bb469c2a31d43b4e9c830017504bc498ee2b5e`, `loro-repo` 0.23.3, `streams-crdt` 0.19.1, `streams-client` 0.12.1 and real Loro 1.16.3. Downloaded npm tarball SHA-512 values match this checkout's lockfile. Experiments use synthetic content, in-memory HTTP boundaries and explicit completion signals; none measures a user's network or the deployed service. The installed app's revision, upload/download split and traffic counter scope remain unknown. + +## Full snapshot upload amplification + +[`createCliStreamsTransport`](../../../../apps/cli/src/lib/loro/streams-transport.ts) enables snapshot uploads with a five-second debounce. Plaintext foreground renderer transport also permits uploads. The locked SDK's `transport-snapshot.ts` and `transport-internal.ts` compare the current stream tail offset with the remote snapshot offset; the default minimum difference is only 100 KiB. Passing that gate exports the entire document snapshot, compresses it, and PUTs it. This is an absolute tail threshold, independent of the existing snapshot's size. + +The timer is scheduled after a successful local append. Remote imports alone do not schedule it. Repeated appends reset the debounce; unchanged offsets and a tail below the threshold do not upload. The manager rechecks the remote snapshot offset and remote-writer observations can suppress uploads. It is therefore incorrect to describe this as an unconditional upload every five seconds. + +A deterministic experiment invokes the exact copied SDK manager with real Loro and its bundled zstd, replacing only HEAD/PUT and the debounce wait. Eight synthetic 128 KiB writes produce 1,049,574 bytes of framed increments and eight full compressed snapshot bodies totaling 115,431,823 bytes. The initial snapshot is 16,777,540 bytes raw and 13,938,153 bytes compressed. Zero tail, a tail below 100 KiB and an unchanged post-upload cursor each produce no upload. These are measured fixture encodings and body sizes, not measured wire traffic or a production upload rate. + +A separate manual-clock experiment verifies debounce reset, no periodic idle upload, suppression after a changed remote version and preservation after an equal-version echo. The threshold counts all stream-tail bytes since the remote snapshot, even though only a successful local append schedules the timer. + +For scale only: 100 MB of successful full-snapshot transfers every 36 seconds totals 80 GB in eight hours. Real sizing must use compressed request/response bytes, not document memory or raw snapshot size. + +## Disposable cloud prefetch repeats bootstrap + +[`runEagerSyncWorkerTask`](../../../../packages/components/src/providers/eager-sync-worker-task.ts) imports its own cached snapshot, then constructs a fresh cloud transport without a matching remote cursor. The SDK creates an empty cursor store and bootstraps again; the cached document's version does not make this bootstrap request incremental. A synthetic real-adapter experiment starts with 50,000 characters and adds one character for each of two subsequent rounds. All three rounds request `/bootstrap` and receive full snapshots, even though the later replicas already hold the preceding state. + +This applies to the cloud worker path. Dual-mode desktop routes its own-machine prefetch through the local transport; remote-machine sessions use cloud, while cloud-only desktop uses cloud for all sessions. Automatic candidates are limited to 20, execution is serial, and the coordinator skips joined/materialized sessions and covered activity checkpoints. Sidebar/visibility/offline gates pause work. New activity or another eligible scheduling event is required; the 15-second freshness value is not an unconditional polling period. + +There is an additional large-document failure path. [`writeEagerSyncSnapshot`](../../../../packages/components/src/providers/eager-sync-snapshot-cache.ts) refuses a raw Loro snapshot larger than 128 MiB. The worker then returns `skipped`, and [`background-sync-coordinator.ts`](../../../../packages/components/src/providers/background-sync-coordinator.ts) records a completed activity checkpoint only for `synced`. Any session metadata event reseeds the bounded candidate list, so another session's metadata changes can requeue the same oversized document even when its own activity timestamp is unchanged. A deterministic coordinator experiment keeps both session timestamps fixed, returns `skipped`, and observes three prefetches of the large session after startup and two other-session metadata events. This is not a single integrated oversized-cloud download test; it combines a cache rejection proven in source, a real-adapter repeated-bootstrap test, and a coordinator scheduling test. The 128 MiB cache limit is raw document encoding, not compressed network size. + +The [serial worker decision](../../implemented/architecture/2026-09-12-session-prefetch-worker.md) deliberately avoids sharing the foreground cursor, and the [bounded window](../../implemented/architecture/2026-09-16-bounded-session-prefetch-window.md) limits scope. Any optimization must persist a worker's own snapshot and corresponding cursor together; copying another replica's cursor can skip missing data. + +## Connection errors and traffic are different evidence + +Chromium defines `ERR_CONNECTION_CLOSED` as a TCP FIN and `ERR_PROXY_CONNECTION_FAILED` as failure to resolve or connect to the proxy itself, explicitly excluding an HTTP CONNECT-stage failure. See [Chromium's error definitions](https://chromium.googlesource.com/chromium/src/+/main/net/base/net_error_list.h). A group of affected API, presence and control streams supports investigating a shared proxy path; it does not identify who closed the earlier connections or prove repeated large transfers. + +In the locked SDK, same-room `rejoin()` retains its cursor and ordinary live-read failures back off. A synthetic test bootstraps at offset 100000, saves offset 100123 in an injected in-memory cursor store, then injects a network failure. The next SSE request uses offset 100123 and the bootstrap count remains one; disk durability is not tested. Missing cursors, server 410 responses and incomplete-import recovery are separate bootstrap paths. The historical [read-loop spin](2026-10-19-streams-read-loop-spin.md) can stop making requests and must not be used as evidence of high network volume. + +Renderer Chromium networking and daemon Node networking also have different proxy owners. [`system-proxy-env.ts`](../../../../apps/electron/src/main/services/system-proxy-env.ts) fills absent child-process proxy variables from Chromium's resolved system proxy at launch. Existing environment values win. This child environment is a startup snapshot, while the renderer uses Chromium's proxy configuration; disabling the environment fallback does not disable the renderer proxy. + +## Idle and sleep boundaries + +[`CliService`](../../../../apps/electron/src/main/services/cli-service.ts) defaults `preventSleepEnabled` to true, applies a persisted override if present, and starts `prevent-app-suspension` without checking whether a task or agent is running. [Electron documents](https://www.electronjs.org/docs/latest/api/power-save-blocker) that this keeps the system active while allowing the display to turn off. [`desktop-log.ts`](../../../../apps/electron/src/main/desktop-log.ts) records suspend/resume but does not pause Streams or refresh proxies. Renderer recovery uses visibility/online events and a 60-second health check; these do not establish that the remote service or selected proxy is reachable. + +The prefetch coordinator has no autonomous retry after `skipped`. A manual-clock experiment finds zero remaining timers after that outcome and no retry during unchanged idle conditions; hidden-to-visible resume reseeds once. A real Repo/Flock experiment also confirms that a same-value metadata write emits no event, while a real PR or read-receipt change on another session can requeue the skipped document. Presence heartbeats use a separate ephemeral channel; unchanged PR polls do not write metadata. Therefore no-task operation weakens the repeated-upload hypothesis, and ordinary idle polling alone is not evidence of repeated full prefetch. + +## Idle incomplete-import recovery can bypass backoff + +In exact `streams-crdt` 0.19.1, `recoverLiveIncompleteRemoteApply` calls `bootstrapState`. If bootstrap is partial, that method calls `catchupOrGone`; when catch-up reaches the tail with missing CRDT dependencies, it calls `bootstrapState` again. The retry remains inside the recovery call and never reaches its outer backoff. The SSE inactivity timer is cleared on entry to the error handler, before recovery. Successful individual requests do not impose a bound on the entire recursive recovery. + +A bounded experiment runs the real SDK read loop, real Loro adapter and bundled zstd with an injected already-joined read-only state and synthetic client responses. The host intentionally omits one causal predecessor and places the dependent update beyond the first bootstrap page. With zero local appends it returns three identical 3,485,325-byte compressed snapshots (10,455,975 bytes total), zero cursor saves and zero backoff timers before the experiment stops the fourth attempt. The control places that same incomplete tail in a complete bootstrap response: one snapshot is read, then backoff is scheduled. This tests the SDK recovery algorithm, not real HTTP, desktop sleep or production stream corruption. + +Partial bootstrap is a supported server behavior, not an invented response shape. The inspected [server at `470ce12`](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/rr/src/protocol.rs#L4187) bounds the retained update page, defaults to a 64 MiB tail budget, and streams the snapshot separately. The [protocol](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/specs/ds-extensions.md#L377) requires ordinary GET continuation after a partial bootstrap. The experiment uses a smaller synthetic page; under the inspected default the bad tail must lie beyond the much larger initial page. The deployed server revision/configuration and the user's missing dependencies remain unverified. + +A second, separate gap accepts a regressing response offset when the current cursor still equals the request offset (`resolveNextOffsetAfterConcurrentAdvance`). A real-SDK synthetic SSE experiment alternates an empty control returning offset zero with existing updates: six reads replay 786,729 framed bytes without local writes or document growth. This requires the remote response to return an older offset; there is no evidence that the user's server or proxy did so. It is not a consequence of sleep alone. + +## Proposed containment and available logs + +### Persist imported pending updates as a separate payload + +Further source analysis and real Loro 1.16.3 experiments confirm a separate persistence gap: ordinary snapshot and full-history update exports omit remotely imported updates parked for missing causal dependencies. With an existing `base|` document, receiving B before its predecessor A leaves B pending. Without restarting, supplying A produces `base|AB`. Restoring only the ordinary snapshot and then supplying A produces `base|A`; restoring the snapshot plus saved original B bytes and then supplying A produces `base|AB`. This concerns imported remote pending changes, not an uncommitted local transaction. + +The separate [Loro implementation](https://github.com/loro-dev/loro/pull/1220), based on `80a0e882141de314b211acd6e30886abe6dcc349`, now provides `exportPendingUpdates(): Uint8Array | undefined` and Rust `export_pending_updates() -> Result>, LoroEncodeError>`. It re-encodes the decoded pending changes with the existing updates format rather than retaining original input bytes. Empty pending returns undefined/None; nonempty bytes restore through ordinary `import`. Export does not consume pending, commit local edits, advance applied versions or emit events. Behavior tests cover partial applied ranges, multiple peers, unknown Lamport values and duplicate/overlapping entries, including counter gaps and restoration after missing dependencies arrive. The optional-byte API is also implemented in `loro-dev/loro-ffi`, with real Swift/Python restoration and typed-error tests. Its review branch pins the core implementation by Git revision for portable validation; a package release must align with a released core version. Rustdoc, JSDoc and `loro-dev/loro-docs` explain consistent checkpoints and their limits. These development changes are unpublished and do not alter this Lody checkout. + +Later Repo/Streams integration must recover the ordinary snapshot and pending payload together. An empty payload must clear obsolete stored pending data. A checkpoint that means "received bytes are durable" is valid only after all corresponding applied and pending data are recoverable; it must remain distinct from applied-version/completeness and must not authorize a complete remote snapshot or log compaction. Current Streams 0.19.1 deliberately withholds durable cursor saves while imports remain incomplete, so the export gap alone does not prove the reporter lost data. Some Repo storage/native paths already retain raw pending journals; the existing Lody Promise document export path does not make ordinary snapshot/update export include pending. Adding a Loro API alone does not change that storage boundary or fix the SDK recovery recursion. Permanently absent remote predecessors still require actual missing data or a usable replacement snapshot; retaining pending cannot manufacture them. + +The proposed SDK repair has two layers: prohibit another bootstrap from a bootstrap's own catch-up after repeated incomplete apply, and bound each room's full-recovery attempts and cumulative download bytes. An unchanged snapshot/tail/missing-dependency fingerprint must not trigger unlimited full reads. A 30-second delay alone is insufficient for large snapshots. Preserve the local document, pending imports, unsent edits and safe cursor; isolate only the affected room. Remote tail changes can supply missing predecessors even without a new snapshot, so low-cost checkpoint observation may permit a bounded new attempt. Wake, token refresh and reconnect alone must not reset the budget. Exhaustion needs an explicit state and recovery action; budget policy remains a proposal, not implemented behavior. + +The selected proposed attempt limit is **one additional full recovery per unresolved missing-dependency episode**, not one HTTP request and not one recovery per reconnect. Normal continuation pages belong to that single recovery. The current outer network policy defaults to unlimited attempts with a 30-second delay cap; the internal bootstrap recursion has no attempt counter at all. A new episode gate must survive internal self-restart, rejoin and replacement of a room session in the same runtime. Merely throwing a terminal error is insufficient because the current read loop self-restarts. Reset the allowance only after missing dependencies are actually resolved and safe catch-up completes, or an explicit repair/retry action grants another attempt; a changed remote offset alone is not proof of repair. Preserving this limit across app restarts would additionally require a durable fault marker and is a separate persistence choice. The user endorsed the one-recovery direction; no code change has been implemented. + +Acceptance should measure transferred fixture bytes through eight hours of fake time, preserve normal cross-page convergence, recover when a missing predecessor or newer snapshot arrives, prevent cursor advancement across incomplete data, and keep independent rooms working. The experiment directly injects an already-joined room and does not establish that the initial join's hard deadline is absent. + +Current logging can reveal the first SSE failure but is not a download ledger. The SDK emits `sse flush chain error` with `CRDT apply incomplete` / `missing deps`. [`installStreamsDiagnostics`](../../../../apps/cli/src/lib/loro/doc.ts) captures `[streams-crdt]` console errors in the CLI file log under `[loro-streams-diag]`; the file sink retains debug by default. Recursive `bootstrapState` / `catchupOrGone` do not log each successful download. `live incomplete apply recovery failed` is emitted only after recovery returns with an error; a long-poll path can enter recursion without the initial SSE log. An absence of these strings cannot exclude the loop, while one incomplete-import error does not establish repeated downloads or their volume. Enabling trace does not create missing request/byte diagnostics. + +Cloud desktop Nightly shares the CLI data root: default `~/.lody/logs/` (Windows `%USERPROFILE%\.lody\logs`), subject to `LODY_DATA_DIR`. Daily files use `YYYY-MM-DD.log`, rotate around 20 MB and retain seven days, including compressed rotations. Desktop logs include build/channel and power `suspend` / `resume`. The in-app Help → Report bug → select machine → Share logs path collects today and yesterday, but limits each day's tail to 4 MiB; older events may require the corresponding rotated files. These are current-source findings; the reporter's complete Nightly build number and bundled SDK still need verification. + +## Investigation decision and limits + +Prioritize per-method/path transferred body totals: request bytes for snapshot PUT/POST and response bytes for bootstrap GET/live reads, including other origins; record stream identity, offsets, recovery attempts, reconnect counts, byte totals and explicit budget-stop reasons without tokens or document content. Separate uploads from downloads and external interfaces from loopback. An error-console count has neither bytes nor retry timing. Current routine diagnostics retain warnings/errors, not a complete byte ledger. + +Automatic updates have timed checks, and managed runtimes reuse installed versions and coalesce same-version downloads; no tight application-owned download loop was found. Hidden public-browser views remain alive, so page-owned media/requests are a conditional alternative. Local preview HTTP/WebSocket traffic may appear in some process counters. Neither is established as the source of the reported total. + +Candidate fixes remain proposed: bounded incomplete-import recovery that reaches backoff, rejection of cursor regression, size-aware snapshot thresholds or a minimum upload interval, replica-bound prefetch checkpoints, and bounded reconsideration of oversized/cache-rejected documents. A skipped download must not be labeled durable or synced. Raising thresholds trades upload cost for more tail replay; removing prefetch trades bandwidth for cold-open latency. First attribute the actual bytes; for an idle report prioritize repeated bootstrap/continuation requests and `apply_incomplete` evidence. No source fix, production capture, deployment or full application check was performed. diff --git a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md new file mode 100644 index 0000000000..526b230be6 --- /dev/null +++ b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md @@ -0,0 +1,88 @@ +# 桌面端流量放大：快照传输与断线重连的边界 + +Status: proposed +Translation: current + +[English](2026-10-22-desktop-network-amplification.md) + +## 摘要 + +源码分析确认了快照传输放大，以及一条空闲恢复循环：缺少依赖的数据位于 bootstrap 首分页之后时，会绕过重试等待、反复下载完整快照。这些机制都用锁定版本做了合成复现，但不能直接解释某位用户的一夜流量总数。普通网络重连会保留读取位置，空闲预取在没有新事件时也不会自行重试。Lody 默认阻止应用自动挂起，且不以有任务为条件，因此熄屏不代表同步停止。应先归因实际请求或响应字节再决定修复；本次没有修改产品行为。 + +## 范围与证据 + +检查版本为 Lody `57bb469c2a31d43b4e9c830017504bc498ee2b5e`、`loro-repo` 0.23.3、`streams-crdt` 0.19.1、`streams-client` 0.12.1，以及真实 Loro 1.16.3。下载的 npm 包 SHA-512 与本仓库锁文件一致。实验使用合成内容、内存中的 HTTP 边界和明确的完成信号，没有测量用户网络或已部署服务。用户安装版本、上传下载比例和统计工具范围仍未知。 + +## 完整快照上传的放大 + +[`createCliStreamsTransport`](../../../../apps/cli/src/lib/loro/streams-transport.ts) 开启快照上传，设置五秒防抖。明文前台 renderer transport 也允许上传。锁定 SDK 的 `transport-snapshot.ts` 和 `transport-internal.ts` 比较当前流末尾位置与远端快照位置，默认差值门槛只有 100 KiB。通过门槛后，导出整份文档快照，压缩后 PUT 上传。这个绝对增量门槛没有随已有快照大小增长。 + +定时器只在本地增量成功追加后调度，单纯接收远端数据不会调度。连续追加会重置防抖；位置不变或增量不足门槛时不上传。上传前还会检查远端快照位置，观察到其他写者也可能抑制上传。因此不能描述成无条件每五秒上传一次。 + +确定性实验直接调用复制出来的原 SDK manager，使用真实 Loro 与包内 zstd，仅替换 HEAD/PUT 和防抖等待。八次各 128 KiB 合成写入产生 1,049,574 字节的带帧增量，却产生八次完整压缩快照，请求体合计 115,431,823 字节。初始快照原始大小为 16,777,540 字节，压缩后为 13,938,153 字节。没有增量、增量不足 100 KiB，以及上传后位置未变，均不触发上传。这些是合成数据的编码和请求体实测，不是线上网络流量或实际上传频率。 + +另一组手动时钟实验验证了防抖重置、空闲不周期上传、远端版本变化后抑制上传，以及相同版本的自身回声不抑制。阈值计算包含远端快照之后所有流尾部字节，但只有本地成功追加才调度定时器。 + +仅用于说明量级：每 36 秒成功传输一次 100 MB 完整快照，八小时合计 80 GB。实际计算必须使用压缩后的请求或响应大小，不能使用文档内存或未压缩快照大小。 + +## 临时云端预取重复下载初始快照 + +[`runEagerSyncWorkerTask`](../../../../packages/components/src/providers/eager-sync-worker-task.ts) 先导入自己的快照缓存，再创建没有对应远端 cursor 的全新云 transport。SDK 创建空的 cursor store 后仍会重新 bootstrap；已缓存文档的版本不会让这次请求变成增量。真实 adapter 的合成实验初始有 50,000 个字符，后两轮各只新增一个字符。三轮都请求 `/bootstrap` 并接收完整快照，即使后两轮副本已持有上一轮状态。 + +这个结论只适用于云端 Worker 路径。双通道桌面端对本机会话的预取走本地 transport，远端机器会话走云端；纯云模式的桌面端则全部走云端。自动候选限制为 20 个，串行执行，并跳过已加入、已有前台资源或活动检查点已覆盖的会话。侧栏、可见性和离线状态会暂停工作。需要新的活动或其他符合条件的调度事件；15 秒新鲜度不是无条件轮询周期。 + +大文档还有一条额外失败路径。[`writeEagerSyncSnapshot`](../../../../packages/components/src/providers/eager-sync-snapshot-cache.ts)拒绝超过 128 MiB 的原始 Loro 快照。Worker 随后返回 `skipped`，而 [`background-sync-coordinator.ts`](../../../../packages/components/src/providers/background-sync-coordinator.ts)只有收到 `synced` 才记录完成的活动检查点。任何会话的 metadata 事件都会重建候选列表，因此其他会话的 metadata 变化也能重新排入同一个超大文档，即使大会话自己的活动时间未变。确定性协调器实验固定两个会话的时间、让预取返回 `skipped`，在启动及两次其他会话 metadata 事件后，观察到大会话被预取三次。这不是一次完整的超大云文档下载实验：证据分别是源码中的缓存拒绝、真实 adapter 的重复 bootstrap，以及协调器的调度实验。128 MiB 缓存门槛指原始文档编码，不是压缩后的网络大小。 + +[串行 Worker 决策](../../implemented/architecture/2026-09-12-session-prefetch-worker.zh.md)明确避免共享前台 cursor，[候选窗口限制](../../implemented/architecture/2026-09-16-bounded-session-prefetch-window.zh.md)约束预取范围。后续优化必须将 Worker 自己的快照与对应 cursor 一起保存；直接复制其他副本的 cursor 会跳过缺失数据。 + +## 连接错误与流量是不同证据 + +Chromium 将 `ERR_CONNECTION_CLOSED` 定义为 TCP FIN，将 `ERR_PROXY_CONNECTION_FAILED` 定义为解析或连接代理本身失败，明确不包括 HTTP CONNECT 阶段失败。参见 [Chromium 错误定义](https://chromium.googlesource.com/chromium/src/+/main/net/base/net_error_list.h)。API、presence、control 多类流一起失败，支持优先检查共用代理路径；它不能定位此前是谁关闭连接，也不能证明反复传输了大数据。 + +锁定 SDK 的同房间 `rejoin()` 保留 cursor，普通 live read 失败会退避。合成实验先从 100000 完成 bootstrap，再将 100123 写入注入的内存 cursor store，随后注入网络失败。下一次 SSE 从 100123 读取，bootstrap 总数仍为一次；实验没有验证磁盘持久化。缺少 cursor、服务端返回 410、恢复未完整导入等是另外的 bootstrap 路径。历史上的 [read-loop spin](2026-10-19-streams-read-loop-spin.zh.md)可能不再发出请求，不能用它解释大流量。 + +renderer 的 Chromium 网络与 daemon 的 Node 网络也分别管理代理。[`system-proxy-env.ts`](../../../../apps/electron/src/main/services/system-proxy-env.ts) 在启动子进程时，从 Chromium 解析出的系统代理补齐缺失的代理环境变量，已有环境变量优先。子进程环境是启动时的快照，而 renderer 使用 Chromium 的代理配置；关闭环境补齐并不会关闭 renderer 代理。 + +## 空闲与休眠边界 + +[`CliService`](../../../../apps/electron/src/main/services/cli-service.ts) 默认 `preventSleepEnabled=true`，存在持久化设置时采用该设置，启动 `prevent-app-suspension` 时不检查是否有任务或 Agent 在运行。[Electron 文档](https://www.electronjs.org/docs/latest/api/power-save-blocker)说明该模式保持系统活跃，但允许屏幕关闭。[`desktop-log.ts`](../../../../apps/electron/src/main/desktop-log.ts) 仅记录 suspend/resume，没有暂停 Streams 或刷新代理。renderer 依靠可见性、online 事件及 60 秒健康检查恢复连接；这些信号不能证明远端服务或选中的代理可达。 + +预取协调器在 `skipped` 后没有自行重试。手动时钟实验确认该结果后没有残留计时器，持续空闲不重试，从隐藏恢复到可见时才重新评估一次。真实 Repo/Flock 实验还确认：同值 metadata 写入不发事件，其他会话真实的 PR 或已读变化可以让被跳过的文档再次入队。Presence 心跳走独立临时通道，结果没变化的 PR 轮询也不写 metadata。因此，没有任务会削弱反复上传的解释；普通空闲轮询本身不能证明反复全量预取。 + +## 空闲时的不完整导入恢复会绕过退避 + +精确 `streams-crdt` 0.19.1 的 `recoverLiveIncompleteRemoteApply` 调用 `bootstrapState`。如果 bootstrap 只包含部分尾部数据，会继续调用 `catchupOrGone`；如果后续读到末尾仍缺少 CRDT 依赖，又会调用 `bootstrapState`。重试一直停留在同一次恢复调用内部，没有到达外层的退避等待。SSE 无活动定时器在进入错误处理时、开始恢复之前已经清除。单个请求成功也不会约束整次递归恢复的持续时间。 + +有界实验使用真实 SDK 读循环、真实 Loro adapter、包内 zstd，注入已加入的只读状态和合成客户端响应。模拟服务故意省略一个前置操作，并将依赖它的 update 放在 bootstrap 首分页之后。在零本地追加的情况下，连续返回三份相同的 3,485,325 字节压缩快照，合计 10,455,975 字节，没有保存 cursor，也没有退避计时器；第 4 次由实验停止。对照把同一坏尾部直接放入完整 bootstrap：只读取一份快照，随后安排退避。这个实验验证 SDK 恢复算法，不验证真实 HTTP、桌面休眠或生产流损坏。 + +分页 bootstrap 是服务端支持的行为，不是虚构响应格式。检查的[服务端 `470ce12`](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/rr/src/protocol.rs#L4187) 限制尾部增量分页，默认预算 64 MiB，快照另行流式返回。[协议](https://github.com/loro-dev/loro-streams/blob/470ce12d9c7e734e2c42cef11a3e93438c4b0c22/specs/ds-extensions.md#L377)要求部分 bootstrap 之后用普通 GET 续读。实验缩小了合成分页；按检查到的默认值，坏尾部必须位于较大的首分页之外。实际部署版本、配置，以及用户是否缺少这些依赖仍未确认。 + +另一处独立缺口是：当前 cursor 仍等于请求 offset 时，`resolveNextOffsetAfterConcurrentAdvance` 会接受响应中倒退的 offset。真实 SDK 的合成 SSE 实验交替返回 offset=0 的空 control 和已有增量，六次读取重复接收 786,729 字节带帧数据，没有本地写入，文档也没有增长。它要求远端响应给出落后的 offset；没有证据表明用户的服务端或代理确实这样做。休眠本身不能直接推出此条件。 + +## 建议的防护与现有日志 + +### 将导入后等待依赖的更新单独持久化 + +进一步源码检查和真实 Loro 1.16.3 实验确认了另一处持久化缺口：普通 snapshot 和全历史 update 导出不包含因缺少因果依赖而停放的远端更新。已有 `base|` 文档先收到 B、缺少前驱 A 时，B 进入 pending；不重启，补 A 后得到 `base|AB`；仅恢复普通 snapshot 再补 A，只得到 `base|A`；恢复 snapshot 并重新导入另存的 B 原始二进制后，再补 A，则得到 `base|AB`。这里指导入远端后的 pending，不是本地未提交事务。 + +独立 [Loro 实现](https://github.com/loro-dev/loro/pull/1220)已在 `80a0e882141de314b211acd6e30886abe6dcc349` 基线上提供 `exportPendingUpdates(): Uint8Array | undefined`，以及 Rust `export_pending_updates() -> Result>, LoroEncodeError>`。它用现有 updates 格式重新编码解码后的 pending changes，不保留原始输入字节。空结果返回 undefined/None；非空二进制可通过普通 `import` 恢复。导出不消费 pending、不提交本地编辑、不推进已应用版本、不发事件。行为测试覆盖部分已应用范围、多 peer、未知 Lamport，以及重复、交叠和 counter 缺口，并验证补齐缺失依赖后收敛。`loro-dev/loro-ffi` 也已实现可选二进制接口，已完成真实 Swift/Python 恢复及类型化错误测试。审查分支固定引用核心 Git revision，供其他机器直接验证；正式发包时仍需对齐已发布的核心版本。Rustdoc、JSDoc 与 `loro-dev/loro-docs` 已补充一致保存检查点的说明和限制。这些开发改动尚未发布，也没有改变当前 Lody checkout。 + +后续 Repo/Streams 接入必须同时恢复普通快照和 pending 数据；空结果应清理已失效的旧 pending 记录。只有对应的已应用数据与 pending 都可恢复后，才能保存表示“已收到字节已落盘”的检查点；它必须与已应用版本、完整性状态区分，不能据此发布完整远端快照或压缩日志。当前 Streams 0.19.1 在导入不完整时刻意不保存持久 cursor，因此单凭导出缺口不能证明该用户丢过数据。Repo 部分存储和原生路径已有原始 pending journal；Lody 现用 Promise 文档导出路径并不会让普通 snapshot/update 包含 pending。仅增加 Loro API 不会自动改变存储边界或修好 SDK 恢复递归。如果远端前驱永久缺失，仍需要补回真实数据或提供可用新快照，保存 pending 本身不能生成缺失操作。 + +建议在 SDK 做两层防护：bootstrap 自己的后续分页再次遇到不完整导入时，禁止再次进入 bootstrap；同时限制每个房间完整恢复的次数和累计下载字节。快照、尾部位置和缺失依赖都没变化时，不应无限重复全量读取。只增加 30 秒等待仍不足以限制大快照的总流量。保留本地文档、pending imports、未上传修改和安全 cursor，只隔离受影响的房间。即使快照没更新，远端尾部也可能补齐缺失前驱，因此可通过低成本的检查点观察，允许预算内的新尝试。唤醒、token 刷新和重连本身不能重置预算。耗尽后需要明确状态及恢复入口；预算策略仍是提议，尚未实现。 + +本次选定的次数方案是：**同一次未解决的缺依赖故障，最多额外做一次完整恢复**，不是只允许一次 HTTP 请求，也不是每次重连各允许一次。正常的后续分页属于这同一次恢复。当前外层网络策略默认无限次尝试、等待上限约 30 秒；内部 bootstrap 递归则没有次数计数。新的故障额度必须跨内部自重启、rejoin，以及同一 runtime 中的房间 session 重建保留。仅抛出终态错误也不够，因为当前读循环会自行重启。只有缺依赖真正补齐、完成安全 catch-up 后自动恢复额度，或者通过明确修复/重试操作授予新的一次机会；远端 offset 变化本身不证明已修复。如果要保证重启应用后也不再重下，还需要持久化故障标记，这是独立的存储选择。用户认可一次恢复的方向，尚未实施代码修改。 + +验收应统计八小时假时钟下的合成传输字节，保留正常跨分页收敛，验证缺失前驱补来或新快照出现后可恢复，保证 cursor 不越过未完整数据，并保证其他房间继续工作。现有实验直接注入已加入的房间，不证明首次 join 缺少硬超时。 + +现有日志能记录最初的 SSE 失败，但不是下载账单。SDK 会输出 `sse flush chain error`，包含 `CRDT apply incomplete` / `missing deps`。[`installStreamsDiagnostics`](../../../../apps/cli/src/lib/loro/doc.ts) 将 CLI 的 `[streams-crdt]` console error 写入文件，前缀为 `[loro-streams-diag]`，文件默认保留 debug。递归的 `bootstrapState` / `catchupOrGone` 没有逐次成功下载日志。只有整个恢复返回错误后才输出 `live incomplete apply recovery failed`；long-poll 路径也可能没有最初的 SSE 错误就进入递归。因此，没搜到这些词不能排除循环，搜到一条不完整导入也不能证明反复下载或流量大小。打开 trace 不会补出原本不存在的请求或字节诊断。 + +云桌面 Nightly 与 CLI 共用数据根，默认主日志为 `~/.lody/logs/`，Windows 为 `%USERPROFILE%\.lody\logs`，可由 `LODY_DATA_DIR` 覆盖。按日期命名 `YYYY-MM-DD.log`，约 20 MB 轮转、保留七天，包含压缩轮转文件。桌面日志记录构建、通道及电源 `suspend` / `resume`。应用内 Help → Report bug → 选择机器 → Share logs 会收集今天和昨天，但每天只取末尾 4 MiB，较早事件可能需要对应轮转文件。以上是当前源码结论，仍需核对用户完整 Nightly 构建号及内置 SDK。 + +## 调查决策与限制 + +优先按方法和路径统计传输体大小：snapshot PUT/POST 看请求体，bootstrap GET、live read 看响应体，并覆盖其他域名；仅记录流标识、offset、恢复次数、重连数、字节总数和明确的预算停止原因，不记录 token 或文档内容。区分上传与下载、外网接口与 loopback。控制台错误条数既没有字节数，也没有重试时间。现有常规诊断保留警告和错误，并不构成完整的流量账单。 + +自动更新按固定间隔检查，托管 runtime 复用已安装版本、合并同版本下载，本次没有发现应用自己快速循环下载。隐藏的公共浏览器 view 仍然存活，因此网页媒体和请求是另一条有条件的候选。本地预览 HTTP/WebSocket 也可能进入某些进程统计。这些都没有被确认为用户流量来源。 + +候选修复仍是提议：限制不完整导入的恢复次数并进入退避、拒绝 cursor 倒退、依据快照大小设置增量门槛或最短上传间隔、为预取副本保存配套检查点，以及限制超大或无法写入缓存的文档再次进入候选的频率。不能把跳过的下载标成已持久化或已同步。提高门槛会减少上传但增加尾部重放，取消预取会减少流量但增加冷打开等待。应先归因实际字节；空闲报告优先检查反复 bootstrap/续读和 `apply_incomplete` 证据。本次没有产品修复、生产抓包、部署或全量应用检查。 From ef183cfc756a3cf275e49eb4c91f100e8c2d95f8 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Thu, 22 Oct 2026 13:29:51 +0800 Subject: [PATCH 2/2] docs: align pending export note with released core fixes Update the bilingual investigation proposal to the rebased Loro core baseline and preserve checkpoint error-reporting requirements. Model: gpt-6.1-sol --- .../bug-fix/2026-10-22-desktop-network-amplification.md | 4 ++-- .../bug-fix/2026-10-22-desktop-network-amplification.zh.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md index e6cb74a8dc..e7549a5c61 100644 --- a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md +++ b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.md @@ -65,9 +65,9 @@ A second, separate gap accepts a regressing response offset when the current cur Further source analysis and real Loro 1.16.3 experiments confirm a separate persistence gap: ordinary snapshot and full-history update exports omit remotely imported updates parked for missing causal dependencies. With an existing `base|` document, receiving B before its predecessor A leaves B pending. Without restarting, supplying A produces `base|AB`. Restoring only the ordinary snapshot and then supplying A produces `base|A`; restoring the snapshot plus saved original B bytes and then supplying A produces `base|AB`. This concerns imported remote pending changes, not an uncommitted local transaction. -The separate [Loro implementation](https://github.com/loro-dev/loro/pull/1220), based on `80a0e882141de314b211acd6e30886abe6dcc349`, now provides `exportPendingUpdates(): Uint8Array | undefined` and Rust `export_pending_updates() -> Result>, LoroEncodeError>`. It re-encodes the decoded pending changes with the existing updates format rather than retaining original input bytes. Empty pending returns undefined/None; nonempty bytes restore through ordinary `import`. Export does not consume pending, commit local edits, advance applied versions or emit events. Behavior tests cover partial applied ranges, multiple peers, unknown Lamport values and duplicate/overlapping entries, including counter gaps and restoration after missing dependencies arrive. The optional-byte API is also implemented in `loro-dev/loro-ffi`, with real Swift/Python restoration and typed-error tests. Its review branch pins the core implementation by Git revision for portable validation; a package release must align with a released core version. Rustdoc, JSDoc and `loro-dev/loro-docs` explain consistent checkpoints and their limits. These development changes are unpublished and do not alter this Lody checkout. +The separate [Loro implementation](https://github.com/loro-dev/loro/pull/1220) has been rebased onto core main `136b944f2501b16eb82ef1c51541627958199d75`, including the published Rust 1.16.2 fixes, and now provides `exportPendingUpdates(): Uint8Array | undefined` and Rust `export_pending_updates() -> Result>, LoroEncodeError>`. It re-encodes the decoded pending changes with the existing updates format rather than retaining original input bytes. Empty pending returns undefined/None; nonempty bytes restore through ordinary `import`. Export does not consume pending, commit local edits, advance applied versions or emit events. Behavior tests cover partial applied ranges, multiple peers, unknown Lamport values and duplicate/overlapping entries, including counter gaps and restoration after missing dependencies arrive. The optional-byte API is also implemented in `loro-dev/loro-ffi`, with real Swift/Python restoration and typed-error tests. Its review Git pin must include those published fixes for portable validation; a package release must still align with a released core version. Rustdoc, JSDoc and `loro-dev/loro-docs` explain consistent checkpoints and their limits. These development changes are unpublished and do not alter this Lody checkout. -Later Repo/Streams integration must recover the ordinary snapshot and pending payload together. An empty payload must clear obsolete stored pending data. A checkpoint that means "received bytes are durable" is valid only after all corresponding applied and pending data are recoverable; it must remain distinct from applied-version/completeness and must not authorize a complete remote snapshot or log compaction. Current Streams 0.19.1 deliberately withholds durable cursor saves while imports remain incomplete, so the export gap alone does not prove the reporter lost data. Some Repo storage/native paths already retain raw pending journals; the existing Lody Promise document export path does not make ordinary snapshot/update export include pending. Adding a Loro API alone does not change that storage boundary or fix the SDK recovery recursion. Permanently absent remote predecessors still require actual missing data or a usable replacement snapshot; retaining pending cannot manufacture them. +Later Repo/Streams integration must recover the ordinary snapshot and pending payload together. An empty payload must clear obsolete stored pending data. The proposed integration should explicitly warn about persistent pending-encoding errors and retain the previous checkpoint rather than silently remaining stuck. A checkpoint that means "received bytes are durable" is valid only after all corresponding applied and pending data are recoverable; it must remain distinct from applied-version/completeness and must not authorize a complete remote snapshot or log compaction. Current Streams 0.19.1 deliberately withholds durable cursor saves while imports remain incomplete, so the export gap alone does not prove the reporter lost data. Some Repo storage/native paths already retain raw pending journals; the existing Lody Promise document export path does not make ordinary snapshot/update export include pending. Adding a Loro API alone does not change that storage boundary or fix the SDK recovery recursion. Permanently absent remote predecessors still require actual missing data or a usable replacement snapshot; retaining pending cannot manufacture them. The proposed SDK repair has two layers: prohibit another bootstrap from a bootstrap's own catch-up after repeated incomplete apply, and bound each room's full-recovery attempts and cumulative download bytes. An unchanged snapshot/tail/missing-dependency fingerprint must not trigger unlimited full reads. A 30-second delay alone is insufficient for large snapshots. Preserve the local document, pending imports, unsent edits and safe cursor; isolate only the affected room. Remote tail changes can supply missing predecessors even without a new snapshot, so low-cost checkpoint observation may permit a bounded new attempt. Wake, token refresh and reconnect alone must not reset the budget. Exhaustion needs an explicit state and recovery action; budget policy remains a proposal, not implemented behavior. diff --git a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md index 526b230be6..88746e3409 100644 --- a/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md +++ b/.agents/notes/proposed/bug-fix/2026-10-22-desktop-network-amplification.zh.md @@ -65,9 +65,9 @@ renderer 的 Chromium 网络与 daemon 的 Node 网络也分别管理代理。[` 进一步源码检查和真实 Loro 1.16.3 实验确认了另一处持久化缺口：普通 snapshot 和全历史 update 导出不包含因缺少因果依赖而停放的远端更新。已有 `base|` 文档先收到 B、缺少前驱 A 时，B 进入 pending；不重启，补 A 后得到 `base|AB`；仅恢复普通 snapshot 再补 A，只得到 `base|A`；恢复 snapshot 并重新导入另存的 B 原始二进制后，再补 A，则得到 `base|AB`。这里指导入远端后的 pending，不是本地未提交事务。 -独立 [Loro 实现](https://github.com/loro-dev/loro/pull/1220)已在 `80a0e882141de314b211acd6e30886abe6dcc349` 基线上提供 `exportPendingUpdates(): Uint8Array | undefined`，以及 Rust `export_pending_updates() -> Result>, LoroEncodeError>`。它用现有 updates 格式重新编码解码后的 pending changes，不保留原始输入字节。空结果返回 undefined/None；非空二进制可通过普通 `import` 恢复。导出不消费 pending、不提交本地编辑、不推进已应用版本、不发事件。行为测试覆盖部分已应用范围、多 peer、未知 Lamport，以及重复、交叠和 counter 缺口，并验证补齐缺失依赖后收敛。`loro-dev/loro-ffi` 也已实现可选二进制接口，已完成真实 Swift/Python 恢复及类型化错误测试。审查分支固定引用核心 Git revision，供其他机器直接验证；正式发包时仍需对齐已发布的核心版本。Rustdoc、JSDoc 与 `loro-dev/loro-docs` 已补充一致保存检查点的说明和限制。这些开发改动尚未发布，也没有改变当前 Lody checkout。 +独立 [Loro 实现](https://github.com/loro-dev/loro/pull/1220)已 rebase 到核心 main `136b944f2501b16eb82ef1c51541627958199d75`，包含已发布的 Rust 1.16.2 修复，并提供 `exportPendingUpdates(): Uint8Array | undefined`，以及 Rust `export_pending_updates() -> Result>, LoroEncodeError>`。它用现有 updates 格式重新编码解码后的 pending changes，不保留原始输入字节。空结果返回 undefined/None；非空二进制可通过普通 `import` 恢复。导出不消费 pending、不提交本地编辑、不推进已应用版本、不发事件。行为测试覆盖部分已应用范围、多 peer、未知 Lamport，以及重复、交叠和 counter 缺口，并验证补齐缺失依赖后收敛。`loro-dev/loro-ffi` 也已实现可选二进制接口，已完成真实 Swift/Python 恢复及类型化错误测试。审查分支固定引用的核心 Git revision 必须包含这些已发布修复，供其他机器直接验证；正式发包时仍需对齐已发布的核心版本。Rustdoc、JSDoc 与 `loro-dev/loro-docs` 已补充一致保存检查点的说明和限制。这些开发改动尚未发布，也没有改变当前 Lody checkout。 -后续 Repo/Streams 接入必须同时恢复普通快照和 pending 数据；空结果应清理已失效的旧 pending 记录。只有对应的已应用数据与 pending 都可恢复后，才能保存表示“已收到字节已落盘”的检查点；它必须与已应用版本、完整性状态区分，不能据此发布完整远端快照或压缩日志。当前 Streams 0.19.1 在导入不完整时刻意不保存持久 cursor，因此单凭导出缺口不能证明该用户丢过数据。Repo 部分存储和原生路径已有原始 pending journal；Lody 现用 Promise 文档导出路径并不会让普通 snapshot/update 包含 pending。仅增加 Loro API 不会自动改变存储边界或修好 SDK 恢复递归。如果远端前驱永久缺失，仍需要补回真实数据或提供可用新快照，保存 pending 本身不能生成缺失操作。 +后续 Repo/Streams 接入必须同时恢复普通快照和 pending 数据；空结果应清理已失效的旧 pending 记录。接入方案还应对持续的 pending 编码错误明确告警，并保留旧检查点，不能静默永久停留。只有对应的已应用数据与 pending 都可恢复后，才能保存表示“已收到字节已落盘”的检查点；它必须与已应用版本、完整性状态区分，不能据此发布完整远端快照或压缩日志。当前 Streams 0.19.1 在导入不完整时刻意不保存持久 cursor，因此单凭导出缺口不能证明该用户丢过数据。Repo 部分存储和原生路径已有原始 pending journal；Lody 现用 Promise 文档导出路径并不会让普通 snapshot/update 包含 pending。仅增加 Loro API 不会自动改变存储边界或修好 SDK 恢复递归。如果远端前驱永久缺失，仍需要补回真实数据或提供可用新快照，保存 pending 本身不能生成缺失操作。 建议在 SDK 做两层防护：bootstrap 自己的后续分页再次遇到不完整导入时，禁止再次进入 bootstrap；同时限制每个房间完整恢复的次数和累计下载字节。快照、尾部位置和缺失依赖都没变化时，不应无限重复全量读取。只增加 30 秒等待仍不足以限制大快照的总流量。保留本地文档、pending imports、未上传修改和安全 cursor，只隔离受影响的房间。即使快照没更新，远端尾部也可能补齐缺失前驱，因此可通过低成本的检查点观察，允许预算内的新尝试。唤醒、token 刷新和重连本身不能重置预算。耗尽后需要明确状态及恢复入口；预算策略仍是提议，尚未实现。