Skip to content

Port upstream 0.66.0: shared Chromium LevelDB + Snappy reader - #646

Closed
Finesssee wants to merge 111 commits into
port/upstream-0.66.0from
port/micro-0.66.0-browser-leveldb-reader
Closed

Finesssee wants to merge 111 commits into
port/upstream-0.66.0from
port/micro-0.66.0-browser-leveldb-reader

Conversation

@Finesssee

Copy link
Copy Markdown
Collaborator

Summary

Adds a shared, read-only LevelDB reader for Chromium browser storage at rust/src/browser/leveldb/, with no new crate (user-approved: hand-written LevelDB log/table parser plus a Snappy block decoder). It is infrastructure only: nothing calls it yet, so there is no behavior change. It unblocks the two upstream 0.66.0 items that were deferred for lack of a LevelDB reader: Kimi Chromium local-storage access_token import (steipete#3923) and MiniMax browser-storage discovery (steipete#3883).

  • read_entries(dir): scans *.log (write-ahead log) and *.ldb/*.sst (sorted tables, prefix-compressed blocks, none/Snappy compression), resolves each user key to its newest sequence, and drops deleted keys.
  • local_storage::read_local_storage_entries(dir, origin): decodes Chromium Local Storage items (_<origin>\0<format byte><key>, Latin-1 or UTF-16LE text) for one exact origin. local_storage_dir(profile_dir) gives <profile>/Local Storage/leveldb.
  • snappy::decompress(input, max_len): raw Snappy blocks with a caller-set output cap.
  • Bounded and tolerant: files over 64 MiB and blocks inflating past 16 MiB are skipped; a truncated live-log tail keeps earlier records; a corrupt file or block is skipped with a tracing::debug line (no key or value content is logged).

Known limits (documented in the module docs): the MANIFEST is not consulted (a compacted-away table not yet deleted can still be read; newer sequences win), checksums are not verified, blocks with compression types other than none/Snappy are skipped, IndexedDB value decoding is out of scope.

Upstream reference

  • Release bullet: Kimi "import web access tokens from Chromium local storage" (fix(kimi): import kimi.ai session from browser local storage steipete/CodexBar#3923, 49e7ff3e) and MiniMax "discover browser session storage" (fix(devin): share Chromium browser session discovery steipete/CodexBar#3883, 62cdd065), v0.66.0.
  • Upstream reads LevelDB through SweetCookieKit's ChromiumLocalStorageReader.readEntries(for: origin, in: levelDBDirectory), an external Swift package, so there is no upstream reader source at the tag to copy. Consumer call sites at v0.66.0: Sources/CodexBarCore/BrowserLocalStorageAPI.swift, Sources/CodexBarCore/Providers/Kimi/KimiCookieImporter.swift (localStorageTokens), Sources/CodexBarCore/ChromiumLocalStorageDiscovery.swift.
  • The on-disk formats implemented are the public LevelDB log/table formats and the Snappy format description; test fixtures are built from those layouts.

Ported / Deferred

Ported: the reader, Chromium Local Storage decoding, Snappy decoder.

Deferred to follow-up micro PRs (queue items 20 and 21):

  • Kimi localStorageTokens (JWT filter, exp check, ordering after cookie sources); needs the Kimi region work from Integrate reviewed provider, history, and tray ports #610.
  • MiniMax storage discovery across the browser catalog (Comet/Yandex are not in BrowserType) and IndexedDB origin-prefix scanning.
  • Browser/profile enumeration helpers for consumers; detection::BrowserProfile.path plus local_storage_dir already covers Local Storage.
  • Checking the reader against a live Chrome/Edge profile: not done in this PR (an automated read of the local browser profile was blocked by the sandbox policy). Fixtures are self-built from the format specs, so a real-profile smoke check is worth doing in the Kimi follow-up.

Validation

Toolchain cargo +1.98.0, slot-4 target dir, E-core wrappers.

  • cargo +1.98.0 fmt --all: clean
  • cargo +1.98.0 clippy --workspace --all-targets -- -D warnings: pass
  • cargo +1.98.0 test -p codexbar leveldb -- --test-threads=4: 24 passed, 0 failed
  • cargo +1.98.0 test -p codexbar browser:: -- --test-threads=4: 39 passed, 0 failed

Tests cover Snappy literal/copy-1/2/4 elements and malformed streams, log framing across block edges and padding, truncated tails, table prefix compression across blocks, Snappy-compressed blocks, bad footer/handle/compression, newest-sequence resolution across log and table, tombstones, and Local Storage Latin-1/UTF-16 decoding with exact-origin matching. I did not run the full cargo test -p codexbar (no shared code touched; only browser/mod.rs gained one pub mod line).

Affected areas

  • Rust backend (rust/src/browser/)
  • Tauri shell
  • Frontend
  • Settings / bridge types
  • No new dependencies. No file over 1000 lines (largest: tests.rs, 625).

UI proof

Not applicable

Finesssee and others added 30 commits September 22, 2026 00:34
@Finesssee

Copy link
Copy Markdown
Collaborator Author

Thermo-nuclear review

Reviewed by Codex gpt-6-luna (xhigh); verified and validated by Claude

  • Medium, rust/src/browser/leveldb/log.rs:82: a malformed write batch could emit earlier operations before a later one failed, and sequence numbers could wrap. Fixed: whole batch validated (including trailing bytes) and sequence range checked before emitting.
  • Medium, rust/src/browser/leveldb/table.rs:94: malformed block entries were treated as normal end-of-block, letting partial records through. Fixed: entry errors propagate, restart array validated, records emitted only after the whole block decodes.
  • Medium, rust/src/browser/leveldb/table.rs:126: corrupt block handles could read across table section boundaries, and uncompressed blocks bypassed the block size limit. Fixed: reads bounded by index/footer offsets, raw blocks capped at MAX_BLOCK_BYTES.
  • Medium, rust/src/browser/leveldb/mod.rs:128: a file could grow after the size check and exceed the read limit. Fixed: read capped at limit + 1 byte and rejected if exceeded.
  • Low, rust/src/browser/leveldb/mod.rs:16: docs implied the per-file limits bound total memory. Fixed: docs now state the snapshot grows with live data.

@Finesssee

Copy link
Copy Markdown
Collaborator Author

Follow-up: all 5 thermo review findings above are fixed in the "Address thermo review" commit. Nothing left open.

Commands run (E-core wrappers, Rust 1.98.0): cargo fmt --all -- --check, cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings (clean), cargo test --manifest-path rust/Cargo.toml --lib browser (47 passed). The change is confined to rust/src/browser/leveldb/, so the Tauri crate clippy and frontend tests were not re-run; no UI change.

Port the remaining upstream 0.63.0 fd2414d conditions: the legacy weekly
counters must be reliable, counters are integers (an invalid used falls
back to a valid remaining balance), and the replacement window is built
from those counters. Translate the upstream KimiRatioPoolTests cases.
Make the new tray panel the default layout
@Finesssee

Copy link
Copy Markdown
Collaborator Author

Adversarial validation (lane-A) at 347faab

Review verdict: the reader matches the 0.66.0 audit row-22/23 shared-infra requirement: a hand-written Chromium LevelDB + Snappy reader under rust/src/browser/leveldb/ (mod.rs dispatch, log.rs WAL/manifest parsing, table.rs sorted tables, snappy.rs block decompression, varint.rs, local_storage.rs user-key extraction; 1353 lines total, 24 focused tests, zero new crates). Blocks with a compression type other than none/Snappy are skipped rather than mis-parsed; each user key resolves through the log and table layers per the module contract.

Merge outcome: the branch absorbed the finished #620 chain (4663218d) so the items stacked on this reader (#651/#649/#709) build on the current build_fetch_context fix. Merge 347faabf, zero conflicts, no code changes to the reader itself.

Checks at 347faab (CARGO_TARGET_DIR=W:\cargo-target\lane-a, jobs=4, RUST_TEST_THREADS=4):

  • cargo fmt --all --check: pass.
  • cargo clippy both manifests --all-targets -- -D warnings: only the 3 documented pre-existing main-drift findings; 0 in this PR's diff.
  • cargo test rust manifest: 2310 passed / 0 failed / 1 ignored (leveldb focused: 24/0).
  • cargo test desktop manifest: 489 passed / 1 failed — bootstrap_payload_exposes_every_provider_variant, the documented Isolate bootstrap payload test from real settings #684 environment-dependent baseline on branches without Make the bootstrap catalog test hermetic (#684) #711 (expected; hermetic fix lives on release/v0.70.0).

Fast-forward pushed 7cfde609..347faabf (ls-remote verified).

Finesssee added a commit that referenced this pull request Oct 2, 2026
…branch

# Conflicts:
#	apps/desktop-tauri/src-tauri/src/commands/mod.rs
@Finesssee

Copy link
Copy Markdown
Collaborator Author

Shipped in v0.70.0: this PR's head is included in main via #735 (merge commit 9d0a37a). Closing as integrated.

@Finesssee Finesssee closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants