fix(app): rate-limit repeated command error logging - #7372
Open
lbellows wants to merge 1 commit into
Open
Conversation
A failing Tauri command is retried by the frontend, so one persistent fault can log the same error thousands of times per second. Observed on Linux with a saturated SQLite pool: ~4,000-5,000 identical lines/sec, 338k errors in a two-minute window, and 189 MB single-session log files. Feeding that volume of error events into the WebView drives WebKitWebProcess to 11-15 GiB until it is OOM-killed or crashes, leaving the window grey but still closable because the Tauri frame outlives the renderer. Deduplicate by error message: log the first occurrence immediately, then at most once per 5 seconds per distinct message, reporting how many identical occurrences were dropped. Output is unchanged when nothing was suppressed. The suppression table is capped at 256 entries with least-recently-logged eviction, and hashing goes through a Hasher sink rather than allocating a String, so the suppressed path stays cheap under a storm.
Contributor
Pull request changelogAppAddedChangedDeprecatedRemovedFixedSecurityWebsiteAddedChangedDeprecatedRemovedFixedSecurityHostingAddedChangedDeprecatedRemovedFixedSecurity |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #7371.
Problem
A failing Tauri command is retried by the frontend, and the error is logged from inside
SerializeforTheseusSerializableError(apps/app/src/api/mod.rs). So one persistent fault turns into an unbounded log storm rather than a single message.On the machine in #7371 a saturated SQLite pool produced, measured from the launcher logs:
launcher_logs/total (it gzips 43:1, which is itself a measure of how repetitive it is)Pushing that many error events into the WebView drives
WebKitWebProcessto 11–15 GiB until it is OOM-killed or crashes. Because the Tauri frame outlives the renderer, the user is left with a grey window that still has a working titlebar — which is how this presents in the wild, and why it reads as a rendering bug rather than a logging one.What this changes
Deduplicate by rendered error message:
suppressedcount of what was droppedTwo details to keep the fix from becoming its own problem:
Hashersink implementingfmt::Writerather than allocating aString, so the common (suppressed) path does no allocationScope
This is deliberately narrow: it fixes the amplifier, not the underlying fault. The root cause in #7371 is a separate file-descriptor/connection leak — that install had 1,774 open fds on
app.dbafter under four hours, with exactly 100app.db-walhandles matchingmax_connections(100)inpackages/app-lib/src/state/db.rs. I have not diagnosed that half and am not touching it here.I think this is worth doing on its own merits regardless of that: any fault that makes a frequently-retried command fail will take the renderer down through this path, and rate-limiting is the general defence.
Testing
apps/appdoes not currently build on my machine (no webkit2gtk dev headers), so I could not run the crate's own test suite. What I did verify, on rustc 1.95.0 / edition 2024, matchingrust-toolchain.toml:tracingas its only dependency, and 7 unit tests pass — first-occurrence, suppression counting and backlog reporting, independence of distinct messages, table bounding under 4× overload, hash stability, theLazyLock/Mutexwrapper, and reachability of every macro arm#[cfg(test)] mod testsagainstshould_log_in, which is split out fromshould_logprecisely so it is testable without the global tablerustfmt --edition 2024 --config max_width=80 --checkis cleanSo the logic and syntax are verified; the integration against the real
theseus::Erroris not, and I would appreciate CI confirming it.Note on the interval
REPEAT_INTERVALof 5 seconds is a judgement call. It is short enough that a genuinely recurring problem stays visible in the logs and long enough to cut a 5,000/sec storm to 0.2/sec. Happy to change it, or to make it scale with observed rate, if you would rather.