Conversation
Per-write pod quota on the CSS file backend recursively walks the whole pod per stream chunk (chunks × O(N)), causing extreme slowness and memory exhaustion on large inboxes. This adds two pivot components wired via config Overrides (no CSS fork): - DuSizeReporter: SizeReporter measuring apparent bytes (du -sb / BSD -s -A -B 1) with a per-path TTL cache, Node-walk fallback when du is unavailable (Windows), and invalidate() to drop resource + ancestor (pod root) cache entries after writes. - FastQuotaStrategy: extends PodQuotaStrategy; computes available space ONCE per write and only tracks the write's own byte delta per chunk (was: full pod walk per chunk). Config: config/storage/backend/quota-fast-file.json overrides urn:solid-server:default:SizeReporter + QuotaStrategy (70 MB, apparent bytes, ignoreFolders ^/\.internal$, ttl 5000); imported from customise-me.json. Benchmark (5000 files/1KB pod, 4MB write in 64KB chunks): guard 36 279 ms -> 3.3 ms (~11 000x); cached getSize 0.13 ms. Equivalence proof: 12/12 random trees byte-identical to FileSizeReporter (du path + Node fallback). Includes unit tests and scripts/benchmark-quota.js + scripts/verify-size-equivalence.js.
Moved from docs/ (which is gitignored) to the repo root so it is tracked. Documents the problem (per-chunk pod walks), the A+B design (DuSizeReporter + FastQuotaStrategy), the decisions (apparent bytes, persist-per-write, A+B first), staleness/recovery (SS4.8), and the verification section SS7 (benchmark ~11 000x guard speedup + 12/12 byte-identical equivalence).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.