Repository navigation
Backport release/v6.7: Fix pruning issue in SS causing huge disk spike - #4508
seidroid[bot] wants to merge 1 commit into
Conversation
* Pruning in SS removes old state to keep a recent window of x blocks. * From Pebble perspective, pruning means iterating all keys and removing the ones older than the retention window (+ keeping some past ones if there were no changes). But deleting does delete right a way. For pebble a deletion is inserting a tombstone value that means that a given key no longer exists. Then Pebble has to "clean that up" removing the old value + the tombstone during compaction and other background tasks. * The bug: When iterating the keys to prune, the iterator was closed after (via defer) after completing the compaction. But Pebble could not compact because the iterator was active, hence "locking" the data that was supposed to be removed. left: fix right: the bug <img width="378" height="339" alt="image" src="https://github.com/user-attachments/assets/6545ea1f-3035-46a5-a424-376926dee7b1" /> (cherry picked from commit 58b7631)
PR SummaryMedium Risk Overview An open Pebble iterator pins readState, which blocks deletion of obsolete SSTables while compaction runs—so tombstoned data could accumulate on disk during long post-prune compactions. A guarded Also adds seilog and an info log for prune compaction duration, version, and error. Reviewed by Cursor Bugbot for commit 86e6496. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
This is a clean backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before compactPrunedRange runs, so the iterator no longer pins Pebble's readState and stops obsolete sstables from being deleted during a long compaction. The guarded double-close is correct, the deleted-key bounds are cloned so closing the iterator early is safe, and the new seilog logger matches how other sei-db packages create theirs. I found no issues.
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- None at the file/PR level.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/v6.7 #4508 +/- ##
================================================
- Coverage 61.84% 60.80% -1.04%
================================================
Files 2168 2068 -100
Lines 188740 176791 -11949
================================================
- Hits 116727 107501 -9226
+ Misses 61159 59406 -1753
+ Partials 10854 9884 -970
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Backport of #4321 to
release/v6.7.