Skip to content

Backport release/v6.7: Fix pruning issue in SS causing huge disk spike - #4508

Draft
seidroid[bot] wants to merge 1 commit into
release/v6.7from
backport-4321-to-release/v6.7
Draft

seidroid[bot] wants to merge 1 commit into
release/v6.7from
backport-4321-to-release/v6.7

Conversation

@seidroid

@seidroid seidroid Bot commented Oct 8, 2026

Copy link
Copy Markdown

Backport of #4321 to release/v6.7.

* Pruning in SS removes old state to keep a recent window of x blocks.
* From Pebble perspective, pruning means iterating all keys and removing
the ones older than the retention window (+ keeping some past ones if
there were no changes). But deleting does delete right a way. For pebble
a deletion is inserting a tombstone value that means that a given key no
longer exists. Then Pebble has to "clean that up" removing the old value
+ the tombstone during compaction and other background tasks.
* The bug: When iterating the keys to prune, the iterator was closed
after (via defer) after completing the compaction. But Pebble could not
compact because the iterator was active, hence "locking" the data that
was supposed to be removed.

left: fix
right: the bug
<img width="378" height="339" alt="image"
src="https://github.com/user-attachments/assets/6545ea1f-3035-46a5-a424-376926dee7b1"
/>

(cherry picked from commit 58b7631)
@cursor

cursor Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes MVCC descending prune lifecycle timing around Pebble iterators and compaction; low logic change but affects disk reclamation during retention.

Overview
Fixes a state-store pruning disk spike by closing the full-DB scan iterator before compactPrunedRange runs, instead of keeping it open until function return.

An open Pebble iterator pins readState, which blocks deletion of obsolete SSTables while compaction runs—so tombstoned data could accumulate on disk during long post-prune compactions. A guarded closeItr() avoids double-close on errors and still defers cleanup if the scan loop fails early.

Also adds seilog and an info log for prune compaction duration, version, and error.

Reviewed by Cursor Bugbot for commit 86e6496. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedOct 8, 2026, 9:13 AM

@masih
masih marked this pull request as draft October 8, 2026 09:11

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a clean backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before compactPrunedRange runs, so the iterator no longer pins Pebble's readState and stops obsolete sstables from being deleted during a long compaction. The guarded double-close is correct, the deleted-key bounds are cloned so closing the iterator early is safe, and the new seilog logger matches how other sei-db packages create theirs. I found no issues.

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.

@masih masih added the on-hold Ticket is on hold label Oct 8, 2026
@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.80%. Comparing base (fccc24e) to head (86e6496).

Files with missing lines Patch % Lines
sei-db/db_engine/pebbledb/mvcc/db.go 84.61% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           release/v6.7    #4508      +/-   ##
================================================
- Coverage         61.84%   60.80%   -1.04%     
================================================
  Files              2168     2068     -100     
  Lines            188740   176791   -11949     
================================================
- Hits             116727   107501    -9226     
+ Misses            61159    59406    -1753     
+ Partials          10854     9884     -970     
Flag Coverage Δ
sei-chain-pr 64.74% <84.61%> (?)
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/db_engine/pebbledb/mvcc/db.go 67.43% <84.61%> (+0.21%) ⬆️

... and 100 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport on-hold Ticket is on hold

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants