Fix pruning issue in SS causing huge disk spike

[alrevuelta]

• Pruning in SS removes old state to keep a recent window of x blocks.
• From Pebble perspective, pruning means iterating all keys and removing the ones older than the retention window (+ keeping some past ones if there were no changes). But deleting does delete right a way. For pebble a deletion is inserting a tombstone value that means that a given key no longer exists. Then Pebble has to "clean that up" removing the old value + the tombstone during compaction and other background tasks.
• The bug: When iterating the keys to prune, the iterator was closed after (via defer) after completing the compaction. But Pebble could not compact because the iterator was active, hence "locking" the data that was supposed to be removed.

left: fix
right: the bug

[cursor]

PR Summary

Medium Risk
Touches the descending MVCC prune path and compaction ordering; behavior change is intentional and localized, but incorrect iterator lifecycle directly affected disk growth and compaction progress.

Overview
Fixes a disk spike during state-store pruning by releasing the Pebble scan iterator before post-prune compaction runs.

In pruneDescending, the full-DB iterator used to stay open until function return (deferred Close), so compactPrunedRange could run for a long time while that iterator still pinned Pebble readState and prevented deletion of obsolete SSTables. The change adds an idempotent closeItr helper, closes the iterator immediately after the scan (before batch commit and compaction), and keeps the defer only for error paths mid-loop.

Also wires seilog for this package and logs an info line after compaction with version, elapsed time, and error.

^{Reviewed by Cursor Bugbot for commit e615a10. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Oct 8, 2026, 9:10 AM

[seidroid]

The fix is correct: deleted keys are cloned via bytes.Clone, so closing the scan iterator before commit and compaction is safe, and it releases the pinned readState before compactPrunedRange runs. The notes below cover structure, log noise, and the same bug still sitting in the ascending path.

Findings: 0 blocking | 4 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] No regression test comes with the change. A test could check that the deferred-close path still works when the loop returns an error. Better, it could assert that obsolete sstables are freed after pruneDescending, for example through db.storage.Metrics() or the table count / disk usage after a prune plus compaction. Either would stop the iterator lifetime from quietly regressing.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.
• 1 non-blocking pre-existing issue(s) listed below under pre-existing issues.

Pre-existing issues

• [suggestion] sei-db/db_engine/pebbledb/mvcc/db_ascending.go (pruneAscending, around line 122) still has the bug this PR fixes. It closes the scan iterator with defer only after compactPrunedRange returns, so nodes on the legacy ascending encoding keep the same sstable pinning and disk spike. Apply the same early close there, ideally through a shared helper. (Also raised by Codex.)

[seidroid]

[suggestion] AGENTS.md ('Structural corrections') asks that this kind of fix be made by naming a step, not by adding an itrOpen flag, a closeItr closure, and two long inline comments. Consider moving the scan into its own function, e.g. scanAndDeletePruned(...) (first, last []byte, err error), that opens the iterator and closes it with a plain defer. pruneDescending then reads as scan → commit → compact, and the iterator is closed before compaction by construction. The note about readState pinning belongs in that function's doc comment. pruneAscending has the same bug and could use the same shape.

[seidroid]

[suggestion] This Info line runs on every prune pass, even when nothing was deleted and compactPrunedRange returned immediately (first == nil). It also logs failures at Info as err=<error>. Consider logging only when a compaction actually ran (firstDeletedKey != nil), and moving failures to Error/Warn or leaving them to the caller, which already receives the returned error.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 184297f. Configure here.}

[cursor]

Ascending prune still pins iterator

Medium Severity

The early iterator close before compactPrunedRange is only applied in pruneDescending. pruneAscending still defers itr.Close() until after compaction, so the scan iterator keeps Pebble's readState pinned and obsolete sstables stay undeletable. Legacy and archive nodes on the ascending path still hit the same disk spike this change targets.

 

^{Triggered by learned rule: sei-db pebbledb: ascending and descending MVCC iterators must stay aligned}

^{Reviewed by Cursor Bugbot for commit 184297f. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 66.44%. Comparing base (f978be0) to head (e615a10).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/db_engine/pebbledb/mvcc/db.go	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4321      +/-   ##
==========================================
- Coverage   67.50%   66.44%   -1.07%     
==========================================
  Files        2181     2060     -121     
  Lines      167848   155789   -12059     
==========================================
- Hits       113306   103511    -9795     
+ Misses      54532    52268    -2264     
  Partials       10       10              

Flag	Coverage Δ
sei-chain-pr	76.01% <92.30%> (?)
sei-db	74.50% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/pebbledb/mvcc/db.go	78.29% <92.30%> (+0.20%)	⬆️

... and 160 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yzang2019]

Nice Finding!

[masih]

/backport

[seidroid]

Successfully created backport PR for release/v6.7:

• Backport release/v6.7: Fix pruning issue in SS causing huge disk spike #4508