Repository navigation
fix: ChainGetTipSetByHeight backfills missing headers (Curio StorageGCMark on fresh nodes) - #190
Merged
Merged
Conversation
…CMark on fresh nodes) A fresh node only holds headers from its boot anchor forward, so ChainGetTipSetByHeight for anything older than the node's runtime returned header/store: not found. Curio's StorageGCMark asks for head - (WinningPoStSectorSetLookback + EpochsInDay + 1), about 24h back, and failed on every run for the node's first day. New chain/header/backfill resolver: walks parent links back from the start tipset (head, or the tsk argument), fetching missing parent tipsets by the CIDs their child references, via ChainExchange runs when a peer client is wired and the content-addressed block getter otherwise. Every header must hash to a referenced CID, block sets must match Parents exactly and heights must decrease, so no new trust is added. Headers are persisted; canonical pointers are recorded below finality so the next call is a local read. Bounded: max lookback (default 2*2880+120, --header-backfill-lookback), per-call budget (default 20s, --header-backfill-timeout) returning a retryable error, a single shared background walk that keeps going after a caller times out, bounded per-tipset fetch concurrency. Lotus semantics: null round resolves to the first non-null tipset below (ByHeight) or above (AfterHeight); tsk is the chain walked. ChainGetTipSet and ChainGetBlock fetch unknown keys/CIDs by CID too. PDP/Full tiers prefetch the last ~day in the background at startup (--header-backfill-startup, 0 skips). RUN-PDP.md troubleshooting row.
This was referenced Oct 31, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a fresh node (calibration, stock Curio as the only chain consumer) Curio's StorageGCMark task fails on every run with
tasks/gc/storage_gc_mark.go calls ChainGetTipSetByHeight(head - (WinningPoStSectorSetLookback + EpochsInDay + 1), head.Key()), about 2891 epochs (~24h) back. ChainGetTipSetByHeight only read the header store, and a freshly initialised node only holds headers from its boot anchor forward, so anything older than the node's runtime was "not found". It healed itself after ~24h of uptime, but operators saw errors all through the first day, and other lookback readers hit the same gap.
Fix
New package chain/header/backfill (Resolver), used by ChainGetTipSetByHeight, ChainGetTipSetAfterHeight, ChainGetTipSet and ChainGetBlock when the header store is wired.
Store additions: SetCanonicalSegment (chunked, refuses unlinked runs, never touches the head), CanonicalLink, HasBlock.
Tests
Live check (calibration)
Built from this branch, fresh home: lantern init --network calibration --no-wallet, then lantern daemon on loopback with --header-backfill-startup=0 so the on-demand path is what gets measured. After ~3 minutes of uptime, Filecoin.ChainGetTipSetByHeight(head - 2891, head key), compared with a public calibration RPC for the same height:
The first two walks went over ChainExchange (~2600 and ~2100 epochs, about 3.0 s each, per the daemon log). Restarting the daemon on the same home: head-2891 answered in 80 ms, so the headers and canonical pointers persist.
Fallback path only (--no-libp2p, so no ChainExchange or bitswap, block getter = gateway/RPC), ~800-epoch gap: the first calls ran out of the 20 s budget and returned "chain: header backfill to height H still in progress (local chain reaches R), retry shortly: per-call budget 20s exhausted, backfill continues in the background". R kept dropping between calls (about 5 tipsets/s), and about 2 minutes later the same lookup returned in 122 ms with the matching key.