Skip to content

fix: ChainGetTipSetByHeight backfills missing headers (Curio StorageGCMark on fresh nodes) - #190

Merged
Reiers merged 2 commits into
mainfrom
fix/tipset-by-height-backfill
Oct 30, 2026
Merged

Reiers merged 2 commits into
mainfrom
fix/tipset-by-height-backfill

Conversation

@Reiers

@Reiers Reiers commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Problem

On a fresh node (calibration, stock Curio as the only chain consumer) Curio's StorageGCMark task fails on every run with

get finality tipset: chain: header/store: not found

tasks/gc/storage_gc_mark.go calls ChainGetTipSetByHeight(head - (WinningPoStSectorSetLookback + EpochsInDay + 1), head.Key()), about 2891 epochs (~24h) back. ChainGetTipSetByHeight only read the header store, and a freshly initialised node only holds headers from its boot anchor forward, so anything older than the node's runtime was "not found". It healed itself after ~24h of uptime, but operators saw errors all through the first day, and other lookback readers hit the same gap.

Fix

New package chain/header/backfill (Resolver), used by ChainGetTipSetByHeight, ChainGetTipSetAfterHeight, ChainGetTipSet and ChainGetBlock when the header store is wired.

  • Hash-linked walk from a trusted tipset, no new trust. The walk starts from the tipset named by tsk (empty = store head) and follows parent links. Missing parent tipsets are fetched by the CIDs their child references. Each header must hash to one of those CIDs (raw bytes and re-encoding both checked), the fetched block set must equal the child's Parents exactly, and heights must strictly decrease. A forged or mismatching header is rejected and never persisted.
  • Sources: ChainExchange (/fil/chain/xchg, the existing peer client, up to 500 tipsets per request, re-verified level by level; the longest verified prefix is kept) when wired, else the content-addressed block getter (bitswap / gateway / RPC fallback), one tipset per hop with the blocks of a tipset fetched concurrently (default 4).
  • Lotus semantics: a null round at h resolves to the first non-null tipset below h for ByHeight and above h for AfterHeight; h above the start tipset is an error; tsk is honoured as the chain to walk (a walk from a non-canonical tip never writes canonical pointers).
  • Persisted: fetched headers go into the header store; for walks from a canonical tipset, canonical pointers are recorded strictly below finality (900 epochs), and stale pointers at epochs that are null rounds on the walked chain are removed. A second call is answered from canonical pointers after checking that they form an unbroken parent-linked run up to the start tipset, so stale islands left by an earlier stale-reset are never served.
  • Bounded: max lookback 2*2880+120 epochs (--header-backfill-lookback), per-call budget 20s (--header-backfill-timeout). The walk runs as a single shared background job (10 min cap). A caller that runs out of budget gets a retryable error (chain: header backfill to height H still in progress (local chain reaches R), retry shortly: ...) instead of "not found", and the job keeps going so the next call finds the headers locally. Concurrent callers join the same job.
  • Startup prefetch: on PDP/Full tiers the daemon prefetches the last ~day of headers (2880+200 epochs) in the background, newest first, 20s after start, so Curio's first GC run is already local. --header-backfill-startup=N overrides the depth, 0 skips it. Light tier stays on demand only.
  • RUN-PDP.md: troubleshooting row for the symptom on nodes younger than ~24h (harmless for PDP, fixed in the next release).

Store additions: SetCanonicalSegment (chunked, refuses unlinked runs, never touches the head), CanonicalLink, HasBlock.

Tests

  • chain/header/backfill: gap backfill by CID with multi-block tipsets and null rounds, persisted + second call needs no fetch; null-round semantics for ByHeight / AfterHeight; hash mismatch rejected and not persisted; forged ChainExchange ancestry rejected with fallback to per-CID fetch; ChainExchange batching; timeout returns a retryable error and the background walk completes; fetch failure is retryable; beyond-lookback; fork tip honoured (no canonical writes from a non-canonical walk); disconnected island ignored and repaired; concurrent callers share one job; startup prefetch.
  • chain/header/store: SetCanonicalSegment (writes below maxHeight only, deletes stale null-epoch pointer, idempotent, rejects unlinked runs, head untouched), CanonicalLink.
  • rpc/handlers: ChainGetTipSetByHeight / AfterHeight / ChainGetTipSet / ChainGetBlock below store coverage through the handler.
  • CGO_ENABLED=0 go build ./..., go vet ./..., go test ./... all green.

Live check (calibration)

Built from this branch, fresh home: lantern init --network calibration --no-wallet, then lantern daemon on loopback with --header-backfill-startup=0 so the on-demand path is what gets measured. After ~3 minutes of uptime, Filecoin.ChainGetTipSetByHeight(head - 2891, head key), compared with a public calibration RPC for the same height:

Lookup 1st call 2nd call Tipset key vs public RPC
head-2891 (StorageGCMark offset) 3072 ms 70 ms match
head-2891, one epoch later 75 ms 70 ms match
head-5000 3128 ms 109 ms match
head-7000 (over the 5880 lookback) 26 ms, error "beyond the header backfill lookback" same n/a (expected)

The first two walks went over ChainExchange (~2600 and ~2100 epochs, about 3.0 s each, per the daemon log). Restarting the daemon on the same home: head-2891 answered in 80 ms, so the headers and canonical pointers persist.

Fallback path only (--no-libp2p, so no ChainExchange or bitswap, block getter = gateway/RPC), ~800-epoch gap: the first calls ran out of the 20 s budget and returned "chain: header backfill to height H still in progress (local chain reaches R), retry shortly: per-call budget 20s exhausted, backfill continues in the background". R kept dropping between calls (about 5 tipsets/s), and about 2 minutes later the same lookup returned in 122 ms with the matching key.

…CMark on fresh nodes)

A fresh node only holds headers from its boot anchor forward, so
ChainGetTipSetByHeight for anything older than the node's runtime
returned header/store: not found. Curio's StorageGCMark asks for
head - (WinningPoStSectorSetLookback + EpochsInDay + 1), about 24h
back, and failed on every run for the node's first day.

New chain/header/backfill resolver: walks parent links back from the
start tipset (head, or the tsk argument), fetching missing parent
tipsets by the CIDs their child references, via ChainExchange runs
when a peer client is wired and the content-addressed block getter
otherwise. Every header must hash to a referenced CID, block sets must
match Parents exactly and heights must decrease, so no new trust is
added. Headers are persisted; canonical pointers are recorded below
finality so the next call is a local read.

Bounded: max lookback (default 2*2880+120, --header-backfill-lookback),
per-call budget (default 20s, --header-backfill-timeout) returning a
retryable error, a single shared background walk that keeps going after
a caller times out, bounded per-tipset fetch concurrency.

Lotus semantics: null round resolves to the first non-null tipset below
(ByHeight) or above (AfterHeight); tsk is the chain walked. ChainGetTipSet
and ChainGetBlock fetch unknown keys/CIDs by CID too.

PDP/Full tiers prefetch the last ~day in the background at startup
(--header-backfill-startup, 0 skips). RUN-PDP.md troubleshooting row.
@Reiers
Reiers merged commit 009d6d0 into main Oct 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant