[Reiers]

Stock Curio-PDP 1.29.0 using Lantern v1.10.1-rc1 as its only chain node on calibration (our soak, and the first external operator) has two symptoms:

1. eth_calls time out during Curio's periodic read bursts: failed to process empty PDP proving periods: failed to get next challenge epoch for data set N: chain: read tcp 127.0.0.1:x->127.0.0.1:<port>: i/o timeout. In our soak that was about 256 per 24 h with ~29 data sets, and ~15% of PDPv0_Prove / PDPv0_ProvPeriod attempts were retried.
2. Curio's Chain page often shows Lantern as Sync Status: No, slow (55s behind) while Lotus on the same box is in sync.

This PR fixes both. Root causes, Curio's own definitions and before/after numbers below.

Problem 1: read tcp ...: i/o timeout on eth_call

Curio's timeouts (v1.29.0):

• Curio's eth client is go-ethereum ethclient over a websocket (deps/apiinfo.go GetEthClient: erpc.DialOptions(ws://.../rpc/v1); only the first configured endpoint is used).
• Per call: every PDP contract call goes through contract.EthCallOpts(ctx), a 10 s context (pdp/contract/ethcall.go, EthCallTimeout = 10 * time.Second). When that fires the error is context deadline exceeded, not i/o timeout.
• Per connection: go-ethereum v1.17.6 rpc/websocket.go: after 30 s without a client write (wsPingInterval) the client sends a ping and sets the connection read deadline 30 s ahead (wsPongTimeout); only a pong clears it. When it expires the read loop fails with read tcp ...: i/o timeout, the connection is dropped and every request in flight on it fails with that error, however fast the server is.

Root cause: Lantern's RPC server is go-jsonrpc v0.10.1, whose websocket ping handler records the ping as activity and does not reply with a pong (v0.10.3 does). Every Curio eth connection that sat idle for 30 s between bursts was torn down 30 s later, and whatever eth_calls were in flight at that moment failed with exactly the error operators see. Curio's work is periodic with idle gaps, so this hits a steady fraction of calls.

Reproduced with go-ethereum's own client (curioburst -ws-idle-probe: a newHeads subscription stays open while the client writes nothing):

before: ws-idle-probe: FAIL at t+60s after 2 heads: read tcp 127.0.0.1:55023->127.0.0.1:12378: i/o timeout
after:  ws-idle-probe: OK (connection survived the keepalive ping, 11 heads received)

Fix: go-jsonrpc v0.10.3 (answers pings with a protocol-level pong). Regression test rpc/server/ws_keepalive_test.go.

Burst latency (secondary). Before, every eth_call on a new head re-walked the state tree on its own, and a cold block was fetched once per caller. The eth_call layer now has:

• content- and state-keyed caches: blocks (CID-verified), actors per (state root, address), decoded EVM states, bytecode, storage slots per (storage root, slot), and results per execution state (execution tipset key + state root; the two most recent states are kept);
• singleflight in front of block fetches, actor resolution and identical calls, on a context detached from the first caller, so one client giving up never fails the others;
• a head warmer: the actors and slots read on the previous head are re-read at the new state root, concurrently, the moment the head lands;
• an f410 -> actor ID hint that skips the Init actor walk, used only when the actor at that ID in the current state carries exactly that delegated address.

Nothing is keyed by "latest": every entry is keyed by content address or by the exact state it was computed on, so a new head can never be served an older head's value.

Problem 2: "slow (55s behind)"

Curio's definition (web/api/webrpc/sync_state.go, SyncerState): age = now - ChainHead().MinTimestamp(); ok if age < 1.5 * BlockDelaySecs (45 s), slow (<age> behind) if age < 5 * BlockDelaySecs (150 s), else behind. A node at the tip shows ages of roughly 3 to 33 s (epoch timestamp plus propagation, up to one epoch). 55 s means one epoch behind.

Root cause (Lantern side): net/blockingest built the head from gossip blocks one at a time and dropped every block at or below the head height (bh.Height <= curHead). So:

• the head was always a one-block tipset, and the other blocks of that tipset were never stored;
• the next epoch's blocks list the whole tipset as Parents, so allParentsPresent failed; the inline backfill only fills epochs above the store head, so it did nothing and the block was skipped;
• the head moved only when a block two epochs up arrived and its missing parent epoch was backfilled. On calibration, where most tipsets have several blocks, Lantern advanced every other epoch, one epoch behind.

Live evidence from the baseline run: Lantern head heights 4205317, 4205319, 4205321, 4205323, ... (burst log below), and a side-by-side sample: Lantern head 4205321 with 1 block while Glif was at 4205322 with 4 blocks; Lantern then jumped straight to 4205323.

It is not #180 (ChainHead returns the header store head directly; latest = head-1 only applies to eth block tags), not head validation or state fetch (the gossip head path is header arithmetic), and not the polling Sync interval (inactive bridge-off).

Fix (net/blockingest):

• A block at the head height that shares the head's parent-derived fields (Parents, ParentStateRoot, ParentMessageReceipts, ParentWeight, which every block of one tipset shares) is stored, and the head tipset grows to include it, through the same divergence and corroboration gates as any head adoption. That is Lotus' behaviour: the head is the heaviest tipset, and a tipset gets heavier as its blocks arrive. ChainNotify emits the usual revert/apply for the grown tipset, as Lotus does. A competing block at the same height (different parent fields) is ignored; fork choice still happens on its children by ParentWeight.
• A head+1 block whose parent tipset contains a block we never received (late or missed gossip) fetches exactly those parent headers (ChainExchange, else per-CID), checks the re-derived CIDs against its Parents and the shared parent fields, and is adopted in the same epoch instead of skipped.
• No check is weakened: CID re-derivation, the fence for lower heights, fork choice, the weight-monotonic guard, the divergence gate and the corroboration gate all still apply.

Numbers

Local, calibration, a fresh LANTERN_HOME per build (init --network calibration --no-wallet, node-type pdp, bridge-off, loopback listener), same machine and time window. Before = origin/main (009d6d0), after = this branch. scripts/bench/curioburst: one shared go-ethereum websocket client like Curio's; per burst 30 live data sets x (getNextChallengeEpoch, getDataSetLeafCount, getChallengeRange) = 90 concurrent eth_calls, plus 8 receipt polls, eth_getBlockByNumber latest, eth_blockNumber and StateGetRandomnessDigestFromBeacon; 5 s client timeout; head age sampled every 5 s.

A. Burst every 30 s for 10 min (as specified; includes the first burst on a node started seconds earlier):

eth_call, per view	n	p50	p99	max	errors at 5 s
before	570	2642 ms	5000 ms	5000 ms	30 (5.3%)
after	600	1.6 ms	5000 ms	5000 ms	30 (5.0%)

In both runs all 30 errors per view come from the first burst on the cold node. Burst wall time before: 3.2 to 4.3 s on a new head, ~80 ms on a repeat (the head only moved every 60 s); after: 3 to 14 ms for all 19 later bursts (the 30 s phase lands after the head warmer finished). eth_getBlockByNumber latest max 3368 ms before, 14 ms after; receipts p50 6.8 ms before, 1.7 ms after.

B. Head-triggered bursts for 5 min (a burst within 2 s of every head change, like Curio's chain scheduler; first burst skipped as warmup; both nodes side by side):

eth_call, per view	n	p50	p99	max	errors at 5 s
before	150	4408 to 4963 ms	5002 ms	5002 ms	59 to 62 timeouts + 30 read tcp ...: i/o timeout (~60%)
after	270	1550 to 1572 ms	2196 to 2282 ms	2780 ms	0

This is the case that matters for Curio: the burst lands on a state root nobody has read yet. Before, the 30 i/o timeout errors are the websocket teardown from Problem 1 reproducing inside a normal burst. After, the first burst on a new head is bounded by the chain of dependent block fetches (state tree path, EVM actor head, storage KAMT path: about a dozen sequential Bitswap round trips), done once and shared by all 90 calls; every later call on that state is served from cache in milliseconds (table A).

The long pole of an after burst is now eth_getBlockByNumber("latest") on a fresh head: p50 4193 ms, one 5 s timeout in 9 (before: p50 4849 ms, 2 of 5 failed). It walks the receipts, events and messages of the newest executed tipset from the network and is unchanged here; it is under Curio's 10 s and is the next thing to look at.

C. Head age (now - ChainHead MinTimestamp), during the 10 min runs:

	samples	p50	p95	max	Curio "slow" (>= 45 s)
before	122	33.9 s	58.9 s	58.9 s	30 (24.6%)
after	121	18.0 s	28.0 s	28.1 s	0
Glif (reference, during the after run)	121	33.0 s	53.0 s	58.0 s

Glif's own ChainHead trails by up to an epoch, so wall-clock age is the meaningful number; the after node was ahead of Glif in 66 of 121 samples and never behind it (before: behind in 33, level 60, ahead 29).

Correctness

• eth_call results identical to Glif at the same block: the bench reads the node's latest block number, runs all 90 views, re-reads latest (discarding the round if the head moved), then asks Glif for the same views at that pinned block number. Before: 90/90 identical (block 0x402b18). After: 90/90 identical (block 0x402b22).
• Cache invalidation tests (rpc/handlers/ethcall_perf_test.go): a new head with a new state root is never served the previous result; the same state root with a different execution tipset is not reused either (NUMBER/TIMESTAMP differ); reverts replay as reverts; the f410 hint is re-verified against the current state; bad bytes from the network are rejected and never cached.
• Singleflight tests: a 48-caller burst on a fresh head fetches every block exactly once and runs the interpreter once per distinct call; a caller that cancels does not fail the others.
• Head tests (net/blockingest/sibling_test.go): a sibling completes the head and the next epoch adopts immediately without any backfill source; a competing block at the head height is neither stored nor merged; a closed divergence gate or failed corroboration stores but does not grow the head; a missed sibling parent is fetched and adopted; a fetched parent that disagrees on parent fields is rejected.
• CGO_ENABLED=0 go build ./... && go vet ./... && go test ./...: green.

Review of the earlier work on this branch

Kept: the go-jsonrpc bump and test, the eth_call caches / singleflight / warmer, coalesced LRU touches in the persistent block cache (one Badger write per block per minute instead of per read), parallel prefetch of event AMT roots for eth block gas/bloom, opt-in pprof on the metrics listener (LANTERN_PPROF=1, off by default), the bench tool. Changed: the in-memory block layer only holds blocks up to 16 KiB, 16384 entries (worst case 256 MiB; it was 256 KiB x 32768, an 8 GiB worst case); the bench verifies at a pinned block, samples head age and can skip warmup bursts. Kept with a caveat: the long-lived Bitswap session (one session remembers which peers serve chain state instead of broadcasting for every block); reasoned, not separately measured. Config.SessionPerGet restores the old behaviour.

Build

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/lantern at 74a2f6c: sha256 58099ca6688fd954d65e13458ed0c60d829ea9c3317d9bc518a7ec39d5a98293.

Open

• The 24 h soak (256 timeouts/day) was not re-run here; the probe reproduces the exact error and its fix directly. A soak on this build should show the i/o timeouts gone.
• StateGetRandomnessDigestFromBeacon at head-10 returned "beacon entry ... not found in walked tipsets" for some epochs in both runs (5/19 before, 3/20 after). Separate issue, not touched here.