fix(tasks): retry transient source-chain RPC errors in the CCTP relay scan - #3005
Merged
Merged
Conversation
sparrowDom
requested review from
clement-ux,
naddison36 and
shahthepro
as code owners
September 9, 2026 08:54
shahthepro
approved these changes
Sep 9, 2026
… scan The relay actions (relayCCTPMessage, crossChainRelayHyperEVM) scan the source chain with eth_blockNumber + eth_getLogs over a bare ethers JsonRpcProvider that never retries. The load-balanced dRPC endpoints behind BASE_PROVIDER_URL / HYPEREVM_PROVIDER_URL intermittently answer with transient errors, and every one failed the whole run: 500 "Temporary internal error. Please retry" (Base), 408 timeouts, and 400 "Unknown block" on HyperEVM when eth_getLogs lands on a node that has not yet seen the tip eth_blockNumber just returned (the failing toBlock was mined the same second the run started). Wrap the three reads in a 5-attempt exponential backoff; they are idempotent.
sparrowDom
force-pushed
the
sparrow/cctp-relay-retry-source-reads
branch
from
September 9, 2026 12:39
35fb556 to
6f096d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
relayCCTPMessage/crossChainRelayHyperEVMwere dying on the source-chain tip scan becauseeth_blockNumber+eth_getLogsgo through a bare ethersJsonRpcProviderwith no retries. BehindBASE_PROVIDER_URL/HYPEREVM_PROVIDER_URL, load-balanced dRPC nodes intermittently return:400 Unknown block—getLogslands on a peer that has not yet seen the tipgetBlockNumberjust returned (failingtoBlockmined the same second the run started)500 Temporary internal error. Please retryand408timeoutsThose failures aborted the whole Talos run several times a day.
Change
withRetry(5 attempts, exponential backoff 1/2/4/8s) around the three source reads infetchTxHashesFromCctpTransactionswithRetry+fetchTxHashesFromCctpTransactionsfor unit testsNo deploy scripts, descriptors, or action wiring changes. Existing
cross_chain_*schedules pick this up on the next runner image.