Skip to content

Dist test stability & discovery improvement fixes - #176

Open
dryajov wants to merge 14 commits into
mainfrom
fix/battery-stability
Open

dryajov wants to merge 14 commits into
mainfrom
fix/battery-stability

Conversation

@dryajov

@dryajov dryajov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@dryajov dryajov changed the title Fix/battery stability Dist test stability fixes Aug 6, 2026
@dryajov dryajov changed the title Dist test stability fixes Dist test stability & discovery improvement fixes Aug 6, 2026
@greptile-apps

greptile-apps Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

Greptile Summary

The PR improves distributed-test stability and peer discovery while making marketplace recovery and network streaming more resilient.

  • Adds marketplace and circuit-download startup retries.
  • Introduces discovery backoff, advertisement retries, peer-query rotation, and randomized equal-score ranking.
  • Reconstructs marketplace purchases and sales from contract event history after restart.
  • Retries incomplete block batches and propagates storage expiry through erasure repair.
  • Reduces block and batching defaults to lower memory and transport pressure.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename Overview
archivist/blockexchange/engine/discovery.nim Adds bounded per-CID discovery backoff and removes stale backoff state when blocks leave the want list.
archivist/blockexchange/engine/engine.nim Rotates refresh targets through a query quarantine while allowing newly queued wants to bypass it.
archivist/blockexchange/peers/peercontext.nim Tracks query timing and gives unscored peers a cold-start aggregate score.
archivist/marketplace/purchasing.nim Re-adopts signer-owned purchases from historical storage-request events during startup.
archivist/marketplace/sales.nim Re-adopts signer-owned sales slots from historical slot-filled events during startup.
archivist/streams/storestream.nim Retries missing indices from partial block batches before ordered stream copying.
archivist/archivist.nim Retries marketplace connection attempts during persistent node startup.
archivist/rest/api.nim Avoids completing responses whose streamed body is shorter than the declared content length.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Node startup] --> B{Marketplace reachable?}
  B -->|No| C[Retry with configured delay]
  C --> B
  B -->|Yes| D[Load historical marketplace events]
  D --> E[Re-adopt owned purchases and sales]
  E --> F[Start block exchange]
  F --> G{Enough peers or providers?}
  G -->|No| H[Back off discovery and rotate queried peers]
  H --> G
  G -->|Yes| I[Fetch and stream blocks]
  I --> J{Batch complete?}
  J -->|No| K[Retry missing indices]
  K --> J
  J -->|Yes| L[Complete response]
Loading

Reviews (2): Last reviewed commit: "Pin glibc mmap threshold below block siz..." | Re-trigger Greptile

dryajov added 13 commits August 19, 2026 08:20
- Retry marketplace connect and cirdl circuit download (10x10s)
- Abort REST retrieves with short bodies loudly instead of silently
  truncating
The knowledge-refresh fallback always asked the same aggregate top-K;
stable scores and deterministic tie order pinned the same peers forever,
so a fresh DHT-discovered provider was never refreshed, never reported
presence, and was never fetched from. Add a query quarantine that skips
peers recently sent a want list, randomize equal-score ordering in the
rankers, and exempt peers with unsent wants so newly queued blocks still
reach them. Reuse the orphaned topPeersForCid helper for the known-haves
branch and wake at the earliest quarantine expiry instead of busy-looping.
- Remove the 10-minute advertise retry window: the sparse retry is now
  steady-state, gated only by table health, so small networks keep
  replicating provider records to newly-arriving peers
- Rework the discovery find-backoff to a (due, mult) tuple table with
  multiplier-derived intervals; prune entries once a cid leaves the want
  list, including tree cids of leaf wants
- Make nodesDiscovered a method so tests can override it; export
  DefaultMinAdvertisePeers
- Add steady-state advertiser retry test and backoff-recovery test
erasure.repair stored the reconstructed dataset under the original-tree
overlay with no expiry, so putOverlay assigned the 30-day default TTL
and the maintenance never cleaned it after the request ended (hosts
retained the whole dataset - SlotCleanupTest 97 vs 33 blocks). Thread
the request expiry from storeSlot through repair into the overlay
writes, matching the flow's implicit cleanup design.
- Clamp pastBlockTag to genesis when the requested window exceeds the
  chain head: a fresh chain is shorter than the 30-day request window,
  and the UInt256 subtraction underflowed, failing every marketplace
  start with 'hex number > 64 bits'
- Reload purchases and sales from onchain StorageRequested/SlotFilled
  event history instead of myRequests/mySlots
- Guard errored-state restarts with the onchain request/slot state and
  signer check instead of the myRequests/mySlots membership list
- Reserve slots only when the host token balance covers the collateral
- Pin vendored contractabi with the readTail in-place decode fallback
The event-based sales reload created sale agents for every SlotFilled
event in the window, including other hosts' fills, so the sales slots
list could return foreign slots that 404 on lookup and break clients
asking for them. Look up each slot's onchain host and skip slots that
are not ours.
The onchain reservation allows several hosts per slot with first-fill-wins,
so a host with three queue workers could reserve most slots of a request
and then race other hosts on the same slot. Skip queue items for a request
that already has an agent.
128 KiB blocks (from 512 KiB) plus smaller store/serve batches cut the
in-flight peaks that glibc's arena retention pins as RSS: upload 2 GiB ->
512 MiB, serve 72 MiB/peer -> 6 MiB/peer. Old datasets still decode via
manifest blockSize.
A single transient fetch miss (peer dropped a want-batch) or a store
CAS retry-out aborted the whole download stream. Re-fetch missing
indices up to 5 times with backoff before failing.
Concurrent putBlocks on the same treeCid (swarm downloads from many
peers) CAS-conflict on the shared overlay/refCount records; the 10-retry
bound could exhaust under the burst. 30 retries let the writer storm
drain; the middleware refetches and re-applies on fresh values.
glibc's mmap-threshold ratchet demotes the 128 KiB block buffers to the
heap, where interleaved small allocations pin the arena tops and freed
chunks never return to the OS - RSS stays at the historical peak forever
(measured: 627 MB RSS -> 61 MB after malloc_trim(0)). Pinning the
threshold to 64 KiB keeps block buffers on the mmap path so frees return
memory at once, and disables the ratchet. Makes the 20-node x 300 MB
StreamOneToMany swarm leg pass with flat memory (nodes ~700 MB vs 2.5 GB
hoarded); threshold beats per-poll malloc_trim in the A/B (no measurable
speed cost, trim was 7-10% slower at swarm scale).
@dryajov
dryajov force-pushed the fix/battery-stability branch from 5d15143 to ea10326 Compare August 19, 2026 15:07

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your organization has used all 50 credits included in the free plan this billing period. To keep receiving reviews, upgrade your plan.

The branch carried an old contractabi gitlink (1c41c22e) that was never
pushed to the durability-labs fork - CI submodule fetch fails with
'not our ref'. Main's pin (e9092906, v0.7.4) is reachable and the branch
was rebased on it; align the pin and the CI submodule stage passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant