Skip to content

fix(sync): retry transient raw-file failures instead of treating them as missing - #79

Merged
kieronlanning merged 1 commit into
mainfrom
fix/raw-fetch-retry
Nov 19, 2026
Merged

kieronlanning merged 1 commit into
mainfrom
fix/raw-fetch-retry

Conversation

@kieronlanning

Copy link
Copy Markdown
Contributor

Why

While looking into the failed deploy https://github.com/purview-dev/purview-dev.github.io/actions/runs/44590097197, I found that fetchRawFile / fetchRawFileContent turned any non-OK response into null. Their callers read that as "this file doesn't exist", so a throttled (429) or failed (5xx, network) request silently dropped a docs page or skill file. The build passed unless the dropped file was the docs root.

That wasn't what broke this deploy. The cause was a malformed .config/purview-docs.yml in containers, already fixed in purview-dev/containers#31. This PR fixes the weakness the investigation exposed, which matters more now that #76/#77 fetch files 8 at a time.

What changes

New src/src/lib/docs/raw-fetch.ts with fetchRawText. Both raw fetchers now use it:

  • 404 → null, as before ("this file doesn't exist").
  • 408 / 429 / 5xx / network error → retried with exponential backoff (4 attempts in total: waits of 1 s, 2 s, 4 s). A Retry-After header overrides the wait, capped at 30 s.
  • Still failing, or any other non-OK status → throws a RawFetchError with the status and URL.

Callers that already handle errors keep working: sync-assets still falls back to its fixture on a throw. A persistent failure in the docs or skills mirror now fails the sync loudly instead of publishing an incomplete site.

Validation

just validate passes in full after a live data sync, run by the pre-push hook. That covers format, lint, typecheck, 464 unit tests (including 10 new ones in tests/unit/raw-fetch.test.ts), catalogue, build, discovery, SEO, link and generated-output checks. check-projects printed its 5 advisory catalogue observations, which don't fail the build; this PR doesn't touch the catalogue.

… as missing

fetchRawFile and fetchRawFileContent returned null for any non-OK
response, so a 429, a 5xx or a dropped connection read exactly like a
file that does not exist. With concurrent fetching that silently dropped
pages and skill files from the mirror, or failed far from the cause when
the dropped file was the docs root.

Only a 404 now means "absent". 408, 429, 5xx and network errors are
retried with exponential backoff (honouring Retry-After), and a failure
that persists throws with the status and URL.
@kieronlanning
kieronlanning merged commit d578170 into main Nov 19, 2026
1 check passed
@kieronlanning
kieronlanning deleted the fix/raw-fetch-retry branch November 19, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant