Repository navigation
fix(sync): retry transient raw-file failures instead of treating them as missing - #79
Merged
Merged
Conversation
… as missing fetchRawFile and fetchRawFileContent returned null for any non-OK response, so a 429, a 5xx or a dropped connection read exactly like a file that does not exist. With concurrent fetching that silently dropped pages and skill files from the mirror, or failed far from the cause when the dropped file was the docs root. Only a 404 now means "absent". 408, 429, 5xx and network errors are retried with exponential backoff (honouring Retry-After), and a failure that persists throws with the status and URL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
While looking into the failed deploy https://github.com/purview-dev/purview-dev.github.io/actions/runs/44590097197, I found that
fetchRawFile/fetchRawFileContentturned any non-OK response intonull. Their callers read that as "this file doesn't exist", so a throttled (429) or failed (5xx, network) request silently dropped a docs page or skill file. The build passed unless the dropped file was the docs root.That wasn't what broke this deploy. The cause was a malformed
.config/purview-docs.ymlin containers, already fixed in purview-dev/containers#31. This PR fixes the weakness the investigation exposed, which matters more now that #76/#77 fetch files 8 at a time.What changes
New
src/src/lib/docs/raw-fetch.tswithfetchRawText. Both raw fetchers now use it:null, as before ("this file doesn't exist").Retry-Afterheader overrides the wait, capped at 30 s.RawFetchErrorwith the status and URL.Callers that already handle errors keep working:
sync-assetsstill falls back to its fixture on a throw. A persistent failure in the docs or skills mirror now fails the sync loudly instead of publishing an incomplete site.Validation
just validatepasses in full after a live data sync, run by the pre-push hook. That covers format, lint, typecheck, 464 unit tests (including 10 new ones intests/unit/raw-fetch.test.ts), catalogue, build, discovery, SEO, link and generated-output checks.check-projectsprinted its 5 advisory catalogue observations, which don't fail the build; this PR doesn't touch the catalogue.