Repository navigation
[GitHub client] retry search pages independently and adapt page size on RESOURCE_LIMITS_EXCEEDED - #146
Open
StanislavKozachenko wants to merge 1 commit into
Conversation
Author
|
@ashkulz friendly ping — would appreciate a review when you have a moment. Thanks! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
SearchUserssurvive the occasional GraphQL errors that large locations hit, instead of failing the whole location (and, with the current workflow, the rest of the run).Key changes:
github/github.go—retryCountis reset after each successfully processed page, somaxRetryCountnow means 10 consecutive errors rather than 10 errors per location (a large location is ~200 pages at--consider 1000)github/github.go— when every error in a response isRESOURCE_LIMITS_EXCEEDED, the page size is halved (5 → 2 → 1) before retrying, so the retried query is actually cheaper instead of failing the same way again; other errors keep the current behaviourgithub/github.go— the 10s retry pause is now a package variable (retryDelay) so tests don't sleepgithub/github_test.go— new tests using a fakenet.Wrapper(no network): page size shrinks only onRESOURCE_LIMITS_EXCEEDED, and more than 10 non-consecutive errors no longer abort the search (fails onmasterwithToo many errors received. Quitting.)Verified with
go vet ./...,go build ./...,go test ./..., and real runs of the built binary (--preset belarus,--preset united_states,--output yaml), both completing with exit code 0 and unchanged output format. No GraphQL errors happened during those two runs, so the retry/page-size path itself is covered by the unit tests rather than the live runs.Merge order: this only touches
github/github.go(+ a new test file), so it's independent of #141 (workflow changes) and can be merged before or after it — together they address #143. It also complements #137/#138, which reduce the query cost upfront; this adapts when a page is still too expensive.Closes #145
Refs #143