Skip to content

[GitHub client] retry search pages independently and adapt page size on RESOURCE_LIMITS_EXCEEDED - #146

Open
StanislavKozachenko wants to merge 1 commit into
ashkulz:masterfrom
StanislavKozachenko:fix-search-resource-limits
Open

StanislavKozachenko wants to merge 1 commit into
ashkulz:masterfrom
StanislavKozachenko:fix-search-resource-limits

Conversation

@StanislavKozachenko

Copy link
Copy Markdown

Makes SearchUsers survive the occasional GraphQL errors that large locations hit, instead of failing the whole location (and, with the current workflow, the rest of the run).

Key changes:

  • github/github.go — retryCount is reset after each successfully processed page, so maxRetryCount now means 10 consecutive errors rather than 10 errors per location (a large location is ~200 pages at --consider 1000)
  • github/github.go — when every error in a response is RESOURCE_LIMITS_EXCEEDED, the page size is halved (5 → 2 → 1) before retrying, so the retried query is actually cheaper instead of failing the same way again; other errors keep the current behaviour
  • github/github.go — the 10s retry pause is now a package variable (retryDelay) so tests don't sleep
  • github/github_test.go — new tests using a fake net.Wrapper (no network): page size shrinks only on RESOURCE_LIMITS_EXCEEDED, and more than 10 non-consecutive errors no longer abort the search (fails on master with Too many errors received. Quitting.)

Verified with go vet ./..., go build ./..., go test ./..., and real runs of the built binary (--preset belarus, --preset united_states, --output yaml), both completing with exit code 0 and unchanged output format. No GraphQL errors happened during those two runs, so the retry/page-size path itself is covered by the unit tests rather than the live runs.

Merge order: this only touches github/github.go (+ a new test file), so it's independent of #141 (workflow changes) and can be merged before or after it — together they address #143. It also complements #137/#138, which reduce the query cost upfront; this adapts when a page is still too expensive.

Closes #145
Refs #143

@StanislavKozachenko

StanislavKozachenko commented Oct 14, 2026 •

Copy link
Copy Markdown
Author

@ashkulz friendly ping — would appreciate a review when you have a moment. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Large locations fail with RESOURCE_LIMITS_EXCEEDED and never refresh (e.g. united_states)

1 participant