Expand scraper coverage and diagnostics - #2334
Conversation
Pin the release workflow to a known action revision and use the dedicated release token. Reclaim runner disk before test-clickhouse builds its image, which had started failing on disk exhaustion. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Adds organization-level scraping to the GitHub scraper: org settings and metadata, installed apps, code security configuration, and the org RBAC graph of members, teams and roles, with the CRD and JSON schema regenerated to match. Auxiliary API calls degrade instead of aborting the scrape, and scraping stops before the rate limit is exhausted, reporting the reset time so a throttled run is distinguishable from a broken one. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Walks the organization, folder and project hierarchy and attaches resource-scoped IAM bindings, linking hierarchy roots back to the projects already in inventory so the two graphs join up. Hierarchy discovery failures fall back to the flat policy results rather than failing the whole scrape. Promotes cloud.google.com/go/iam from an indirect to a direct dependency. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Pin @babel/core through a pnpm override so the lockfile resolves, declare the iconify-icon custom element on preact's JSX namespace, and give JsonView's entries an explicit tuple type. tsc now type-checks the scrape UI cleanly. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Move the specs that launch a real browser into an e2e-tagged file so the pure parseOutput specs compile and run without bun, chromium or the e2e build tag. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Run gavel over the same scope as test.yml's `test` job — same runner, timeout, step order, pinned SHAs and cache key — so the two checks agree on what they cover. tests/e2e is ignored to match `gotest`'s --skip-package (and is the only consumer of the loki/opensearch services, so the job needs none), and bench is left to benchmark.yml since ginkgo -r never runs Go benchmarks. An explicit envtest step remains because gavel invokes packages directly rather than through `make test`, so it does not inherit that target's prerequisites. The check fails the build on test failures rather than reporting advisory-only. Claude-Session-Id: 30a271bb-42a0-47aa-843e-c4cf03ce62df
Introduce a doctor API and CLI for validating scraper access against local configs or persisted scraper IDs. Add GitHub repository and organization probes with pass/fail/skip results and permission evidence, while avoiding inferred repository grants for selected app installations and making rulesets independently opt-in.
WalkthroughChangesDoctor checks and GitHub organization scraping
GCP IAM hierarchy and resource-scoped access
CI and release workflow updates
Frontend and Playwright test maintenance
Sequence Diagram(s)sequenceDiagram
participant Operator
participant DoctorCLI
participant DoctorRunner
participant GithubScraper
participant GitHubAPI
Operator->>DoctorCLI: Provide fixture path or scraper UUID
DoctorCLI->>DoctorRunner: Load configuration and run doctors
DoctorRunner->>GithubScraper: Invoke Doctor
GithubScraper->>GitHubAPI: Probe repositories and organizations
GitHubAPI-->>GithubScraper: Responses, scopes, and errors
GithubScraper-->>DoctorRunner: DoctorResults
DoctorRunner-->>DoctorCLI: Render table or JSON failure output
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
scrapers/gcp/iam_hierarchy.go (2)
116-168: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd negative-path tests for hierarchy validation.
buildResourceManagerHierarchyis a pure function, so its "incomplete hierarchy: expected X, got Y" (mismatched parent) and "missing X" (unterminated chain) error branches are inexpensive to cover with unit tests, but only the successful chain is currently exercised iniam_test.go.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scrapers/gcp/iam_hierarchy.go` around lines 116 - 168, Add unit tests for buildResourceManagerHierarchy covering both hierarchy-validation failures: nodes whose metadata name differs from the expected parent should return the “expected X, got Y” error, and a chain ending with a non-empty expectedName should return the “missing X” error. Keep the existing successful-chain test unchanged and assert the returned error and nil results for each negative case.
68-97: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDon't discard hierarchy metadata just because
GetIamPolicyfails.
fetchResourceManagerNodetreats aGetIamPolicyfailure as fatal for the whole node, even though theGetcall already succeeded. Sinceresourcemanager.{folders,organizations}.getIamPolicyis commonly a narrower permission than.get, this means a single missing IAM-policy permission on one ancestor discards the entire hierarchy chain (via the warn-and-fallback infetchIAMPolicies), losing folder/org structure that was otherwise fully available. This runs counter to the PR's goal of reporting permission conditions more precisely.Consider tolerating a
GetIamPolicyfailure by keeping the resource with a nilPolicy(and logging/warning), so the ancestor chain and config items are still emitted even when only bindings are inaccessible.♻️ Proposed fix
switch { case strings.HasPrefix(name, "folders/"): folder, err := service.Folders.Get(name).Context(ctx).Do() if err != nil { return resourceManagerNode{}, fmt.Errorf("get GCP folder %s: %w", name, err) } policy, err := service.Folders.GetIamPolicy(name, request).Context(ctx).Do() if err != nil { - return resourceManagerNode{}, fmt.Errorf("get IAM policy for GCP folder %s: %w", name, err) + ctx.Warnf("gcp iam policies: IAM policy for GCP folder %s unavailable: %v", name, err) + return resourceManagerNode{Resource: folder}, nil } return resourceManagerNode{Resource: folder, Policy: policy}, nil case strings.HasPrefix(name, "organizations/"): organization, err := service.Organizations.Get(name).Context(ctx).Do() if err != nil { return resourceManagerNode{}, fmt.Errorf("get GCP organization %s: %w", name, err) } policy, err := service.Organizations.GetIamPolicy(name, request).Context(ctx).Do() if err != nil { - return resourceManagerNode{}, fmt.Errorf("get IAM policy for GCP organization %s: %w", name, err) + ctx.Warnf("gcp iam policies: IAM policy for GCP organization %s unavailable: %v", name, err) + return resourceManagerNode{Resource: organization}, nil } return resourceManagerNode{Resource: organization, Policy: policy}, nil🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scrapers/gcp/iam_hierarchy.go` around lines 68 - 97, Update fetchResourceManagerNode so GetIamPolicy failures for folders and organizations do not discard the successfully fetched resource: retain the resource with a nil Policy, and emit an appropriate warning or log for the policy error. Continue returning errors from the resource Get calls, and preserve successful policy retrieval behavior so the hierarchy remains available when only IAM bindings are inaccessible.cmd/doctor.go (1)
134-148: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winFile targets shouldn't hard-fail on an unreachable database.
When
DB_URLis set but the database is down,doctor fixtures/github-doctor.yamlaborts withinitialize database: ...even though a file target needs no DB. Consider only connecting whenrequireDatabaseis true (or falling back todutycontext.New()on connect failure for file targets).♻️ Proposed change
func newDoctorContext(requireDatabase bool) (dutycontext.Context, error) { config := dutyapi.DefaultConfig.ReadEnv() - if config.ConnectionString == "" { - if requireDatabase { - return dutycontext.Context{}, fmt.Errorf("scraper-id doctor target requires a configured database") - } + if config.ConnectionString == "" || !requireDatabase { + if requireDatabase { + return dutycontext.Context{}, fmt.Errorf("scraper-id doctor target requires a configured database") + } return dutycontext.New(), nil }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cmd/doctor.go` around lines 134 - 148, Update newDoctorContext so database initialization via duty.Start occurs only when requireDatabase is true; when a database is not required, return dutycontext.New() even if ConnectionString is configured or the database is unreachable. Preserve the existing error for required database targets.scrapers/github/doctor_result.go (1)
31-60: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winHandle GitHub rate-limit errors in
githubDoctorResult.
githubHTTPResponseonly restores theResponsefor*gogithub.ErrorResponse, andgithubGrantEvidencereturns"request denied"for any error. When go-github returns*gogithub.RateLimitErroror*gogithub.AbuseRateLimitError, the result stays generic instead of surfacing the rate-limit condition consistently withresolveDoctorRepositories, who explicitly emitsMessage: "GitHub API rate limit reached". Add centralized rate-limit detection before the current permissions/evidence handling.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scrapers/github/doctor_result.go` around lines 31 - 60, Update githubDoctorResult to centrally detect *gogithub.RateLimitError and *gogithub.AbuseRateLimitError before deriving permissions and grant evidence, and set the result message to "GitHub API rate limit reached" consistently with resolveDoctorRepositories. Preserve the existing status, knownDisabled handling, and normal permission/evidence behavior for non-rate-limit errors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/gavel.yml:
- Around line 33-34: Update the “Checkout code” step using actions/checkout to
set persist-credentials to false, disabling persistent Git credentials while
leaving the existing checkout action and version unchanged.
In `@scrapers/github/apps.go`:
- Around line 174-191: Update the installation handling around
effectiveInstallationRole so an installation with repositories but no effective
repository role is skipped rather than causing buildAppInstallations to return
an error. Preserve role emission for installations with a valid role and
continue processing all remaining installations.
In `@scrapers/github/code_security.go`:
- Around line 20-53: Update scrapeCodeSecurityConfigurations and
codeSecurityConfigurationRepositories to iterate through every paginated
response from GetCodeSecurityConfigurations and
GetRepositoriesForCodeSecurityConfiguration, passing the returned cursor or page
options on each request. Preserve existing error handling and aggregate all
configurations and repositories before building results so larger organizations
are fully represented.
---
Nitpick comments:
In `@cmd/doctor.go`:
- Around line 134-148: Update newDoctorContext so database initialization via
duty.Start occurs only when requireDatabase is true; when a database is not
required, return dutycontext.New() even if ConnectionString is configured or the
database is unreachable. Preserve the existing error for required database
targets.
In `@scrapers/gcp/iam_hierarchy.go`:
- Around line 116-168: Add unit tests for buildResourceManagerHierarchy covering
both hierarchy-validation failures: nodes whose metadata name differs from the
expected parent should return the “expected X, got Y” error, and a chain ending
with a non-empty expectedName should return the “missing X” error. Keep the
existing successful-chain test unchanged and assert the returned error and nil
results for each negative case.
- Around line 68-97: Update fetchResourceManagerNode so GetIamPolicy failures
for folders and organizations do not discard the successfully fetched resource:
retain the resource with a nil Policy, and emit an appropriate warning or log
for the policy error. Continue returning errors from the resource Get calls, and
preserve successful policy retrieval behavior so the hierarchy remains available
when only IAM bindings are inaccessible.
In `@scrapers/github/doctor_result.go`:
- Around line 31-60: Update githubDoctorResult to centrally detect
*gogithub.RateLimitError and *gogithub.AbuseRateLimitError before deriving
permissions and grant evidence, and set the result message to "GitHub API rate
limit reached" consistently with resolveDoctorRepositories. Preserve the
existing status, knownDisabled handling, and normal permission/evidence behavior
for non-rate-limit errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 50678211-ccb3-4e33-b5e3-14088985b8db
📒 Files selected for processing (46)
.github/workflows/gavel.yml.github/workflows/release.yml.github/workflows/test.ymlREADME.mdapi/global.goapi/v1/common.goapi/v1/doctor.goapi/v1/doctor_test.goapi/v1/github.goapi/v1/zz_generated.deepcopy.gochart/crds/configs.flanksource.com_scrapeconfigs.yamlcmd/doctor.gocmd/doctor_suite_test.gocmd/doctor_test.gocmd/scrapeui/frontend/package.jsoncmd/scrapeui/frontend/src/components/JsonView.tsxcmd/scrapeui/frontend/src/iconify-icon.d.tsconfig/schemas/config_github.schema.jsonconfig/schemas/scrape_config.schema.jsonfixtures/github-doctor.yamlfixtures/github.yamlgo.modscrapers/doctor.goscrapers/doctor_test.goscrapers/gcp/iam.goscrapers/gcp/iam_hierarchy.goscrapers/gcp/iam_test.goscrapers/github/apps.goscrapers/github/apps_test.goscrapers/github/code_security.goscrapers/github/code_security_test.goscrapers/github/doctor.goscrapers/github/doctor_organization.goscrapers/github/doctor_organization_apps.goscrapers/github/doctor_organization_members.goscrapers/github/doctor_organization_settings.goscrapers/github/doctor_result.goscrapers/github/doctor_security.goscrapers/github/doctor_test.goscrapers/github/org_rbac.goscrapers/github/org_rbac_test.goscrapers/github/organization.goscrapers/github/organization_test.goscrapers/github/scraper.goscrapers/playwright/browser_e2e_test.goscrapers/playwright/parse_test.go
💤 Files with no reviewable changes (1)
- scrapers/playwright/parse_test.go
| - name: Checkout code | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files =="
git ls-files .github/workflows || true
echo "== target workflow excerpt =="
if [ -f .github/workflows/gavel.yml ]; then
nl -ba .github/workflows/gavel.yml | sed -n '1,140p'
else
echo "missing .github/workflows/gavel.yml"
fi
echo "== search token/action usage in workflows =="
rg -n "GITHUB_TOKEN|permissions:|actions/checkout|gavel|dependency|envtest|persist-credentials|environment:" .github/workflows || trueRepository: flanksource/config-db
Length of output: 517
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target workflow excerpt =="
cat -n .github/workflows/gavel.yml | sed -n '1,160p'
echo "== token/action usage in gavel workflow =="
grep -nE "GITHUB_TOKEN|permissions:|actions/checkout|gavel|dependency|envtest|persist-credentials|environment:" .github/workflows/gavel.yml || true
echo "== token/action usage in all workflows =="
grep -RInE -n "\${{ secrets\.|GITHUB_TOKEN|permissions:|actions/checkout|envtest|dependency setup|persist-credentials" .github/workflows || trueRepository: flanksource/config-db
Length of output: 8120
🌐 Web query:
actions/checkout README persist-credentials default true stores credentials git config
💡 Result:
The persist-credentials input in the actions/checkout GitHub Action controls whether the repository's authentication credentials (either the GITHUB_TOKEN or an SSH key) are configured in the local Git environment, allowing subsequent workflow steps to perform authenticated Git operations [1][2][3]. Although the default value for persist-credentials is true [4][1][2], its implementation has evolved to improve security [3][5][6]: 1. Historical Behavior: Originally, this setting wrote the authentication token or SSH key directly into the local.git/config file [2][5]. 2. Current Behavior: As of version v6.0.0, the action was updated to store credentials in a separate file located under $RUNNER_TEMP instead of writing them directly to.git/config [3][5][6]. This change reduces the risk of accidental exposure and improves security while maintaining the same functionality for subsequent Git commands [3][6]. The token or key remains active for the duration of the job and is removed during the post-job cleanup process [1][2][3]. Users who do not require authenticated Git access in subsequent steps can opt out by setting persist-credentials: false in their workflow configuration [1][3]. Note that while there have been community discussions and proposals to change the default value to false to enhance security, the default remains true as of the current versions [7][4][2][8].
Citations:
- 1: https://github.com/marketplace/actions/checkout?version=v3.6.0
- 2: https://github.com/actions/checkout/tree/v6.0.2?tab=readme-ov-file
- 3: https://github.com/actions/checkout/?tab=readme-ov-file
- 4: https://github.com/actions/checkout/blob/cbb722410c2e876e24abbe8de2cc27693e501dcb/action.yml
- 5: https://github.com/actions/checkout/tree/v6.0.0
- 6: actions/checkout@v5.0.1...v6.0.0
- 7: Change the default value of persist-credentials to false actions/checkout#1687
- 8: Remove
persist-credentialsor change the default tofalseactions/checkout#485
Disable checkout credential persistence.
actions/checkout enables authenticated Git access for later steps by default; this workflow does not need persistent checkout credentials because dependency/setup/token access is passed explicitly where required. Add persist-credentials: false under the checkout step.
Proposed fix
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Checkout code | |
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Checkout code | |
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 33-34: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/gavel.yml around lines 33 - 34, Update the “Checkout code”
step using actions/checkout to set persist-credentials to false, disabling
persistent Git credentials while leaving the existing checkout action and
version unchanged.
Source: Linters/SAST tools
| if len(installed.Repositories) == 0 { | ||
| continue | ||
| } | ||
|
|
||
| role, err := effectiveInstallationRole(permissions) | ||
| if err != nil { | ||
| return appInstallations{}, fmt.Errorf("github app %q: %w", installation.GetAppSlug(), err) | ||
| } | ||
| roleAlias := githubRepositoryRoleAlias(input.Owner, role) | ||
| if _, ok := seenRoles[roleAlias]; !ok { | ||
| seenRoles[roleAlias] = struct{}{} | ||
| result.Roles = append(result.Roles, models.ExternalRole{ | ||
| Tenant: input.Owner, | ||
| Aliases: pq.StringArray{roleAlias}, | ||
| RoleType: "GitHub::Repository", | ||
| Name: role, | ||
| }) | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
One org-scoped app can abort the whole installation scrape.
An installation with repository_selection: all but only organization-scoped permissions (e.g. organization_administration) yields a non-empty installed.Repositories, so effectiveInstallationRole returns missing effective repository role and buildAppInstallations aborts — discarding results for every installation in the org. The same app installed with selected is handled gracefully by the len(...) == 0 guard, so the failure mode is inconsistent.
Consider skipping access/role emission for that installation (as the selected path does) instead of failing the batch.
🛠️ Proposed fix
role, err := effectiveInstallationRole(permissions)
if err != nil {
- return appInstallations{}, fmt.Errorf("github app %q: %w", installation.GetAppSlug(), err)
+ // Organization-scoped installations carry no repository role; the
+ // config item is still emitted, just without access edges.
+ continue
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if len(installed.Repositories) == 0 { | |
| continue | |
| } | |
| role, err := effectiveInstallationRole(permissions) | |
| if err != nil { | |
| return appInstallations{}, fmt.Errorf("github app %q: %w", installation.GetAppSlug(), err) | |
| } | |
| roleAlias := githubRepositoryRoleAlias(input.Owner, role) | |
| if _, ok := seenRoles[roleAlias]; !ok { | |
| seenRoles[roleAlias] = struct{}{} | |
| result.Roles = append(result.Roles, models.ExternalRole{ | |
| Tenant: input.Owner, | |
| Aliases: pq.StringArray{roleAlias}, | |
| RoleType: "GitHub::Repository", | |
| Name: role, | |
| }) | |
| } | |
| if len(installed.Repositories) == 0 { | |
| continue | |
| } | |
| role, err := effectiveInstallationRole(permissions) | |
| if err != nil { | |
| // Organization-scoped installations carry no repository role; the | |
| // config item is still emitted, just without access edges. | |
| continue | |
| } | |
| roleAlias := githubRepositoryRoleAlias(input.Owner, role) | |
| if _, ok := seenRoles[roleAlias]; !ok { | |
| seenRoles[roleAlias] = struct{}{} | |
| result.Roles = append(result.Roles, models.ExternalRole{ | |
| Tenant: input.Owner, | |
| Aliases: pq.StringArray{roleAlias}, | |
| RoleType: "GitHub::Repository", | |
| Name: role, | |
| }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scrapers/github/apps.go` around lines 174 - 191, Update the installation
handling around effectiveInstallationRole so an installation with repositories
but no effective repository role is skipped rather than causing
buildAppInstallations to return an error. Preserve role emission for
installations with a valid role and continue processing all remaining
installations.
| func scrapeCodeSecurityConfigurations(ctx api.ScrapeContext, scrape *organizationScrape) v1.ScrapeResults { | ||
| var results v1.ScrapeResults | ||
| org := scrape.name() | ||
|
|
||
| configurations, _, err := scrape.client.Client.Organizations.GetCodeSecurityConfigurations(ctx, org) | ||
| if err != nil { | ||
| if isOrganizationFeatureUnavailable(err) { | ||
| ctx.Logger.V(2).Infof("skipping code security configurations for %s: %v", org, err) | ||
| return nil | ||
| } | ||
|
|
||
| results.Errorf(err, "failed to list code security configurations for GitHub organization %s", org) | ||
| return results | ||
| } | ||
|
|
||
| for _, configuration := range configurations { | ||
| if configuration.GetID() == 0 { | ||
| results.Errorf(fmt.Errorf("missing id"), "invalid code security configuration %q for GitHub organization %s", | ||
| configuration.GetName(), org) | ||
| continue | ||
| } | ||
|
|
||
| repositories, err := codeSecurityConfigurationRepositories(ctx, scrape, configuration.GetID()) | ||
| if err != nil { | ||
| results.Errorf(err, "failed to list repositories for code security configuration %q of GitHub organization %s", | ||
| configuration.GetName(), org) | ||
| continue | ||
| } | ||
|
|
||
| results = append(results, buildCodeSecurityConfigurationResult(scrape, configuration, repositories)) | ||
| } | ||
|
|
||
| return results | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does go-github v73's Organizations.GetCodeSecurityConfigurations and GetRepositoriesForCodeSecurityConfiguration support pagination via ListOptions, or do they return all results in one call?
💡 Result:
The go-github library methods Organizations.GetCodeSecurityConfigurations and Organizations.GetRepositoriesForCodeSecurityConfiguration do not automatically return all results in a single call; they adhere to the GitHub REST API's pagination standards [1][2][3]. Regarding pagination support: 1. Organizations.GetCodeSecurityConfigurations: According to the GitHub REST API documentation for this endpoint, it supports per_page, before, and after query parameters, which are characteristic of cursor-based pagination [4][5]. While go-github provides a ListOptions struct for standard offset-based pagination (using Page and PerPage), endpoints that utilize cursor-based pagination (using before and after parameters) often require passing a struct that includes these cursor fields rather than the generic ListOptions [2][6]. 2. Organizations.GetRepositoriesForCodeSecurityConfiguration: This endpoint also follows standard GitHub REST API pagination conventions [4][5]. Consequently, it does not aggregate all results into one response [2][3]. To retrieve all results, you must implement a loop that checks the Response object returned by these methods. You can inspect the NextPage field (for offset pagination) or use the cursors provided in the response headers (for cursor pagination) to make subsequent requests until all data has been fetched [1][2][6]. Alternatively, you can use middleware such as gofri/go-github-pagination to automatically handle the aggregation of paginated results [1][2][3].
Citations:
- 1: https://github.com/google/go-github/tree/v73.0.0
- 2: https://github.com/google/go-github
- 3: https://github.com/google/go-github?tab=readme-ov-file
- 4: https://docs.github.com/en/rest/code-security/configurations
- 5: https://docs.github.com/en/rest/code-security/configurations?apiVersion=2022-11-28
- 6: https://github.com/google/go-github/blob/master/github/github.go
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files matching code_security.go:"
fd -a 'code_security\.go$' . || true
echo
echo "Inspect relevant file outline and sections:"
file=$(fd 'code_security\.go$' . | head -n1 || true)
if [ -n "$file" ]; then
echo "FILE=$file"
wc -l "$file"
ast-grep outline "$file" || true
sed -n '1,120p' "$file" | cat -n
fi
echo
echo "Search for all code security API calls:"
rg -n "GetCodeSecurityConfigurations|GetRepositoriesForCodeSecurityConfiguration|ListOptions|NextPage" . --glob '!vendor/**' --glob '!node_modules/**' || trueRepository: flanksource/config-db
Length of output: 16754
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Inspect existing pagination handling for the same code-security endpoints:"
sed -n '120,170p' scrapers/github/doctor_organization_settings.go | cat -n
echo
echo "Determine go-github v73 dependency path from module files:"
fd '^go\.(mod|sum)$' . | while read -r f; do
echo "--- $f"
rg -n 'github.com/google/go-github/v73|github.com/google/go-github' "$f" || true
done
echo
echo "Search generated/external go-github signatures in installed module cache, if present:"
module_version=$(rg -n 'github.com/google/go-github/v73|github.com/google/go-github' go.mod go.sum 2>/dev/null | sed -n '1,5p' || true)
echo "$module_version"
goenv=$(go env GOPATH 2>/dev/null || true)
echo "GOPATH=$goenv"
if [ -n "$goenv" -a "$goenv" != "" ]; then
fd 'organizations*|code_security*.*' "$goenv/pkg/mod/github.com/google!go-github" 2>/dev/null | sort | sed -n '1,80p' || true
fi
python3 - <<'PYTHON'
# Read-only behavioral probe: inspect installed go-github source signatures if cache exists.
# This does not import/configure/run repository code.
PYTHONRepository: flanksource/config-db
Length of output: 2569
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
gopath = Path.home() / "go"
candidates = []
for root, dirs, files in Path(gopath / "pkg" / "mod" / "github.com").glob("google/go-github**"):
if not root.is_dir():
continue
candidates.append((root, "organizations.go", "code_security_configurations.go"))
print("GOPATH", gopath)
print("candidates", len(candidates))
for root, org, cfg in candidates:
print("--- root", root)
for name in (org, cfg):
p = root / "github" / name
if p.exists():
print(f"FOUND github/{name}")
text = p.read_text(errors="replace").splitlines()
for i, line in enumerate(text, 1):
if "GetCodeSecurityConfigurations" in line or "GetRepositoriesForCodeSecurityConfiguration" in line or "ListCursorOptions" in line or "ListOptions" in line:
# nearby 20 lines
start=max(1, i-15)
if name == "code_security_configurations.go":
end=min(len(text), i+40)
else:
end=min(len(text), i+40)
for j in range(start, end+1):
print(f"{name}:{j}:{text[j-1]}")
PYRepository: flanksource/config-db
Length of output: 933
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
gopath = Path.home() / "go"
pkg = gopath / "pkg" / "mod" / "github.com"
found = []
for root in pkg.iterdir():
if not root.is_dir() or root.name != "google":
continue
try:
for root2 in root.iterdir():
if not root2.is_dir() or root2.name != "go-github":
continue
for root3 in root2.iterdir():
if root3.is_dir() and any(part in ("v73.0.0", "v73") for part in root3.parts):
found.append(root3)
except PermissionError:
print("permission error under", root)
print("found:", found)
for root in found:
for name in ["organizations.go", "code_security_configurations.go"]:
p = root / "github" / name
if not p.exists():
print(f"MISSING {root}/{name}")
print("github children:", [x.name for x in (root / "github").iterdir() if x.is_file()][:80], "...")
continue
print(f"\n--- {root}/{name}")
lines = p.read_text(errors="replace").splitlines()
for i, line in enumerate(lines, 1):
if "GetCodeSecurityConfigurations" in line or "GetRepositoriesForCodeSecurityConfiguration" in line:
start = max(1, i - 20)
end = min(len(lines), i + 45)
for j in range(start, end + 1):
print(f"{j:4}: {lines[j-1]}")
PYRepository: flanksource/config-db
Length of output: 169
🌐 Web query:
github.com/google/go-github v73.0.0 GetCodeSecurityConfigurations GetRepositoriesForCodeSecurityConfiguration signatures ListCursorOptions
💡 Result:
In the google/go-github library (version 73.0.0), the Code Security Configurations API and cursor-based pagination are implemented as follows: Code Security Configurations Methods Support for Code Security Configurations was introduced to interact with the GitHub REST API endpoints [1]. Relevant methods available in the library include: GetCodeSecurityConfigurations: Used to list code security configurations, typically available under the relevant service (e.g., organizations or enterprises) [1][2]. GetRepositoriesForCodeSecurityConfiguration: Used to retrieve repositories associated with a specific code security configuration [3]. Note that in later versions of the library, the return type for this method was updated to RepositoryAttachment [4][5]. ListCursorOptions The ListCursorOptions struct is used for cursor-based pagination, which is required by many modern GitHub REST API endpoints that do not use traditional page-number pagination [6]. It is defined in the github package and contains fields to support navigation using cursors provided in response Link headers [6]. The struct definition includes the following fields [6]: Page (string): For paginated result sets, the page of results to retrieve. PerPage (int): The number of results to include per page. First (int): The number of results to include per page (max 100), starting from the first matching result. Last (int): The number of results to include per page (max 100), starting from the last matching result. After (string): A cursor, as given in the Link header; the query only searches for results after this cursor. Before (string): A cursor, as given in the Link header; the query only searches for results before this cursor. Cursor (string): A cursor, as given in the Link header; the query continues the search using this cursor. These options can be passed directly to list methods or embedded within more specific options structs [7][8]. Cursor information is also available via the github.Response struct returned by API calls [7][6].
Citations:
- 1: feat: Support Code Security Configurations API google/go-github#3319
- 2: https://docs.github.com/en/rest/code-security/configurations
- 3: https://docs.github.com/pt/enterprise-cloud@latest/rest/code-security/configurations
- 4: chore(deps): update module github.com/google/go-github/v74 to v82 - autoclosed AkashRajpurohit/git-sync#149
- 5: dep: update module github.com/google/go-github/v70 to v80 - autoclosed lindell/multi-gitter#602
- 6: https://github.com/google/go-github/blob/master/github/github.go
- 7: https://pkg.go.dev/github.com/google/go-github/v73@v73.0.1-0.20250721233455-99ffabedb43d
- 8: https://github.com/google/go-github/blob/master/README.md
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import urllib.request
base = "https://raw.githubusercontent.com/google/go-github/v73.0.0/github"
files = [f"{base}/code_security_configurations.go", f"{base}/organizations.go"]
for url in files:
print(f"\n--- {url.split('/')[-1]}")
try:
text = urllib.request.urlopen(url, timeout=20).read().decode()
except Exception as e:
print(f"ERROR: {e}")
continue
lines = text.splitlines()
for i, line in enumerate(lines, 1):
if "GetRepositoriesForCodeSecurityConfiguration" in line or "GetCodeSecurityConfigurations" in line or "ListCursorOptions" in line:
start = max(1, i-15)
end = min(len(lines), i+45)
for j in range(start, end+1):
print(f"{j:4}: {lines[j-1]}")
PYRepository: flanksource/config-db
Length of output: 502
🌐 Web query:
GitHub API code security configurations list code security configurations cursor-based pagination before after
💡 Result:
The GitHub REST API endpoints for listing code security configurations support cursor-based pagination using the before and after query parameters [1][2][3]. These parameters function as follows: after (string): If specified, the API returns results immediately following the provided cursor [1][3][4]. before (string): If specified, the API returns results immediately preceding the provided cursor [1][3][4]. These cursors are obtained from the Link header returned in the API response [1][2][3]. Additionally, the per_page parameter can be used to control the number of results returned per page, with a maximum value of 100 [1][3][5]. For general guidance on implementing pagination with GitHub's REST API, including handling Link headers, developers should refer to the official documentation on using pagination in the REST API [6][7].
Citations:
- 1: https://docs.github.com/en/enterprise-cloud@latest/rest/code-security/configurations?apiVersion=2026-03-10
- 2: https://docs.github.com/en/rest/code-security/configurations
- 3: https://docs.github.com/en/rest/code-security/configurations?apiVersion=2022-11-28
- 4: https://docs.github.com/en/enterprise-server@3.21/rest/code-security/configurations?apiVersion=2026-03-10
- 5: https://docs.github.com/en/enterprise-server@3.17/rest/code-security/configurations
- 6: https://github.com/github/docs/blob/main/content/rest/using-the-rest-api/using-pagination-in-the-rest-api.md
- 7: https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api?apiVersion=2026-03-10
Paginate both code security list calls.
GetCodeSecurityConfigurations and GetRepositoriesForCodeSecurityConfiguration are not unbounded; they return paginated responses and do not automatically fetch all results. Loop with the returned response (or go-github cursors) and pass the appropriate cursor options, or attachments and/or configurations can be silently dropped for larger organizations.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scrapers/github/code_security.go` around lines 20 - 53, Update
scrapeCodeSecurityConfigurations and codeSecurityConfigurationRepositories to
iterate through every paginated response from GetCodeSecurityConfigurations and
GetRepositoriesForCodeSecurityConfiguration, passing the returned cursor or page
options on each request. Preserve existing error handling and aggregate all
configurations and repositories before building results so larger organizations
are fully represented.
1 participant
What
Why
Summary by CodeRabbit
New Features
doctorcommand to diagnose GitHub scraper configurations from a local file or scraper ID.Documentation
Bug Fixes