Skip to content

feat(config): api.data.gov key and retry count in config (#56) - #128

Merged
JordanCoin merged 3 commits into
mainfrom
feat/56-config-keys
Nov 6, 2026
Merged

JordanCoin merged 3 commits into
mainfrom
feat/56-config-keys

Conversation

@JordanCoin

Copy link
Copy Markdown
Owner

Part of #56

In

  • records.api_data_gov_key (env OPENFOIA_API_DATA_GOV_KEY, file key api_data_gov_key or _api_data_gov_key). Used by fec, govinfo and regulations when no explicit key is passed; DEMO_KEY stays the default. The three adapters share a new DataGovAdapter base (DEMO_KEY default, key redaction). get_adapter(name, api_data_gov_key=...) hands the key to DataGovAdapter subclasses; records search (single and --all-sources) and crossref pass it from config. The adapter docstrings and the records help text say how to set it. config --show masks it. GovInfo's existing OPENFOIA_GOVINFO_API_KEY still works as a fallback.
  • Key redaction for FEC and Regulations.gov. They send the key as a query parameter, and httpx puts the URL in HTTPStatusError, so a personal key would have reached logs and terminal output. Their error text now goes through the same _redact GovInfo already used.
  • network.max_attempts (env OPENFOIA_NETWORK_MAX_ATTEMPTS, default 3, 1 disables retry). Carried on EgressPolicy, set in _egress_policy_from, passed to RetryTransport by egress_client; net.py does not import config. Values below 1 are clamped to 1.
  • Log levels in openfoia/records/*.py: rate limits and rejected queries that already return a clear message to the user drop from warning to info (FEC/Regulations/GovInfo 429, DocumentCloud 429/503 and Cloudflare block, USAspending 422). Timeouts, unexpected errors and failed fetches stay at warning. No message rewrites.

Out of scope

  • Shared client / connection pooling across adapters.
  • records search and records fetch build adapters with a default EgressPolicy, not _egress_policy_from(config), so network.max_attempts (and network.tor, which was already the case before this PR) does not apply there; crossref, records filings, web ingest and the CIA path do honor it. Worth its own change.
  • A personal api.data.gov key is tied to the signup email, so it identifies the user even over Tor. The docstring says so; whether to withhold the key in Tor mode is a separate decision.

Tests

tests/test_config_network_keys.py: defaults, file key and underscore variant, env-over-file precedence for both settings, bad env value ignored, config --show masking, records search -s fec sending the configured key and not echoing it in the error, and max_attempts=1 via _egress_policy_from sending a single request on 429. Existing get_adapter / _get_available_sources test stubs now accept the new keyword.

ruff check + ruff format --check clean. Full suite: 569 passed, 26 skipped, 2 failed (TestLLMExtraction, needs Ollama, failing on main too).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce

JordanCoin and others added 2 commits November 6, 2026 11:03
records.api_data_gov_key (OPENFOIA_API_DATA_GOV_KEY, _api_data_gov_key
accepted) replaces the shared DEMO_KEY for fec, govinfo and regulations.
The three adapters now share a DataGovAdapter base (DEMO_KEY default,
key redaction). get_adapter() takes the key and hands it to DataGovAdapter
subclasses; records search and crossref pass it from config. FEC and
Regulations.gov send the key as a query parameter, so their error text
(httpx puts the URL in HTTPStatusError) is now redacted the way GovInfo's
already was; with DEMO_KEY that never mattered, with a personal key it
would have reached logs and terminal output. The key is also masked by
`config --show`.

network.max_attempts (OPENFOIA_NETWORK_MAX_ATTEMPTS, default 3) is carried
on EgressPolicy, set in _egress_policy_from, and handed to RetryTransport
by egress_client, so net.py still does not import config. Values below 1
are clamped to 1.

Log levels: rate limits and rejected queries that already return a clear
message to the user (FEC/Regulations/GovInfo 429, DocumentCloud 429/503
and Cloudflare block, USAspending 422) drop from warning to info.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
…attempts from file

A personal key is tied to the signup email, so sending it over Tor
identifies the user anyway. crossref now falls back to DEMO_KEY in Tor
mode and logs that once. A string max_attempts in config.json no longer
raises TypeError in every egress client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
@JordanCoin

Copy link
Copy Markdown
Owner Author

Review (Claude opus reviewer, read-only, 2026-11-06): MERGE. Verified the key never reaches logs, SearchResult.error, --json, describe_egress, FtM or case notes (all exception paths redact first; httpx logger never at INFO). RecordAdapter interface unchanged; get_adapter kwarg consistent across callers; log-level drops lose no user-visible signal since every message still lands in SearchResult.error. Follow-up commit: personal api.data.gov key is withheld in Tor mode (crossref falls back to DEMO_KEY, logs once) because the key identifies the user regardless of route; string max_attempts in config.json is coerced instead of raising.

Side finding, pre-existing on main and out of scope here: records search / --all-sources / fetch (documentcloud, nara) / download (muckrock) build adapters with the default DIRECT policy and ignore network.tor. Tracked as its own issue.

… re-inject a personal key

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant