Repository navigation
feat(config): api.data.gov key and retry count in config (#56) - #128
Conversation
records.api_data_gov_key (OPENFOIA_API_DATA_GOV_KEY, _api_data_gov_key accepted) replaces the shared DEMO_KEY for fec, govinfo and regulations. The three adapters now share a DataGovAdapter base (DEMO_KEY default, key redaction). get_adapter() takes the key and hands it to DataGovAdapter subclasses; records search and crossref pass it from config. FEC and Regulations.gov send the key as a query parameter, so their error text (httpx puts the URL in HTTPStatusError) is now redacted the way GovInfo's already was; with DEMO_KEY that never mattered, with a personal key it would have reached logs and terminal output. The key is also masked by `config --show`. network.max_attempts (OPENFOIA_NETWORK_MAX_ATTEMPTS, default 3) is carried on EgressPolicy, set in _egress_policy_from, and handed to RetryTransport by egress_client, so net.py still does not import config. Values below 1 are clamped to 1. Log levels: rate limits and rejected queries that already return a clear message to the user (FEC/Regulations/GovInfo 429, DocumentCloud 429/503 and Cloudflare block, USAspending 422) drop from warning to info. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
…attempts from file A personal key is tied to the signup email, so sending it over Tor identifies the user anyway. crossref now falls back to DEMO_KEY in Tor mode and logs that once. A string max_attempts in config.json no longer raises TypeError in every egress client. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
|
Review (Claude opus reviewer, read-only, 2026-11-06): MERGE. Verified the key never reaches logs, SearchResult.error, --json, describe_egress, FtM or case notes (all exception paths redact first; httpx logger never at INFO). RecordAdapter interface unchanged; get_adapter kwarg consistent across callers; log-level drops lose no user-visible signal since every message still lands in SearchResult.error. Follow-up commit: personal api.data.gov key is withheld in Tor mode (crossref falls back to DEMO_KEY, logs once) because the key identifies the user regardless of route; string max_attempts in config.json is coerced instead of raising. Side finding, pre-existing on main and out of scope here: records search / --all-sources / fetch (documentcloud, nara) / download (muckrock) build adapters with the default DIRECT policy and ignore network.tor. Tracked as its own issue. |
… re-inject a personal key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
Part of #56
In
records.api_data_gov_key(envOPENFOIA_API_DATA_GOV_KEY, file keyapi_data_gov_keyor_api_data_gov_key). Used byfec,govinfoandregulationswhen no explicit key is passed;DEMO_KEYstays the default. The three adapters share a newDataGovAdapterbase (DEMO_KEY default, key redaction).get_adapter(name, api_data_gov_key=...)hands the key toDataGovAdaptersubclasses;records search(single and--all-sources) andcrossrefpass it from config. The adapter docstrings and therecordshelp text say how to set it.config --showmasks it. GovInfo's existingOPENFOIA_GOVINFO_API_KEYstill works as a fallback.HTTPStatusError, so a personal key would have reached logs and terminal output. Their error text now goes through the same_redactGovInfo already used.network.max_attempts(envOPENFOIA_NETWORK_MAX_ATTEMPTS, default 3, 1 disables retry). Carried onEgressPolicy, set in_egress_policy_from, passed toRetryTransportbyegress_client;net.pydoes not import config. Values below 1 are clamped to 1.openfoia/records/*.py: rate limits and rejected queries that already return a clear message to the user drop from warning to info (FEC/Regulations/GovInfo 429, DocumentCloud 429/503 and Cloudflare block, USAspending 422). Timeouts, unexpected errors and failed fetches stay at warning. No message rewrites.Out of scope
records searchandrecords fetchbuild adapters with a defaultEgressPolicy, not_egress_policy_from(config), sonetwork.max_attempts(andnetwork.tor, which was already the case before this PR) does not apply there; crossref,records filings, web ingest and the CIA path do honor it. Worth its own change.Tests
tests/test_config_network_keys.py: defaults, file key and underscore variant, env-over-file precedence for both settings, bad env value ignored,config --showmasking,records search -s fecsending the configured key and not echoing it in the error, andmax_attempts=1via_egress_policy_fromsending a single request on 429. Existingget_adapter/_get_available_sourcestest stubs now accept the new keyword.ruff check+ruff format --checkclean. Full suite: 569 passed, 26 skipped, 2 failed (TestLLMExtraction, needs Ollama, failing on main too).🤖 Generated with Claude Code
https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce