Repository navigation
fix(records): honor network.tor in records search/fetch/download - #129
Conversation
records.api_data_gov_key (OPENFOIA_API_DATA_GOV_KEY, _api_data_gov_key accepted) replaces the shared DEMO_KEY for fec, govinfo and regulations. The three adapters now share a DataGovAdapter base (DEMO_KEY default, key redaction). get_adapter() takes the key and hands it to DataGovAdapter subclasses; records search and crossref pass it from config. FEC and Regulations.gov send the key as a query parameter, so their error text (httpx puts the URL in HTTPStatusError) is now redacted the way GovInfo's already was; with DEMO_KEY that never mattered, with a personal key it would have reached logs and terminal output. The key is also masked by `config --show`. network.max_attempts (OPENFOIA_NETWORK_MAX_ATTEMPTS, default 3) is carried on EgressPolicy, set in _egress_policy_from, and handed to RetryTransport by egress_client, so net.py still does not import config. Values below 1 are clamped to 1. Log levels: rate limits and rejected queries that already return a clear message to the user (FEC/Regulations/GovInfo 429, DocumentCloud 429/503 and Cloudflare block, USAspending 422) drop from warning to info. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
…attempts from file A personal key is tied to the signup email, so sending it over Tor identifies the user anyway. crossref now falls back to DEMO_KEY in Tor mode and logs that once. A string max_attempts in config.json no longer raises TypeError in every egress client. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
… re-inject a personal key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
get_adapter() built every adapter with the RecordAdapter default policy
(DIRECT), so with {"network":{"tor":true}} these commands went out on the
real IP with no Tor check and no route notice: records search -s <non-cia>,
records search --all-sources (cia included, skipping the Tor gate),
records fetch -s documentcloud, records fetch -s nara without
--objects/--list-objects, and records download -s muckrock.
get_adapter() now takes egress= and passes it to the adapter constructor.
Every CLI call site gets its policy from _records_policy_or_exit (renamed
from _cia_policy_or_exit, which was already source-agnostic): config.network
plus the fail-closed Tor reachability check, then prints the
"via Tor / real IP" route. All adapters already build clients through
egress_client(self._egress, ...), so no adapter changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
get_adapter takes both egress and api_data_gov_key; over Tor the literal DEMO_KEY is passed so no env fallback can re-inject a personal key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
|
Review (Claude opus reviewer, read-only, 2026-11-06): MERGE. Enumerated every adapter construction site in cli.py, crossref.py, pipeline/web.py, agent.py, server.py, campaign.py, cases.py: all now carry the config policy or exit before building anything; no DIRECT default remains. Route notice goes to stderr after the Tor check and names the route actually taken; --all-sources checks Tor once before any request including the nara has_key probe; check_tor never raises; renamed helper is byte-identical to the old one and no stale references remain; fail-closed tests prove zero requests via a patched AsyncHTTPTransport that both the Tor and retry transports terminate in. This branch now also carries #128 (merged in, conflicts resolved as the reviewer specified): get_adapter takes egress and api_data_gov_key, and passes the literal DEMO_KEY over Tor. New test covers that with OPENFOIA_GOVINFO_API_KEY set. Full suite 583 passed; the 2 failures are the pre-existing Ollama tests. Below threshold, tracked separately: NARA key, OPENFOIA_MUCKROCK_TOKEN and OPENFOIA_GOVINFO_API_KEY still travel over Tor on paths outside this diff; |
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
Closes #130: records search, --all-sources, fetch (documentcloud, nara) and download (muckrock) ignored network.tor and went out on the real IP.
With
{"network":{"tor":true}}in config.json,records.get_adapter()built every adapter with theRecordAdapterdefault policy (DIRECT). These commands went out on the real IP with no Tor check and no route notice:records search -s <source>(every source except cia)records search --all-sources(all sources, cia included, skipping the Tor gate)records fetch -s documentcloud,records fetch -s narawithout--objects/--list-objectsrecords download -s muckrockWhat changed
get_adapter(name, *, egress=None)passes the policy to the adapter constructor (RecordAdapter.__init__(egress=...), the same mechanism the cia/sec/nara-objects paths already used)._records_policy_or_exit()(renamed from_cia_policy_or_exit, already source-agnostic):_egress_policy_from(load_config())plus the fail-closed_check_tor_or_exit. Each command prints the route ("via Tor, real IP hidden" / "direct, real IP") before sending.--all-sourceschecks Tor once, before any request.--objectspaths no longer build a second adapter; they reuse the one fromget_adapter.openfoia/records/builds its client viaegress_client(self._egress, ...).tests/test_security_egress_cli.py): for search (single and all-sources), fetch (documentcloud, nara) and download (muckrock), with Tor on in config, every adapter built has a TOR policy; with Tor unreachable, exit is non-zero and zero HTTP requests are made. One test asserts--all-sourcesbuilds cia under TOR. All 11 new tests fail on main.get_adapternow accept**_.Composition with #128
#128 adds
get_adapter(name, api_data_gov_key=...)at the same call sites. Whichever merges second needs a small rebase: both kwargs pass through, andget_adaptershould withholdapi_data_gov_keywhenegress.is_tor(the key identifies the user).Local: ruff check + format clean;
pytest tests -q573 passed, 2 failed (pre-existingTestLLMExtraction, need Ollama).🤖 Generated with Claude Code
https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce