Skip to content

fix(records): honor network.tor in records search/fetch/download - #129

Merged
JordanCoin merged 6 commits into
mainfrom
fix/records-tor-egress
Nov 6, 2026
Merged

JordanCoin merged 6 commits into
mainfrom
fix/records-tor-egress

Conversation

@JordanCoin

@JordanCoin JordanCoin commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Closes #130: records search, --all-sources, fetch (documentcloud, nara) and download (muckrock) ignored network.tor and went out on the real IP.

With {"network":{"tor":true}} in config.json, records.get_adapter() built every adapter with the RecordAdapter default policy (DIRECT). These commands went out on the real IP with no Tor check and no route notice:

  • records search -s <source> (every source except cia)
  • records search --all-sources (all sources, cia included, skipping the Tor gate)
  • records fetch -s documentcloud, records fetch -s nara without --objects/--list-objects
  • records download -s muckrock

What changed

  • get_adapter(name, *, egress=None) passes the policy to the adapter constructor (RecordAdapter.__init__(egress=...), the same mechanism the cia/sec/nara-objects paths already used).
  • Every records CLI call site takes its policy from _records_policy_or_exit() (renamed from _cia_policy_or_exit, already source-agnostic): _egress_policy_from(load_config()) plus the fail-closed _check_tor_or_exit. Each command prints the route ("via Tor, real IP hidden" / "direct, real IP") before sending. --all-sources checks Tor once, before any request.
  • The cia search and nara --objects paths no longer build a second adapter; they reuse the one from get_adapter.
  • No adapter changes: every adapter in openfoia/records/ builds its client via egress_client(self._egress, ...).
  • Tests (tests/test_security_egress_cli.py): for search (single and all-sources), fetch (documentcloud, nara) and download (muckrock), with Tor on in config, every adapter built has a TOR policy; with Tor unreachable, exit is non-zero and zero HTTP requests are made. One test asserts --all-sources builds cia under TOR. All 11 new tests fail on main.
  • Existing tests that stub get_adapter now accept **_.

Composition with #128

#128 adds get_adapter(name, api_data_gov_key=...) at the same call sites. Whichever merges second needs a small rebase: both kwargs pass through, and get_adapter should withhold api_data_gov_key when egress.is_tor (the key identifies the user).

Local: ruff check + format clean; pytest tests -q 573 passed, 2 failed (pre-existing TestLLMExtraction, need Ollama).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce

JordanCoin and others added 5 commits November 6, 2026 11:03
records.api_data_gov_key (OPENFOIA_API_DATA_GOV_KEY, _api_data_gov_key
accepted) replaces the shared DEMO_KEY for fec, govinfo and regulations.
The three adapters now share a DataGovAdapter base (DEMO_KEY default,
key redaction). get_adapter() takes the key and hands it to DataGovAdapter
subclasses; records search and crossref pass it from config. FEC and
Regulations.gov send the key as a query parameter, so their error text
(httpx puts the URL in HTTPStatusError) is now redacted the way GovInfo's
already was; with DEMO_KEY that never mattered, with a personal key it
would have reached logs and terminal output. The key is also masked by
`config --show`.

network.max_attempts (OPENFOIA_NETWORK_MAX_ATTEMPTS, default 3) is carried
on EgressPolicy, set in _egress_policy_from, and handed to RetryTransport
by egress_client, so net.py still does not import config. Values below 1
are clamped to 1.

Log levels: rate limits and rejected queries that already return a clear
message to the user (FEC/Regulations/GovInfo 429, DocumentCloud 429/503
and Cloudflare block, USAspending 422) drop from warning to info.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
…attempts from file

A personal key is tied to the signup email, so sending it over Tor
identifies the user anyway. crossref now falls back to DEMO_KEY in Tor
mode and logs that once. A string max_attempts in config.json no longer
raises TypeError in every egress client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
… re-inject a personal key

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
get_adapter() built every adapter with the RecordAdapter default policy
(DIRECT), so with {"network":{"tor":true}} these commands went out on the
real IP with no Tor check and no route notice: records search -s <non-cia>,
records search --all-sources (cia included, skipping the Tor gate),
records fetch -s documentcloud, records fetch -s nara without
--objects/--list-objects, and records download -s muckrock.

get_adapter() now takes egress= and passes it to the adapter constructor.
Every CLI call site gets its policy from _records_policy_or_exit (renamed
from _cia_policy_or_exit, which was already source-agnostic): config.network
plus the fail-closed Tor reachability check, then prints the
"via Tor / real IP" route. All adapters already build clients through
egress_client(self._egress, ...), so no adapter changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
get_adapter takes both egress and api_data_gov_key; over Tor the literal
DEMO_KEY is passed so no env fallback can re-inject a personal key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWQw8DDrDZ2tFzswSSEtce
@JordanCoin

Copy link
Copy Markdown
Owner Author

Review (Claude opus reviewer, read-only, 2026-11-06): MERGE. Enumerated every adapter construction site in cli.py, crossref.py, pipeline/web.py, agent.py, server.py, campaign.py, cases.py: all now carry the config policy or exit before building anything; no DIRECT default remains. Route notice goes to stderr after the Tor check and names the route actually taken; --all-sources checks Tor once before any request including the nara has_key probe; check_tor never raises; renamed helper is byte-identical to the old one and no stale references remain; fail-closed tests prove zero requests via a patched AsyncHTTPTransport that both the Tor and retry transports terminate in.

This branch now also carries #128 (merged in, conflicts resolved as the reviewer specified): get_adapter takes egress and api_data_gov_key, and passes the literal DEMO_KEY over Tor. New test covers that with OPENFOIA_GOVINFO_API_KEY set. Full suite 583 passed; the 2 failures are the pre-existing Ollama tests.

Below threshold, tracked separately: NARA key, OPENFOIA_MUCKROCK_TOKEN and OPENFOIA_GOVINFO_API_KEY still travel over Tor on paths outside this diff; openfoia browse ignores network.tor unless --tor is passed; the Tor-down error goes to stdout under --json.

@JordanCoin
JordanCoin merged commit 0bc0eba into main Nov 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

records search/fetch/download ignore network.tor and go out on the real IP

1 participant