Repository navigation
Unified networking layer: rate limiting, retries, error handling #56
Description
Every adapter creates its own httpx.AsyncClient with ad-hoc error handling. Need a shared networking layer.
Problems
- No unified rate limiting — each adapter guesses independently
- 404s not handled gracefully (ProPublica blows up on special chars)
- No retry with backoff on transient failures
- No connection pooling across adapters
- DEMO_KEY shared across all api.data.gov users — we hit limits fast
- Crossref runs all entities even when dedup/confidence filters should reduce them
Solution
- Shared
APIClientclass wrapping httpx with per-host rate limiters - Token bucket or leaky bucket rate limiting per domain
- Automatic retry on 429/503 with exponential backoff
- URL-encode entity names properly before sending
- Connection pooling via shared client instances
- Log warnings (not errors) for expected failures
Activity
fix: crossref rate limiting, dedup, and error handling
JordanCoin commentedon Aug 12, 2026
Rescoped under #66 (private search substrate). v4.0.0 already landed the base —
openfoia/net.py, a fail-closed Tor egress choke point every adapter routes through. This issue is now specifically the search-layer rate limiting / retries / backpressure on top of that choke point, not a from-scratch networking layer. It blocks the search epic.fix(net): keep env proxies in direct mode under RetryTransport (#56)
feat(net): retry and per-host spacing in egress_client (#56) (#127)
JordanCoin commentedon Nov 6, 2026
Progress 2026-11-06: #127 merged (895fa62). In: transport-level retry for GET/HEAD on 429/503/ConnectError/ReadTimeout (3 attempts, backoff+jitter, Retry-After honored to 30s), per-host 10 req/s spacing, URL-encoding of ids and entity names in crossref/FEC/ProPublica/OpenCorporates/Regulations/GovInfo, env proxies preserved in direct mode, Tor path untouched. All applied inside egress_client so no adapter changed.
Still open from this issue: shared client / connection pooling across adapters, retry count via config, api.data.gov key instead of shared DEMO_KEY, crossref pre-filtering by dedup/confidence, warning-vs-error log levels. Next slice starting now.
feat(config): api.data.gov key and retry count in config (#56) (#128)
fix(records): honor network.tor in records search/fetch/download (#129)
JordanCoin commentedon Nov 6, 2026
Done across #127 and #128: transport-level retry with backoff and Retry-After, per-host spacing, URL-encoded ids, api.data.gov key and retry count in config, env proxies preserved, log-level pass. Deferred: a shared client with connection pooling across adapters. Open a new issue if that becomes measurable.