Repository navigation
feat(tracing): fail-closed rate gate + breaker feed on ingest (#831) - #839
Merged
Merged
Conversation
Producer side of the audit ingest rate limit (audit enforcement landed in #835; SDK breaker in campus-api-python#98). Design: design comment on campus#538. - AuditClient traces.new feeds the campus_python.audit.ratelimit breaker on every ingest response before raising: 429 trips the bucket (Retry-After + tripped key from the body), 2xx clears expired trips, availability errors neutral. Observation is guarded — a breaker failure or stale SDK can never break ingestion. - New check_rate_gate() before_request hook (registered before start_span, so gated requests never open a span): when AUDIT_TRACING_FAIL_CLOSED=1, requests whose identity bucket is tripped get 503 + Retry-After at entry. Per-identity where identity is resolvable (app-level hook ordering), degrading to the producer's learned apikey= fallback bucket or the wildcard. Off by default — producers stay fail-open until flipped per the ratified rollout. - classroom (and any producer on campus.audit.middleware) inherits the gate with a campus-suite bump; no producer-side code needed there. AUDIT_TRACING_FAIL_CLOSED is a new deployment variable; document+flip per producer on dev as the #831 verification step.
CI type checks install campus-api-python from git main, which lacks campus_python.audit.ratelimit until campus-api-python#98 merges — the static imports failed pyright. importlib.import_module with a string name is opaque to type checkers and keeps the stale-SDK degradation runtime behavior under any install. Stale-SDK test now patches importlib.import_module to match.
CI type checks and unit tests resolve campus_python.audit.ratelimit from the poetry.lock-pinned campus-api-python commit; bumping the lock to main@9caf58b (campus-api-python#98 merge) makes the breaker module visible to both, un-skipping the rate-gate suite on CI.
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Producer lane of #831 (Phase 3 of #538) — wires the campus-python circuit breaker (campus-api-python#98) into the campus producers. Audit-side enforcement landed in #835. Design (ratified): Phase 3 design comment on #538.
What
campus/audit/client/v1/traces.py):AuditClient.traces.new— the ingest path for auth, api, and classroom (classroom importscampus.audit.middlewarefrom this package) — now feeds the SDK breaker on every ingest response before raising: 429 trips the bucket (Retry-After + tripped key from the body), 2xx clears expired trips, availability errors neutral. The observation is guarded (contextlib.suppress+ lazy import): a breaker failure or a stalecampus_pythonwithoutaudit.ratelimitcan never break span ingestion.campus/audit/middleware): newcheck_rate_gate()before_requesthook, registered beforestart_spanso a gated request never opens a span (andend_spanno-ops — no wasted ingest against a bucket audit is rejecting). Opt-in per deployment viaAUDIT_TRACING_FAIL_CLOSED=1: tripped buckets short-circuit with503 RATE_LIMITED+Retry-Afterbefore any view runs.apikey=fallback bucket (learned from the first 429 body), or the wildcard before anything is learned.What producers need
Nothing code-wise: classroom (and any
campus.audit.middlewareconsumer) inherits gate + feed with acampus-suitelock bump; auth/api get it when this merges toweekly(they deploy from weekly). The flip is a Railway variable per producer.Testing
tests/unit/audit/test_rate_gate.py(10 tests): flag-off fail-open even when tripped; 503 + Retry-After for the tripped identity; other identities unaffected; learned-fallback and wildcard gating for identity-less requests; availability-error neutrality; verifying-state gate-open + 2xx-clears; client feed trips-and-raises / 201-no-trip; stale-SDK silent degradation.unit,integration,contract,sanityall green locally;ruffclean on touched files.Verification plan (after merge, on dev)
AUDIT_RATE_LIMIT_PER_MINUTE=1on campus.audit briefly → trip a bucket via ingest.AUDIT_TRACING_FAIL_CLOSED=1on one producer → confirm 503 + Retry-After on matching requests while others pass.Closes #831 together with campus-api-python#98 (base is
weekly, so closing by hand per convention).