Skip to content

abuse tiers count pooled-scope 429s against the key that received them #100

Description

@CMGS

Problem

abuse.tiers suspends a key once its daily count of admission rejections crosses a tier. The counter (note_abuse, crates/handler/src/lib.rs) increments on every STOP_LIMIT_MSG 429, and limit_denied (crates/dag/src/nodes.rs) raises that one code for five different limits:

Node Scope of the limit Whose traffic fills it
tenant_rate tenant QPS every key of the tenant
rate_limit key QPS the key
product_qpm product QPM every key of the product
model_qpm model QPM every key calling the model
ak_tpm key TPM the key

Three of the five are pooled. A key that sends one request a second can collect 429s all day because a sibling key saturates the tenant, product, or model bucket, and those 429s count toward its own suspension. The sanction lands on a key that did nothing the limit was written to stop.

The order makes it worse: tenant_rate runs ahead of rate_limit, so when a flooding key saturates the tenant bucket, the flooder's rejections and the bystanders' rejections are indistinguishable — both are tenant-QPS denials.

Proposal

  1. Count only the limits the key owns: key QPS and key TPM. Pooled-scope denials (tenant QPS, product QPM, model QPM) get their own internal error code, still class Throttling, still 429 with the same client-visible shape; note_abuse keys on the key-scoped code only.
  2. Check key QPS ahead of tenant QPS at the admission sites (the model_access DAG, and the realtime, MCP and video-poll gates that repeat the sequence). A flooder then trips its own limit first and is counted; a bystander that stays under its own QPS reaches the pooled check and is not.

With both, a flooder cannot hide behind a pooled limit (its own QPS limit fires first), and a bystander cannot be suspended for a sibling's traffic.

Notes

  • A key with no qps of its own and no tokens_per_minute is never counted after this change. That is the intended reading: abuse tiers sanction a key for exceeding its own limits, so a key without limits has nothing to exceed. Document it in docs/configuration.md next to abuse:.
  • Reordering changes which bucket a rejected request consumes: today a request denied by tenant QPS never touches the key bucket; after the change a request denied by tenant QPS has already taken a key-QPS slot. State it in docs/governance.md.
  • Hot path: zero added work on an admitted request (same checks, different order); the 429 path is unchanged (one governance call).

Acceptance

  • Unit: a pooled-scope denial leaves abuse:{ak} untouched; a key-scoped denial increments it.
  • E2E: two keys in one tenant with a tenant QPS cap and abuse.tiers set; key A floods, key B sends under its own limit. A is suspended, B is not, and B's 429s during the flood carry the pooled code internally.
  • Docs: configuration.md, governance.md, security.md state which limits count.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions