Skip to content

fix(agents): count the tokens a failed call already billed - #1205

Open
HugoXK wants to merge 3 commits into
mainfrom
agent/failed-usage-tokens
Open

HugoXK wants to merge 3 commits into
mainfrom
agent/failed-usage-tokens

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1202.

What this changes

A failed model call today reports its wall-clock but not the tokens the provider already billed. Usage splits wall-clock failures (failures, failure_seconds) and counts them in summary(), but a call that failed after the provider had metered tokens adds nothing to prompt_tokens / completion_tokens / total_tokens — so a reader sees "3 failed" beside a token total that silently excludes them, and the token meter runs behind what the provider will charge.

This adds two fields and records what the provider actually reported:

  • Usage.failed_usage_attempts — failed provider attempts that reported usage.
  • Usage.failed_usage_tokens — the tokens those attempts reported.

Both default to 0, so an old payload or fixture that lacks them loads unchanged; nothing in agents.py serializes Usage, and evolution.py's --json usage dict gains the two keys only (names byte-identical to the fields).

Decision: failed calls with reported usage count toward totals; failed calls without usage count only as failed_calls

The rejected alternative was a reporting-only column that left the totals and the limit untouched — that keeps the limit underestimating what the provider charges, which is the defect.

Why it lands in max_tokens and the governor, not max_usd

max_usd is charged per call, not per token: budget.py:112-135 reserves n * usd_per_call before the call, _taken only grows, and there is no refund. A failed call is therefore already counted on the dollar side today — what was missing is only the token side. Adding these tokens moves the max_tokens meter (async_evolve.py:1109) and the governor's spend (:1223); it does not change max_usd.

How counts are recorded

  • Both cross-module HTTP error paths read the response body once, before record(...), outside the usage is not None guard, and parse counts from the whole body; detail still stops at strip()[:400] and is byte-identical to today's text.
  • _read_sse writes each chunk's usage into a seen dict, so a stream that dies after the provider billed still counts those tokens.
  • _usage_counts accepts {"usage": {...}} or top-level, both wire shapes, integers only, non-negative, booleans ignored. A total_tokens-only body is treated as no evidence — splitting it would be a guess, and the decision is to count on evidence only.
  • The new counters are added inside the same with self._lock block as prompt_tokens/completion_tokens, so the failure column is a subset of the total by construction, not a parallel tally.
  • TokenSink.record(..., reported_usage=...) forwards tokens as before and then calls Usage.note_failed_usage(...), which increments only the two new counters — no prompt_tokens, no calls, nothing double counted.

Invariants the tests pin

  • failed_usage_tokens <= total_tokens on both paths.
  • failed_usage_attempts <= calls on the direct path, where attempts and calls are the same thing.
  • On the spec/TokenSink path the column counts usage-bearing attempts, one actor call can be retried, so it may exceed calls; that path asserts only the token subset, and the docstring says so.

Red first

At the live main this branch merges (e43e847a206cf20206d5460c2e27756df92fac65),
clean worktree, only tests/test_failed_call_usage.py added:

python3 -m pytest -o addopts='' -p no:cacheprovider -q tests/test_failed_call_usage.py
16 failed, 2 passed in 7.68s

The two passers pin unchanged behaviour (a usage=None caller keeps the same error text; the subprocess and SDK paths add no tokens), so they are not part of the red count. Typical failures: TypeError: Usage.record() got an unexpected keyword argument 'reported_usage', AttributeError: 'Usage' object has no attribute 'failed_usage_attempts', and assert 0 == 100 where the base never parses the error body at all.

Green at the tip (18 tests): totals and the new column both move for a usage-bearing failure; a usage-less failure moves failed_calls only; an old five-key payload round-trips with both fields 0; the body-padded row pins "parse untruncated, detail unchanged at 400 bytes"; the streaming row pins a stream that dies after usage; the reconciliation row keeps one usage-bearing + one usage-less failure at 2 failed, 1 with usage; the meter row shows a usage-bearing failure can end the run at stop_reason == "max_tokens" where the same cap without it does not; the sink rows pin the counters-only path and a retried call reporting several attempts.

python3 -m pytest -o addopts='' -p no:cacheprovider -q tests/test_failed_call_usage.py
18 passed in 7.55s
python3 -m pytest -o addopts='' -p no:cacheprovider -q \
    tests/test_failed_call_usage.py tests/test_agents.py tests/test_usage.py \
    tests/test_token_budget.py tests/test_spec_usage.py tests/test_call_budget.py \
    tests/test_budget_governor.py tests/test_backward_compat.py
115 passed, 2 warnings in 21.27s
python3 -m tools.gen_api_docs --check      # docs/api.md is up to date
python3 -m tools.changelog_fragments check # 13 changelog fragment(s) OK

The tip is a plain merge of main at e43e847a into the branch (no rebase, no
force-push); its tree is byte-identical to the merge this change was verified on
before the push, and docs/api.md needed no regeneration after it.

Scope

agents.py (the Usage fields, record(), note_failed_usage(), TokenSink.record(), the two HTTPError sites, _read_sse, _usage_counts, summary()), the evolution.py usage whitelist dict (two additive keys only), docs/api.md regenerated by the tool, one changelog fragment, and the new test file. No engine behaviour change, no new RoundInfo field, no show projection change, no default value changed.

A 4xx body, or a stream that dies after its last chunk, can carry usage
numbers -- tokens the provider charged and reported. The adapter recorded
the call's wall-clock and dropped the tokens, so the run's totals (and the
max_tokens meter) under-reported what was spent.

Both body-reading sites now read the body once, before record(), and parse
counts from the untruncated text while detail stays cut at 400 bytes. The
accumulation happens in the same record() call that adds to the totals, so
the failed-with-usage column cannot drift from total_tokens. TokenSink
mirrors the counters only, so a spec run's provider attempts are visible
without counting anything twice. A failure with no usage records 0 tokens
and is only a failed attempt; no estimation tightens a limit.
`_usage_counts` and `_usage_from_body` return
`Optional[Tuple[int, int]]`, but `Tuple` was never imported: the
typing line stopped at `Sequence, runtime_checkable`. With
`from __future__ import annotations` the paste does not fail at
import, so the tests and CI stayed green, and the gap only showed
when the annotations were resolved -- `typing.get_type_hints` on
either function raised `NameError: name 'Tuple' is not defined`.
@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(agents): a failed model call reports its wall-clock but not the tokens the provider billed

1 participant