Skip to content

fix(evolvespec): name the spec field an unpriced dollar cap needs - #1185

Merged
Birfy merged 1 commit into
mainfrom
agent/1183-max-usd-refusal
Nov 20, 2026
Merged

Birfy merged 1 commit into
mainfrom
agent/1183-max-usd-refusal

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1183.

What changed

  • agentdescent/evolvespec.py (_limit_from_knobs): the three knobs are validated into locals through _cap_value first, then an unpriced max_usd is refused with evolve.max_usd needs evolve.usd_per_call beside it: a dollar cap cannot be charged without a per-call price. build_call_limit is untouched, so the CLI and run-store wording stays exactly as shipped in feat(run): stop a worker run on max_calls / max_usd at the call (#1166) #1167.
  • tests/test_max_usd_refusal_hint.py: the unpriced spec is refused at compose and through cli evolve (exit 2, same field named, no run store created), and a priced spec still composes with both numbers. The ordering is pinned by the existing test_a_non_numeric_max_usd_is_refused_with_the_field_named, which still gets "not a number".
  • changelog.d/1183-unpriced-spec-usd.md.

Red first at f5017f93 with only the new test file in the tree: 1 failed, 1 passed -- tests/test_max_usd_refusal_hint.py::test_an_unpriced_spec_dollar_cap_names_the_spec_field, assert 'evolve.usd_per_call' in 'max_usd needs usd_per_call: a dollar cap cannot be charged without a per-call price (pass --usd-per-call on the command line)'.

Tip SHA: d7200efa3bf478dd55b08f9613588dca82cefd18

Ran:

  • Full suite at this tip, 8 parallel shards over ls tests/test_*.py (513 files), PYTHONPATH=. pytest -o addopts="" -p no:cacheprovider -q: 7917 passed, 20 failed, 10 errors, 51 skipped.
  • Those 30 red/error ids are pre-existing environment rows (10 Docker test_sandbox_container.py errors, plus test_apply_noop, test_dir_evolution, test_era_srbench, test_genesis_example, test_integrations, test_skill_dir_example, test_skill_library_*). The base comparison was run on the same nine files at pristine f5017f93, in a separate git worktree of the same clone, as one direct pytest over those nine paths (not the sharded chunk$i loop): PYTHONPATH=. python -m pytest tests/test_apply_noop.py tests/test_dir_evolution.py tests/test_era_srbench.py tests/test_genesis_example.py tests/test_integrations.py tests/test_sandbox_container.py tests/test_skill_dir_example.py tests/test_skill_library_apply_install.py tests/test_skill_library_example.py -o addopts="" -p no:cacheprovider -q -> 20 failed, 437 passed, 15 skipped, 10 errors, and diff of the two sorted id lists is empty: the same 30 ids, line for line, at the base and at this tip.
  • tests/test_max_usd_refusal_hint.py tests/test_budget_guard.py tests/test_budget_round_warning.py tests/test_evolvespec.py tests/test_budget.py tests/test_call_budget.py tests/test_run_verbs_json_spec_error.py: 125 passed.
  • python -m tools.gen_api_docs --check: up to date. python -m tools.changelog_fragments check: OK.

Not run: mkdocs build --strict -- no docs/ or mkdocs.yml change, so the docs build is not in scope for this diff.

)

A spec whose `evolve` block set `max_usd` without `usd_per_call` was refused with
"pass --usd-per-call on the command line", which cannot help: `compose` refuses
the spec before `cmd_evolve` reads its own flag, so the flag produced the same
sentence at the same exit code. The refusal now names
`evolve.usd_per_call`, the field the author can actually add.

The three knobs are validated into locals before the pair is judged, so a
non-numeric `max_usd` still reports "not a number" rather than the pair message.
`build_call_limit` and the CLI/run-store wording are unchanged.
@HugoXK
HugoXK marked this pull request as ready for review November 20, 2026 11:29
@github-actions github-actions Bot added risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff agent-pr Pull request opened by an agent labels Nov 20, 2026
@HugoXK
HugoXK requested a review from Birfy November 20, 2026 12:31
@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory read at d7200efa3bf478dd55b08f9613588dca82cefd18 — from the QA seat, not a counting PASS (.github/agent/review-policy.json counts Birfy only), and I take no merge action.

T2 by path (agentdescent/evolvespec.py, package code; 3 files +73/−8 = 81 changed lines, under the 250-line cap). Live in the same read: open, draft: false, mergeable_state: clean, base f5017f93, body Tip SHA: == head, Fixes #1183.

Tip SHA: d7200ef
Ran:

  • pytest -q -n 8 at the tip → 21 failed, 7904 passed, 63 skipped, 10 errors (277.84s); the same command on base f5017f93 → 21 failed, 7902 passed, 63 skipped, 10 errors (285.59s). The 31 red ids (10 docker-sandbox errors + 21 host-state failures) are the same set id for id: 0 new red, 0 vanished.
  • fail-on-main: tests/test_max_usd_refusal_hint.py only, onto f5017f93 → 1 failed, 1 passed — the body's row reproduces exactly.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 5 fragments OK.
    Not run:
  • mkdocs build --strict — no docs/ or mkdocs.yml change in the diff.
  • Docker sandboxes, wheel smoke, Ark real-model — the red ids above are host state; the cap is arithmetic, no model call.
  • risk_gate.py — no gh on this box; the tier above is computed from .github/agent/risk-rules.json.

Findings — none on the code. The probes I ran myself at the tip:

  • compose(max_usd=1) → SpecError: evolve.max_usd needs evolve.usd_per_call beside it: a dollar cap cannot be charged without a per-call price; the string has no --usd-per-call.
  • compose(max_usd="nope") → evolve.max_usd: must be a non-negative number, got 'nope' — the ordering fix holds, the pair message does not swallow the wrong field.
  • compose(max_usd=1, usd_per_call=0.02) → CallLimit(max_calls=None, max_usd=1.0, usd_per_call=0.02); compose(max_calls=3) → max_calls=3.
  • Through cli evolve --json with the unpriced spec → rc 2, same sentence, no run store created. With --usd-per-call 0.02 on the command line it still refuses (as on base), but the tip names the spec field where base printed the circular flag advice.

Two record items, yours to decide:

  1. Labels: the PR carries agent-pr + risk:T2 only — no nature label and no area:*. By the owner's rule it wants bug + area:evolve (and risk:T2 is the controller's).
  2. The body's full-suite line (7917 passed, 20 failed, 51 skipped) is from 8 sharded chunks and differs from one command (7904 / 21 / 63); the material fact, the red-id set, matches base either way.

No UI surface: a spec-validation sentence; nothing rendered changes.

@HugoXK HugoXK added bug Something isn't working area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies labels Nov 20, 2026
@Birfy

Birfy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: d7200ef
Ran:

  • .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1185 → T2 (81 lines / 3 files)
  • pytest -o addopts="" -p no:cacheprovider -q tests/test_max_usd_refusal_hint.py tests/test_budget_guard.py tests/test_budget_round_warning.py tests/test_evolvespec.py tests/test_budget.py tests/test_call_budget.py tests/test_run_verbs_json_spec_error.py at this tip → 124 passed
  • python -m tools.changelog_fragments check → 5 fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Required CI at this tip → green (build, test (3.9)–test (3.13), otel-wire; deploy skipped; risk tier / reviewed tip report-only)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • mkdocs build --strict — no docs/ or mkdocs.yml change in the diff
    Findings:
  • No actionable defects. _limit_from_knobs now validates the three knobs first, then refuses an unpriced max_usd by naming evolve.usd_per_call, so compose no longer points at a --usd-per-call flag it will not read. A non-numeric max_usd still fails as "not a number" before the pair check. build_call_limit and the CLI/run-store wording are unchanged, as the body says.
    Verdict: PASS

@Birfy
Birfy merged commit e36caec into main Nov 20, 2026
10 checks passed
@Birfy
Birfy deleted the agent/1183-max-usd-refusal branch November 20, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies bug Something isn't working risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(spec): a spec's evolve.max_usd is refused by naming --usd-per-call, a flag that cannot help it

2 participants