Repository navigation
feat(runstore): operator call ceiling AGENTDESCENT_MAX_CALLS (Partial #1082) - #1181
Conversation
|
Verdict: CHANGES (advisory — Tip SHA: Ran (at this tip unless noted):
Not run:
Blocking:
Non-blocking:
Everything else I checked matches the claim: the clamp is a min in both paths and a caller cannot raise it ( Risk / merge: |
|
Verdict: PASS (advisory — Ran (this host, clone at the tip, Python 3.10.12 / pytest 9.2.0,
|
… the detached stop (#1082)
…ing (#1082) `_clamped_max_calls` returned `operator_bound=True` from the recorded request alone, so a hand-set or inherited `AGENTDESCENT_MAX_CALLS_REQUESTED=50` under a looser ceiling (5) with a caller cap of 2 printed `max_calls (AGENTDESCENT_MAX_CALLS): 2 call(s) made, cap 2` -- blaming a ceiling that never bound the run. The label now requires the applied value to *be* the ceiling. The detached child still names it, because its clamped argv is the ceiling; anywhere the applied `--max-calls` sits below the ceiling the detail stays bare. Enforcement is untouched: argv, the CallLimit value and the exit codes are the same. Red at `0e84778c`: the new row printed the ceiling name. Green here: 9 passed.
…g branch (#1082) Reviewer asked for the two branches the guard created: - a hand-set `AGENTDESCENT_MAX_CALLS_REQUESTED` below the ceiling (ceiling 2, caller 1) must stay bare -- red at `0e84778c` with `max_calls (AGENTDESCENT_MAX_CALLS): 1 call(s) made, cap 1`; - the detached run that names no caller cap: `launch` records the empty string and the child still attributes the stop to the ceiling, which is the one `_requested_calls` branch with no row.
|
Self-review (Birfy) Tip SHA: 9cf109d
|
…`max_calls` / `max_usd` (#1186) Fixes #1184. Claimed under AGENTS.md "Claim, PR, release"; agent-ready is triage-side and applied to 0/18 open issues (repo-wide), so it was not waited on. ## What changed The run-level call cap (#1166) and the operator ceiling (#1082) were both unreachable from the MCP surface: `plan` and `start` took `budget_usd` / `usd_per_call` and nothing else, so a host that wanted "at most N model calls for this run" had no lever except the process-wide `AGENTDESCENT_MAX_CALLS`, which it cannot lower per call. `plan` and `start` (both the tool bodies and the `maybe_tool` wrappers) now take `max_calls` and `max_usd`. They validate through `build_call_limit`, the same rule the CLI and the run store use, so a bad value, or a `max_usd` with no `usd_per_call`, refuses **before a run directory exists** and with the CLI's own sentence rather than a second wording for this surface. The operator's ceiling is applied with the same `min` as `runstore.launch` / `execute`: a caller may lower it, never raise it, and a caller that names no cap still gets it. The reply carries `call_limit`: - `{"max_calls": 3, "max_usd": null}` — nothing clamped it. - `{"max_calls": 2, "max_usd": null, "source": "AGENTDESCENT_MAX_CALLS", "max_calls_requested": 9}` — the ceiling bound a request of 9. - `{"max_calls": 2, "max_usd": null, "source": "AGENTDESCENT_MAX_CALLS"}` — the caller named no cap and the ceiling applies anyway. `source` and `max_calls_requested` appear only when the ceiling bound the run, the same distinction `AGENTDESCENT_MAX_CALLS_REQUESTED` already carries across the detach boundary: without them a host cannot tell "the ceiling lowered your 9 to 2" from "you asked for 2". A request of exactly the ceiling reports **not bound**, matching `_clamped_max_calls`. `start` passes the caller's own numbers to `runstore.launch`, not a pre-clamped value: `launch` applies the same `min` and is what records what was asked for in the child's environment, so handing it an already-clamped number would make `requested == ceiling` and erase the label from `stop_detail`, while the arithmetic stayed right. The enforced clamp still crosses the process boundary as `--max-calls 2`, which the test asserts on the child's argv. No argument means exactly today's behaviour, and plain `plan` has no `call_limit` key. `max_concurrency`, `doctor`, and any USD default for un-priced runs are untouched, as the issue scopes them out: a process-wide dollar ceiling would turn every un-priced run into a `SpecError` through `require_budget_pair`. ## Red test on `f5017f93` `tests/test_mcp_call_limit.py` alone at clean `f5017f93` → **8 failed, 2 passed**. The eight are the missing entry point, not a behaviour disagreement: - `test_plan_reports_the_cap_it_will_enforce` → `TypeError: Tools.plan() got an unexpected keyword argument 'max_calls'` - `test_start_caps_the_run_and_status_shows_it`, `test_max_usd_alone_is_refused_with_the_cli_sentence`, `test_a_caller_cap_above_the_operator_ceiling_is_clamped_and_labelled`, `test_a_cap_equal_to_the_ceiling_is_not_reported_as_bound`, `test_start_hands_the_clamped_cap_to_the_detached_child`, `test_nested_start_is_unchanged` → the same `TypeError` on `start` - `test_a_run_that_names_no_cap_still_gets_the_ceiling` → `KeyError: 'call_limit'`: the base payload has no such key, because only the new path reports the ceiling it will enforce The two that pass are the two "no cap, behaviour unchanged" rows (`test_plan_without_a_cap_is_unchanged`, `test_start_without_a_cap_is_unchanged`), which assert today's behaviour and must hold before and after. Provenance: re-measured at the base with only this branch's `tests/test_mcp_call_limit.py` copied into a clean worktree of `f5017f93` — `PYTHONPATH=$PWD pytest -o addopts='' tests/test_mcp_call_limit.py -q` → `8 failed, 2 passed in 2.46s`; all eight fail on the `TypeError` for the new `max_calls`/`max_usd` kwargs, and the two passers are the rows named above. The behaviour rows are asserted green **at this tip**: the run stops at the cap (`stop_reason="budget"`, the ceiling named in `stop_detail`), the child's argv carries `--max-calls 2` when the caller asked 9 under a ceiling of 2, a refused `max_usd` leaves no run behind, and nested `start` still returns its unchanged stub. Clamp teeth, measured: with the `min` bypassed in the helper and nothing else changed, the file went **2 failed, 8 passed** (`test_a_caller_cap_above_the_operator_ceiling_is_clamped_and_labelled`, `test_a_run_that_names_no_cap_still_gets_the_ceiling`). The probe was reverted, not shipped; the file greps clean and is 10 passed. Tip SHA: 4015bb8 Ran: - `pytest -o addopts='' -q` (the full offline suite) at `4015bb8f` → 21 failed, 10 errors, 7924 passed, 51 skipped in 2169.68 s. The 31 red ids are the **identical set** red on `14ba9529`, a commit that is five commits **older than this PR's base** `f5017f93` — the five being `38f9d90f` (release v0.11.0), `920dc8c3` (#1178), `98655111` (#1180), `a1b41e76` (#1181) and `f5017f93` (#1177), i.e. four code merges sit between the baseline and the base. That is the stronger direction: the same ids are red *before* those merges as well, so neither this branch nor the four merges in between accounts for them; the red ids are docker engine, exec-bit and wheel-smoke environment rows, and the comparison is a same-id `comm` (empty both ways), not a count. No red id touches `mcp`, `plan`, the run store caps or this new file. The one count difference from the earlier `c6000830` measurement is `test_dsh_superagent_run.py::[tmux]`, which is load-flaky (it passes in isolation at both commits) and passed in this run. - `pytest tests/test_mcp_call_limit.py` → 10 passed - focused set (`test_mcp_call_limit`, `test_mcp`, `test_mcp_meta`, `test_mcp_plan_probe`, `test_mcp_http_transport`, `test_operator_run_cap`, `test_budget_guard`) → 116 passed - `python -m tools.changelog_fragments check` → `5 changelog fragment(s) OK` - `python -m tools.gen_api_docs --check` → `docs/api.md is up to date` - `python -m tools.gen_algorithms_docs --check` → `docs/algorithms.md is up to date` - `mkdocs build --strict` → rc 0, 12.4 s, with the `social` og-image plugin stripped in a temporary untracked `mkdocs-local.yml` (it blocks on a font fetch in this sandbox; CI runs the full config) Not run: - real-model probe / USD cap against a live provider → `ARK_API_KEY` is not in this box's environment; every test row is offline and the cap is arithmetic, not a model call - wheel smoke → no packaging or release path change - `doctor`, `max_concurrency` → explicitly out of scope in the issue No UI surface: this adds two tool arguments, one JSON block and a docs row; there is no UI to screenshot.
Partial: #1082. This is one slice of the remainder, not its close.
Claimed under AGENTS.md "Claim, PR, release"; agent-ready is triage-side and applied to 0/18 open issues (repo-wide), so it was not waited on.
What changed
An operator per-run ceiling on model calls,
AGENTDESCENT_MAX_CALLS, clamped into both run paths so a host worker cannot ask for an unbounded run and get it.agentdescent/runstore.py:OPERATOR_MAX_CALLS_ENVandoperator_max_calls(), which reads and validates it (a non-integer or negative value is aRunStoreErrornaming the variable);_clamped_max_calls()is a min: a caller's--max-callslowers the ceiling, never raises it, and a run that passes nomax_callsstill gets the ceiling. Applied inlaunchandexecute, so the CLI, a detached run and MCPstart/resumeall charge through it; the detached child is started with the clamped number as--max-calls, so it cannot be widened across the process boundary.agentdescent/runstore.py, detached label: because the child's argv is already the clamped value,launchalso passes what the caller asked for inAGENTDESCENT_MAX_CALLS_REQUESTED(an internal, label-only variable — a malformed or hand-set value can change nothing but the wording ofstop_detail; enforcement stays the clamped argv, and a present-but-empty value means "the caller named no cap").agentdescent/budget.py:CallLimit.sourceandBudgetStop(source=…). When the operator's ceiling is what bound the run,stop_detailreadsmax_calls (AGENTDESCENT_MAX_CALLS): N call(s) made, cap Ninstead of the baremax_calls— on the foreground and detached paths alike. A dollar stop never carries that source: a USD cap has no operator ceiling behind it (an un-priced one is refused before the run starts), so a run with both a ceiling and--max-usdnamesmax_usd, not the call knob.tests/test_operator_run_cap.py: eleven rows — the ceiling lowers a larger caller cap; it applies with no caller cap at all; a loose ceiling never loosens a caller's smaller cap (detail stays bare); a detachedlaunchrecords--max-calls 2when the caller asked 50; that child learns what the caller asked (AGENTDESCENT_MAX_CALLS_REQUESTED=50); the child's own view (requested 50, ceiling 2, argv 2) names the ceiling; equal caller/ceiling stays bare; a USD stop never names the call ceiling; a hand-setREQUESTED=50does not blame a looser ceiling when the caller's own cap of 2 bound the run; the same from below (ceiling 2, caller 1); a detached run with no caller cap still names the ceiling.docs/plugin-design.md: one bullet in the host-worker guards, next to the recursion guard, stating the foreground and detached behaviour.changelog.d/1082-operator-run-cap.md(oneAddedentry).Why
Partial:and notFixes:AGENTDESCENT_MAX_USDwould turn every un-priced run into aSpecError, becauserequire_budget_pair(runstore.py:90-98) refuses a USD cap with nousd_per_call. That is a contract decision, not a clamp; it stays open on#1082.#1087(f5e9333) —_operator_write_guardatcli.py:3418, used at:3094(materialised tree),:3800(cmd_apply) and:3915; child-env scrubbing is#1088(23979b7) —tests/test_host_env_scrub.py, 13 tests including one parametrised overSESSION_MARKERSitself.max_concurrency, the MCP tool arguments for the caps, anddoctorreporting them.Red first, at
origin/main14ba9529with onlytests/test_operator_run_cap.py—3 failed, 1 passedFAILED …::test_the_operator_ceiling_lowers_a_callers_larger_cap— the ceiling was ignored (assert 4 == 2on the call count)FAILED …::test_the_operator_ceiling_applies_with_no_caller_cap— the uncapped run made its full round's callsFAILED …::test_a_detached_run_carries_the_operator_ceiling— the recorded child argv carried--max-calls 50…::test_a_loose_operator_ceiling_never_loosens_the_callers_cap: a loose ceiling must change nothing, and it already did not.Red first for the review fixes, at
ac82e72—3 failed, 5 passedFAILED …::test_a_dollar_stop_never_names_the_call_ceiling—max_usd (AGENTDESCENT_MAX_CALLS): $0.0000 spent, cap $0.0100FAILED …::test_the_detached_child_learns_what_the_caller_asked— child env had noAGENTDESCENT_MAX_CALLS_REQUESTEDFAILED …::test_the_detached_run_names_the_ceiling_the_parent_applied— the child's detail was the baremax_calls…::test_an_equal_cap_does_not_name_the_ceiling: an equal caller cap was already not attributed to the ceiling.Red first for the label guard, at
0e84778c—2 failed, 9 passedFAILED …::test_a_hand_set_request_does_not_blame_a_looser_ceiling—max_calls (AGENTDESCENT_MAX_CALLS): 2 call(s) made, cap 2on a run whose caller cap of 2 sat under a ceiling of 5: the label followed a stale inheritedAGENTDESCENT_MAX_CALLS_REQUESTED=50instead of the clamp, sooperator_boundnow also requires the applied value to be the ceiling. Enforcement unchanged — that row still stops at 2 calls.FAILED …::test_a_hand_set_request_below_the_ceiling_stays_bare— same class from the other side (ceiling 2, caller 1, stale request 50): the detail saidmax_calls (AGENTDESCENT_MAX_CALLS): 1 call(s) made, cap 1.…::test_a_detached_run_with_no_caller_cap_still_names_the_ceilingcovers the one_requested_callsbranch that had no row:launchrecords an emptyAGENTDESCENT_MAX_CALLS_REQUESTEDand the child still attributes the stop to the ceiling. Green at both tips — coverage, not a regression.Verification at the tip
Tip SHA: 9cf109d
Ran:
pytest -q tests/test_operator_run_cap.py→11 passed(the last three rows are the label-follows-the-clamp cases below)pytest -qover the budget/runstore/surface set (test_budget_guard,test_call_budget,test_budget,test_budgeted_loop,test_budgeted_search,test_budget_round_warning,test_runstore_cli,test_runstore_tilde,test_cli_surface_error_envelope,test_cli_surface_p1, plus the new file) →146 passedpytest -q tests/test_mcp*.py→137 passed; combined with the set above in one invocation →283 passedmkdocs build --strict→ rc 0 (socialplugin excluded on this host: it needs cairo and hangs; the config was otherwise untouched, and the temp config is not committed)python -m tools.gen_api_docs --check→docs/api.md is up to datepython -m tools.changelog_fragments check→86 changelog fragment(s) OKFull-suite note: the suite at this branch's first tip ran as eight parallel chunks —
7884 passed,21 failed,10 errors, all docker/sandbox or environment rows, plus one timing test (test_async_evolve.py::test_shutdown_grace_is_respected_and_warns) that is flaky under load; the same node ids fail at pristine14ba9529, and the one that differs passed on re-run. CI remains the line for the whole suite.Not run:
mkdocs build --strictwith thesocialplugin (see above)No UI surface: an environment variable and a stop detail only.