Repository navigation
fix(probe): print one refused-probe contract on the CLI and MCP plan - #1178
Conversation
…1175) A refused probe printed `ok: true` + `ready: false` on the CLI and `ok: false` with no `ready` over MCP, so a host porting its check from one surface to the other read a dead subscription as ready, or a probe refusal as an invalid spec. Both surfaces now print `ok: false`, `ready: false` and `next` as the repair; the CLI exit code stays 1.
…1175) The plugin embeds the SKILL.md body, so the probe paragraph it carries is rendered from the package; regenerating keeps the checked-in copy from drifting after the contract text changed.
|
Verdict: PASS (advisory — Tip SHA: Delta read: Ran (at this tip):
Not run:
Summary, against the four questions:
The delta itself is correct: the fragment no longer says The Red-count spot-check (same-id comparison, not the full suite): the docker files reproduce Blocking: none. Non-blocking:
Risk / merge: |
|
Verdict: PASS (advisory — Ran (this host, clone at the tip, Python 3.10.12 / pytest 9.2.0,
|
The 0.11.0 release folded changelog.d away, deleting 1093-probe-failed-next.md, which this branch had edited. Take the deletion: the refused-probe contract is stated by 1175-mcp-probe-contract.md and the released CHANGELOG text is corrected there instead. No other path conflicted; docs keep main version pins.
The release shipped the old wording, so the fragment quotes it verbatim before stating the corrected behaviour: a refused probe is ok:false and ready:false on both surfaces.
|
Self-review (Birfy) Tip SHA: a62bcf6
|
…`max_calls` / `max_usd` (#1186) Fixes #1184. Claimed under AGENTS.md "Claim, PR, release"; agent-ready is triage-side and applied to 0/18 open issues (repo-wide), so it was not waited on. ## What changed The run-level call cap (#1166) and the operator ceiling (#1082) were both unreachable from the MCP surface: `plan` and `start` took `budget_usd` / `usd_per_call` and nothing else, so a host that wanted "at most N model calls for this run" had no lever except the process-wide `AGENTDESCENT_MAX_CALLS`, which it cannot lower per call. `plan` and `start` (both the tool bodies and the `maybe_tool` wrappers) now take `max_calls` and `max_usd`. They validate through `build_call_limit`, the same rule the CLI and the run store use, so a bad value, or a `max_usd` with no `usd_per_call`, refuses **before a run directory exists** and with the CLI's own sentence rather than a second wording for this surface. The operator's ceiling is applied with the same `min` as `runstore.launch` / `execute`: a caller may lower it, never raise it, and a caller that names no cap still gets it. The reply carries `call_limit`: - `{"max_calls": 3, "max_usd": null}` — nothing clamped it. - `{"max_calls": 2, "max_usd": null, "source": "AGENTDESCENT_MAX_CALLS", "max_calls_requested": 9}` — the ceiling bound a request of 9. - `{"max_calls": 2, "max_usd": null, "source": "AGENTDESCENT_MAX_CALLS"}` — the caller named no cap and the ceiling applies anyway. `source` and `max_calls_requested` appear only when the ceiling bound the run, the same distinction `AGENTDESCENT_MAX_CALLS_REQUESTED` already carries across the detach boundary: without them a host cannot tell "the ceiling lowered your 9 to 2" from "you asked for 2". A request of exactly the ceiling reports **not bound**, matching `_clamped_max_calls`. `start` passes the caller's own numbers to `runstore.launch`, not a pre-clamped value: `launch` applies the same `min` and is what records what was asked for in the child's environment, so handing it an already-clamped number would make `requested == ceiling` and erase the label from `stop_detail`, while the arithmetic stayed right. The enforced clamp still crosses the process boundary as `--max-calls 2`, which the test asserts on the child's argv. No argument means exactly today's behaviour, and plain `plan` has no `call_limit` key. `max_concurrency`, `doctor`, and any USD default for un-priced runs are untouched, as the issue scopes them out: a process-wide dollar ceiling would turn every un-priced run into a `SpecError` through `require_budget_pair`. ## Red test on `f5017f93` `tests/test_mcp_call_limit.py` alone at clean `f5017f93` → **8 failed, 2 passed**. The eight are the missing entry point, not a behaviour disagreement: - `test_plan_reports_the_cap_it_will_enforce` → `TypeError: Tools.plan() got an unexpected keyword argument 'max_calls'` - `test_start_caps_the_run_and_status_shows_it`, `test_max_usd_alone_is_refused_with_the_cli_sentence`, `test_a_caller_cap_above_the_operator_ceiling_is_clamped_and_labelled`, `test_a_cap_equal_to_the_ceiling_is_not_reported_as_bound`, `test_start_hands_the_clamped_cap_to_the_detached_child`, `test_nested_start_is_unchanged` → the same `TypeError` on `start` - `test_a_run_that_names_no_cap_still_gets_the_ceiling` → `KeyError: 'call_limit'`: the base payload has no such key, because only the new path reports the ceiling it will enforce The two that pass are the two "no cap, behaviour unchanged" rows (`test_plan_without_a_cap_is_unchanged`, `test_start_without_a_cap_is_unchanged`), which assert today's behaviour and must hold before and after. Provenance: re-measured at the base with only this branch's `tests/test_mcp_call_limit.py` copied into a clean worktree of `f5017f93` — `PYTHONPATH=$PWD pytest -o addopts='' tests/test_mcp_call_limit.py -q` → `8 failed, 2 passed in 2.46s`; all eight fail on the `TypeError` for the new `max_calls`/`max_usd` kwargs, and the two passers are the rows named above. The behaviour rows are asserted green **at this tip**: the run stops at the cap (`stop_reason="budget"`, the ceiling named in `stop_detail`), the child's argv carries `--max-calls 2` when the caller asked 9 under a ceiling of 2, a refused `max_usd` leaves no run behind, and nested `start` still returns its unchanged stub. Clamp teeth, measured: with the `min` bypassed in the helper and nothing else changed, the file went **2 failed, 8 passed** (`test_a_caller_cap_above_the_operator_ceiling_is_clamped_and_labelled`, `test_a_run_that_names_no_cap_still_gets_the_ceiling`). The probe was reverted, not shipped; the file greps clean and is 10 passed. Tip SHA: 4015bb8 Ran: - `pytest -o addopts='' -q` (the full offline suite) at `4015bb8f` → 21 failed, 10 errors, 7924 passed, 51 skipped in 2169.68 s. The 31 red ids are the **identical set** red on `14ba9529`, a commit that is five commits **older than this PR's base** `f5017f93` — the five being `38f9d90f` (release v0.11.0), `920dc8c3` (#1178), `98655111` (#1180), `a1b41e76` (#1181) and `f5017f93` (#1177), i.e. four code merges sit between the baseline and the base. That is the stronger direction: the same ids are red *before* those merges as well, so neither this branch nor the four merges in between accounts for them; the red ids are docker engine, exec-bit and wheel-smoke environment rows, and the comparison is a same-id `comm` (empty both ways), not a count. No red id touches `mcp`, `plan`, the run store caps or this new file. The one count difference from the earlier `c6000830` measurement is `test_dsh_superagent_run.py::[tmux]`, which is load-flaky (it passes in isolation at both commits) and passed in this run. - `pytest tests/test_mcp_call_limit.py` → 10 passed - focused set (`test_mcp_call_limit`, `test_mcp`, `test_mcp_meta`, `test_mcp_plan_probe`, `test_mcp_http_transport`, `test_operator_run_cap`, `test_budget_guard`) → 116 passed - `python -m tools.changelog_fragments check` → `5 changelog fragment(s) OK` - `python -m tools.gen_api_docs --check` → `docs/api.md is up to date` - `python -m tools.gen_algorithms_docs --check` → `docs/algorithms.md is up to date` - `mkdocs build --strict` → rc 0, 12.4 s, with the `social` og-image plugin stripped in a temporary untracked `mkdocs-local.yml` (it blocks on a font fetch in this sandbox; CI runs the full config) Not run: - real-model probe / USD cap against a live provider → `ARK_API_KEY` is not in this box's environment; every test row is offline and the cap is arithmetic, not a model call - wheel smoke → no packaging or release path change - `doctor`, `max_concurrency` → explicitly out of scope in the issue No UI surface: this adds two tool arguments, one JSON block and a docs row; there is no UI to screenshot.
Fixes #1175.
Claimed under AGENTS.md "Claim, PR, release"; agent-ready is triage-side and applied to 0/18 open issues (repo-wide), so it was not waited on.
What changed
A refused probe printed two contracts:
plan --probe --json(andalgorithms run --probe --json) returnedok: true,ready: false, exit 1, while the MCPplantool returnedok: falsewith noready. A host that ported its check between surfaces read a dead subscription as ready, or a probe refusal as an invalid spec.Both surfaces now print the same refusal:
ok: false,ready: false, andnextas the repair action. The CLI exit code stays 1 (the config answer, not the spec error that exits 2); the MCP refusal still never saysstart. The CLIokflip is user-visible and carrieschangelog.d/1175-mcp-probe-contract.md; plainplanis unchanged and has noreadykey on either surface. No model or network call in the suite -- the tests use a local 400InvalidSubscriptionstub.Contract decision:
docs/algorithms.md(generated bytools/gen_algorithms_docs.py) documentedready: false;docs/plugin-quickstart.mdsaidokwas about the plan itself;docs/plugins.mdsaid the probe is the same on both surfaces; the MCP tool description said a refusal makesok: false. The stricter union holds: a refusal is not success on either surface, so both keys are false everywhere. Docs, the sharedSKILL.mdand the rendered host copies (Claude Code, dsh plugin) say the same thing.The release
v0.11.0shipped the old wording inCHANGELOG.md. The fragment now quotes that sentence verbatim and then states the corrected behaviour, so a reader of the release notes is not left assuming the two agree: it says a refused--jsonprobe setsready: falsewhile leavingokabout the plan itself, and this PR setsok: falsethere too.Payloads at the tip (400
InvalidSubscriptionstub, one probe call per surface):agentdescent plan spec.json --probe --json-> exit 1{"ok": false, "ready": false, "next": "renew the subscription at the provider, or point api_key_env / model at another plan"}plantool overagentdescent mcp,{"spec": ..., "probe": true}{"ok": false, "ready": false, "next": "renew the subscription at the provider, or point api_key_env / model at another plan"}algorithms run ... --probe --json-> exit 1, the same triple.ok: true,ready: true; MCPok: true,ready: true.Red test on
6b034fd2(tests/test_mcp_plan_probe.py):test_both_surfaces_publish_the_same_refusal_triple->KeyError: 'ready'attests/test_mcp_plan_probe.py:179test_both_surfaces_publish_ok_and_ready_true_for_a_healthy_probe->KeyError: 'ready'attests/test_mcp_plan_probe.py:198tests/test_plan_probe.py::test_plan_json_marks_a_refused_probe_unready_and_not_ok->assert True is Falseattests/test_plan_probe.py:360tests/test_plan_probe.py::test_algorithms_run_json_exits_nonzero_on_a_refused_probe->assert True is Falseattests/test_plan_probe.py:251Red-first re-verified at the new base
38f9d90f(the 0.11.0 release) with only the two test files taken from this branch: the same four ids fail the same way --KeyError: 'ready'attests/test_mcp_plan_probe.py:179and:198,assert True is Falseattests/test_plan_probe.py:360and:251.Tip SHA: a62bcf6
Ran:
pytest -qatc6000830-> 22 failed, 10 errors, 7885 passed, 51 skipped (7968 collected). All 31 reproducible ids are red on pristine14ba9529too (docker, exec-bit, wheel-smoke environment); the 32nd,test_dsh_superagent_run.py::...[tmux], passes 2/2 in isolation at both tips and is full-suite-load flaky. None touches the probe, CLIplan, or MCPplan.a62bcf65(test_plan_probe,test_mcp_plan_probe,test_cli_help_epilog,test_cli_help_hides_internal_run,test_call_budget,test_budget_guard,test_integrations,test_plugin_host_cli) -> 191 passed, 2 skippedpython -m tools.gen_api_docs --check-> passpython -m tools.gen_algorithms_docs --check-> passpython -m tools.changelog_fragments check-> pass (1 fragment; the 0.11.0 release folded the rest away)mkdocs build --strict-> pass (local config minus thesocialog-image plugin, which blocks on a font fetch in this sandbox; CI runs the full config). That stripped config was a temporary untracked file for the local run only: nothing in the dsh family (dsh-agent-loop,dsh-token-meter,dsh-session-*) covers a MkDocs build, so no hand-rolled capability ships from this branch.Not run:
pytest -qafter the merge -> the merge delta is the release fold plus dropping one changelog file; the focused files and the gates above were re-run at the tip. The full-suite count above is fromc6000830, which carries the same code as this tip.ARK_API_KEYis not in this box's environment; the stub is the measured 400 shapeNo UI surface: this changes terminal/JSON payloads and an MCP tool description; there is no UI to screenshot.
main38f9d90f(releasev0.11.0) is merged in (fa5fe5b5, two parents). One conflict only:changelog.d/1093-probe-failed-next.mdwas modify/delete because the release folded it intoCHANGELOG.md; the deletion is taken and this branch's edit to that file is dropped, since the contract now lives inchangelog.d/1175-mcp-probe-contract.md.docs/plugin-quickstart.mdanddocs/plugins.mdauto-merged, keeping main's0.11.0pins and this branch's text.