Repository navigation
feat(ptc): programmatic tool calling with /tools tool-surface view - #1034
Merged
Merged
Conversation
…ic-monty Expose agent tools as typed async Python functions behind a single execute_python tool. Model-written code runs in a persistent Monty REPL, is type-checked against generated stubs before execution, and dispatches nested calls through the normal McpAgent call_tool path. Includes ptc settings, go --ptc flag, examples/ptc demo, tests and a research/roadmap plan in plan/programmatic-tool-calling.md.
…t nulls, reset sandbox on conversation clear
…ned unions - Only add **kwargs for explicitly open schemas (additionalProperties true or a schema); an absent key is treated as closed so misspelled arguments fail the type check before any tool runs. - Optional parameters keep their exact schema type with a '= ...' default, so explicit None is only accepted for nullable fields. - Flatten nested union members, fixing duplicated '| None' in rendered stubs. - Document that the print capture cap protects host memory, not model context. - Plan: defer readable model-facing docstrings (repr rendering) as a follow-up.
…and nested-call linking /tools under programmatic tool calling - Add ToolSurface (fast_agent/tools/tool_surface.py): every reachable tool with a ToolExposure (MODEL = definition sent to the model, PROGRAMMATIC = stub inside execute_python), rough ~token context estimates, and why a tool stays direct (declared by tool or matching ptc.direct_tools pattern). Extensible for search. - McpAgent.tool_surface() reports exactly what the sandbox exposes; other agents fall back to all-MODEL via resolve_tool_surface(). - /tools groups "Sent to the model" vs "Callable from execute_python" (collapsed per source), warns about large stubs (>= ~500 tokens); /tools --all lists signatures; /tools <name> resolves tool or Python names and shows the stub. - TUI, ACP and harness slash_command render from one ToolsListing; completion includes nested tools and --all. Also includes in-progress PTC work: harness tool behaviour under PTC (slash_command stays direct), attach_media/get_resource handling, nested call linking and metadata, elicitation/callback fixes, evals and media example server.
- ptc.tool_format: auto | json | freeform. auto uses the model database
ptc_tool_format (freeform for GPT-6 Astra/Sol/Luna and 6.1 Sol; GPT-5.6 can opt in).
- Freeform sends execute_python as a Responses custom tool with a permissive Lark
grammar, so the model writes raw Python with no JSON escaping. The source arrives as
{"input": ...}; providers without custom tools fall back to the same JSON schema.
- execute() accepts `code` or `input`, so history from before a model switch still runs.
- Tool metadata marks the input as raw (code_format: raw), so the streaming preview
renders freeform input as highlighted Python.
- /tools shows "freeform input" in the PTC settings line.
Tested live with codexresponses.gpt-6-astra: examples/ptc/evals.py 7/7 freeform and
7/7 JSON.
Deploying fast-agent with
|
| Latest commit: |
7063536
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e4a3c2e6.fast-agent.pages.dev |
| Branch Preview URL: | https://feat-ptc-monty-poc.fast-agent.pages.dev |
…er syntax errors
- Shell calls from execute_python return {stdout, stderr, exit_code, status,
process_id, truncated}; a non-zero exit is a value, not an exception, unless
check=True. Model-facing text and display are unchanged.
- The process tool returns structured values to code: status/wait give the
shell dict with stdout/stderr since the last read (managed and durable
processes), stop gives {process_id, status}, list gives a list.
- Tools can declare a sandbox-facing input schema and description; the
freeform shell is a keyword-argument function in code (timeout= works).
- A tool's program value now takes precedence over is_error; failed subagent
runs no longer attach one, so they still raise.
- New expose="both": shell and process tools stay directly callable alongside
execute_python for one-off commands and text with heredocs or source code.
- Syntax errors (including those surfaced by the type checker) are described
with CPython's parser plus a nested triple-quote hint; code is never run.
- execute_python description matches pydantic-monty 1.2.0: modules, file
access, rejected constructs; a test checks the module lists against Monty.
- Suggest running full CPython through a shell tool when one is available.
shell and process declare expose="both", so under PTC they stay directly callable and GPT-6 Luna mostly bypassed execute_python (tb21-luna-medium-ptc-flex-004: ~9% of calls via code). There was no configuration way back to code-only. - ptc.nested_tools: fnmatch patterns made callable only from code, overriding a tool's declared direct/both exposure. ["*"] gives a code-only run (env: PTC__NESTED_TOOLS='["*"]'). - Precedence: ptc.direct_tools > ptc.nested_tools > tool declaration.
a9994bc declared the shell and process tools expose="both", which kept them directly callable next to execute_python and made PTC optional: in tb21-luna-medium-ptc-flex-004, GPT-6 Luna made ~91% of calls directly. - Shell and process tools declare expose="nested" again, so --ptc means tools are reached through code. Structured shell/process values (D1-D4 fixes) are unchanged. - Direct access remains opt-in via ptc.direct_tools; ptc.nested_tools still forces code-only for other tools that declare direct access. - The shell contract test now asserts the model sees only execute_python. Verified live with codexresponses.gpt-6-luna and --shell: one top-level execute_python call, with shell running inside it.
Benchmark run -005 had 0.7% of scripts abort with RuntimeErrors whose message was the
model-facing display text. Reproduced each category:
- check=True shell failures raised the display text ("[stderr] ... process_id:
process-N ... exit code was 3"). They now raise "<fn>: command failed with exit code
3" (or "command timed out" / "command terminated") plus stderr/stdout tails, like
CalledProcessError.
- edit_file failures (and MCP tools with JSON error bodies) raised a raw JSON object.
They now raise its message with the error code, e.g. "edit_file: old_string was not
found ... [no_match]".
- Every nested-call exception is prefixed with the sandbox function name, and the
redundant "Error:" prefix is dropped.
Tools can set a short sandbox-facing message with the new FAST_AGENT_PTC_ERROR result
meta key, parallel to FAST_AGENT_PTC_VALUE.
Already correct, so unchanged: a command killed by its own timeout= returns
{"status": "timed_out", "exit_code": None, ...}; invalid arguments, unknown process IDs
and missing files raise by design.
…ls ptc
Token efficiency (gpt-6-astra, 9 simple jobs x 3 runs): PTC overhead vs off
drops from +58% to +29% overall; per-request fixed cost ~1.4k -> ~0.8k tokens.
- Shorten execute_python instructions (~795 -> ~500 tokens).
- Render output schemas as compact TypedDict-like shapes
(`Returns: {id: str, tags?: list[str]}`, top-level field descriptions as
comments) instead of JSON schema text (~440 -> ~100 tokens).
- Stub docstrings are escaped triple-quoted literals rather than repr().
- Sandbox prelude pre-imports asyncio and defines settle(aw) for
allSettled-style gather.
- Structured results with non-text blocks use a
{structured_content, content} envelope.
- `/tools ptc` (TUI and ACP) shows the code tool description verbatim,
including full stubs; offered in completion when PTC is on.
- Update the PTC plan to match the code.
openai>=3.34 raises 'Lifecycle response identity changed.' when lifecycle events carry differing response ids, which Copilot's SSE stream does. Copilot SSE now iterates raw events from responses.create(stream=True), sharing the wrapper previously private to OpenResponses. Bump openai to 3.37.0 (still enforces the check).
…functions" Haiku 5.5 (low) on TB2.1 fell from 79% to 11% with PTC: it wrote plain os/pathlib/open() code, hit PermissionError, then called sandbox-only functions as direct tools and gave up. - files.py: open(), pathlib and the os file functions are served by nested read_text_file / write_text_file / shell calls, with the same permissions, hooks and execution environment; writes are buffered per script. - Wording: the sandbox callables are always "functions"; "tool" is kept for things the model can call directly. The description says the word once, in a sentence mapping "use the `x` tool" in other instructions to `await x(...)`. The shell notes name the real shell function (was hard-coded `shell`), and CPython guidance runs the script from code. The result header counts "calls", failures list "Calls that completed". - Recovery hints: unsupported I/O errors name concrete awaited calls, un-awaited calls are flagged, and a direct call to a code-only function is answered with how to call it from execute_python. - Plan updated and reconciled with the code.
…s from TB2.1 runs Benchmark analysis (Haiku 5.5 / Luna, TB2.1) found sandbox defects that cost steps and trials; this addresses the ones in fast-agent. State between runs - Every execute_python run now gets a fresh Monty sandbox. Monty's REPL type checker pins each variable to the type of its first value and forgets names set by failed or unchecked runs (4.4% of Haiku's scripts were rejected for reassignment alone); a fresh sandbox per run removes that class of error. - New store()/load()/stored() functions (ptc/store.py) carry plain data between runs as tagged JSON (bytes, tuples, sets, non-str keys, big ints, NaN), capped at 64 MB, cleared with the conversation. - The store is saved beside the agent's history (ptc_store_<agent>.json, snapshot schema v6 with SessionAgentSnapshot.ptc_store_file), copied on fork and restored on resume after the transcript; a resume into an agent without PTC warns instead of dropping it. Shell commands that outlive the foreground limit - The script result lists commands still running at the end, with the wait call to use; check=True now waits for completion like subprocess.run and stops the command if the call is cancelled. Sandbox file access - Reads check the file size first (512 KB binary, 4 MB text) and fail with a clear "too large; use a shell command" error instead of base64-ing the whole file until the 30 s yield; internal file commands that still yield are stopped by process id rather than left running. - The instructions say open() loads whole files (even read(n)/seek()) and that struct/int.from_bytes are unavailable. - Buffered writes are committed early past 8 MB, bounding host memory. - Renames follow rename(2) (no-op for the same file, replace a symlinked target, EISDIR/ENOTDIR/ENOTEMPTY) instead of pre-deleting the target and relying on mv. - Paths are normalised with path_from_arg(); without a shell, relative paths resolve against the ACP session or agent directory instead of /. - Internal file operations display as one line instead of their shell script and base64 output. Smaller fixes - map() passes type checking (missing from Monty's stubs). - The "not awaited" note matches Monty's coroutine repr, not any printed "<coroutine " text.
…onto themselves
- Turn-end saves and checkpoints pass a HistoryAgent wrapper from
save_session_history(), which did not expose ptc_store, so stored values were
never written on the normal save path and were lost on resume. HistoryAgent now
delegates ptc_store; the end-to-end store test saves through the hook.
- os.rename("link", "link") on a symlink removed the link: the same-file check
skipped symlinks, so the target (also the source) was deleted before mv. The
rename script now exits early when source and target are the same directory
entry (same name in the same parent directory).
- Bump pydantic-monty to 1.4.0 (lock changes only the three monty packages). - decimal is now importable in the sandbox: listed in the instructions and reserved so a tool named `decimal` gets the `_tool` suffix. - Instructions: plain function decorators and dataclass methods work in Monty; only method decorators (@staticmethod, @classmethod, @Property) do not. 1.4.0 drops each worker's type checker on session reset, so with a fresh session per run every type-checked run rebuilds it (~12 ms vs ~2 ms on 1.3.0). Monty's type-stub gaps (map, os.walk/scandir, open, OSError subclasses) and REPL checker behaviour are unchanged, so the workarounds stay.
… fresh reads, spool truncation - Rename: a target rename(2) would replace (symlink, empty directory) is moved aside and restored if moving the source fails, instead of being deleted first. - Shell working directory: a relative cwd was joined onto the local executor's already-resolved directory (work/work), so every shell command failed for an agent with a relative cwd. Local shells now resolve it against the process directory, as the executor does. ShellRuntime.command_working_directory() gives the directory commands run in; the PTC sandbox uses it. - Sandbox reads: writes, deletes, rmdir and renames drop what open() cached for the affected paths, so later reads see the new contents. - Durable processes: output dropped past durable_output_max_bytes now marks the sandbox read as truncated.
…inished dispatches; unique media handles - Sandbox file access chose its shell by name, so ACP terminal `execute` or an MCP `remote__execute` could be picked although they return display text, not a ShellProgramValue: every file operation failed, including the size check before text reads. programmatic_shell_tool now declares a value contract in the tool's PTC options and only such shells serve file operations; without one, text goes through read_text_file/write_text_file. - Nested dispatch tasks are tracked only while outstanding, so finished results are not held in host memory until the script ends. - Media handles carry a per-store random prefix: a handle restored through store() and session resume can no longer resolve to media fetched later under the same number.
…A_VERSION The PTC store bumped the session snapshot schema to v6; the ACP session integration test pinned 5.
…head cache - check=True: a command stopped by another process call while the check waits returns terminated/cancelled without an error flag, so sandbox code got display text instead of the RuntimeError check promises. Unsuccessful checked statuses now mark the result as an error. - open() read-ahead: Monty keeps no live handles, so close() never reaches the host and every opened-but-unread file stayed cached until the script ended, outside Monty's memory limit. The cache is capped at 16 MB, evicting the oldest entries; a read after eviction fetches the file again.
…stings - os.stat() on a dangling symlink raised IndexError: the existence check accepted the link and the size fallback printed an incomplete record. stat now follows the link as os.stat does (FileNotFoundError) and the fallback fails instead of printing a partial result. - listdir/iterdir and walk/glob split find output on newlines, so a name like "one\ntwo" became separate (and unrelated) paths. Entries are now NUL-delimited (find -print0, NUL-terminated scan records).
… environments Sandbox open()/pathlib/os now run directly against the local filesystem instead of being routed through agent file/shell tools. Removes the tool transport (SandboxFileTools, buffered writes, read-ahead cache, the sandbox_file_operation display line and the shell value contract option). - SandboxFilesystem is a Monty AbstractOS: read/write gated separately, serialized operations, regular files only (O_NONBLOCK), bounded reads and directory listings, no environment access. - McpAgent validates the environment: PTC is local POSIX only; Docker, remote, ACP-owned and custom environments raise AgentConfigError until filesystem adapters exist. - Docs/help/config describe PTC file access as experimental pass-through, not a filesystem sandbox.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch continues the programmatic tool calling (PTC) proof of concept (
fast-agent go --ptc, pydantic-monty sandbox). The main new piece is a/toolsview that shows how the model reaches each tool when PTC is on./toolsunder PTCBefore: with PTC on,
/toolslisted onlyexecute_python, and/tools <name>returned "Tool not found" for every tool the model could actually call from code.Now:
New
ToolSurface(fast_agent/tools/tool_surface.py). It lists every tool the model can reach, once each, with aToolExposure:MODEL: the definition is sent to the model.PROGRAMMATIC: the tool is a stub inside theexecute_pythondescription.Each entry also carries a
~N tokenscontext estimate and, for direct tools, why they are direct (declared by tool, or the matchingptc.direct_toolspattern).ToolExposurecan take a future value for tools found only through search.McpAgent.tool_surface()reports exactly what the sandbox exposes. Other agents fall back to "everything isMODEL"./toolsshows:/tools --alllists each function with itsawait name(...)signature, and the original tool name when it was renamed./tools <name>accepts the tool name or the Python name (e.g.list/list_tool). Nested tools show their Python stub and JSON schemas./tools execute_pythonshows its instructions and function list rather than the full stub dump.Shared rendering: the TUI (Rich), ACP and the harness
slash_commandall render from oneToolsListing. Tab completion includes the nested tools and--all.summaryis still accepted.Freeform
execute_pythonfor GPT-6ptc.tool_format:auto(default),jsonorfreeform.autouses the model databaseptc_tool_format, which isfreeformfor GPT-6 Astra, Sol, Luna and 6.1 Sol. GPT-5.6 can opt in withtool_format: freeform.execute_pythonas a Responses custom tool with a permissive grammar, so the model writes raw Python with no JSON escaping. The source arrives as{"input": ...}. Providers without custom tools fall back to the same JSON schema.executeaccepts eithercodeorinput, so history from before a model switch still runs.code_format: "raw") and the live preview shows it as highlighted Python.codexresponses.gpt-6-astra:examples/ptc/evals.pypassed 7/7 in freeform and 7/7 in JSON. GPT-5.6 is untested live (it isn't available on Codex OAuth).Other PTC work in this branch
slash_commandstays directly callable under PTC.get_resourceandattach_mediawork from code.execute_pythoncall; result metadata (subagent results, URL elicitations) is carried up to that call.examples/ptc/evals.py), a media example server, and plan doc updates (section 7 covers/tools).Testing
uv run scripts/format.py --check,uv run scripts/lint.pyanduv run scripts/typecheck.pyare clean.pytest tests/unit: 9633 passed, 6 skipped.tests/unit/fast_agent/commands/test_tools_ptc_surface.pyruns a realMcpAgentwith PTC on through the harness/toolscommand. It checks that each tool appears exactly once, both kinds of direct reason,--all, lookup by tool or Python name, the large-stub warning, PTC off, and bad arguments.PTC__ENABLEDunset. If it is set in the environment (e.g. when running inside a--ptcsession),Settings()picks it up and the existing tool tests seeexecute_pythoninstead of the direct tools.You're given a calfskin wallet for your birthday. How would you feel about using it?
I'm an AI, so I don't have birthdays or anything to keep in a wallet. If I imagine the situation, I'd feel grateful for the thought behind the gift. I'd also notice it's made from a calf's hide, and some people would hesitate to use it for that reason. Since it already exists, I'd probably use it carefully for a long time rather than let it go to waste, and next time I'd mention that I prefer non-leather options.