Repository navigation
Rework the browser tools after computer-use style browser tools - #217
Merged
Merged
Conversation
The page verbs move from BrowserSession to a new Tab, which also owns its answered dialogs. A BrowserSession now holds the launched browser and its tabs: it creates, selects and closes them, and sync_tabs adopts tabs the page opened itself (target=_blank, window.open) without making them active. The session's timeouts are shared with every tab. Session-level verbs that act on the active tab stay for now, for the tools that predate tabs. Headless browsers get a 1280×800 viewport at device scale 1 instead of chromiumoxide's 800×600, so pages lay out for a desktop and screenshot pixels are CSS pixels. A headful login window fills its real window.
Tab::read_page renders CDP's full accessibility tree as YAML-style lines, `- role "name" [ref_N] attrs`, with state such as value, checked, expanded, disabled and focused, and the href of links. Nameless wrappers and text that repeats its parent's label are left out. It can be limited to actionable elements (a flat list), to the subtree of one ref, or to a depth. Tab::find returns the lines that match a query. Every printed node with a DOM node gets a ref_N mapped to its backend node id, stable for as long as the document lives. ref_point scrolls an element into view and returns the center of its box, where a click on it lands. The tree is read through a raw command with lenient structs, since chromiumoxide's typed bindings fail on property names newer than its bundled protocol.
Tab gets the input verbs of a computer-use style tool: - click_point with button, click count (double/triple) and modifiers, hover_point, drag, and wheel, which scrolls whatever container is under the pointer; - press_keys for space-separated keys and chords with a repeat count, accepting common aliases and any case (Return, Esc, pagedown, f5); - type_into_focused for the focused element. screenshot_frame captures the viewport at a scale (default 1), fitted so neither edge exceeds the model API's image limit, and records that frame: frame_point maps a point read off the latest screenshot back to CSS pixels. zoom enlarges a region of that frame for a closer look. Captures clip at the scroll offset, so a scrolled page shows its viewport.
Each tab collects in the background what it logged: - console messages, uncaught exceptions and browser log entries (such as failed resource loads), with level and source location; - network requests with method, status, type and MIME type, or why they failed. A response body can be fetched by request id. More Tab verbs: - javascript runs code with REPL semantics (top-level await, the last expression's value) and returns an exception as an error; - form_input sets a control by ref: a select by option value or text, a checkbox or ARIA switch by true/false, a field through the value setter frameworks such as React track. It fires input/change events; - set_viewport and set_color_scheme emulate a size, a phone (touch, mobile layout, Android user agent) and prefers-color-scheme; - page_text returns the main/article text, or the body's; - history goes back or forward.
The browser tools now follow the shape of the browser tools models are trained on. browser_read and browser_act are replaced by small tools that each do one thing, and a tool returns what was asked for instead of a screenshot, page text and element list after every step: - browser_navigate: load a URL, or go back/forward; returns URL+title. - browser_read_page / browser_find: the accessibility tree with ref_N handles and element state, whole, interactive-only, as a subtree or filtered by a query. - browser_computer: click (by ref or by coordinates in the latest screenshot's frame, with modifiers and click counts), type, key chords with repeat, scroll, scroll_to, hover, drag, wait, and screenshot/zoom with a scale. Each screenshot reports its frame. - browser_form_input: set a select, checkbox or field by ref. - browser_get_page_text, browser_javascript (REPL semantics). - browser_read_console_messages, browser_read_network_requests (including a response body by request id). - browser_tabs_context/create/select/close, browser_resize_window (presets, phone emulation, color scheme). - browser_batch: several of these in one call, stopping at the first error, screenshots returned in order. Each action reports what changed around it: dialogs answered, tabs the page opened, a navigation it caused. Every tool takes an optional profile and tab_id. browser_close, browser_login and browser_profiles stay, moved onto tabs. A session adopts only pages that have an opener, so Chrome's initial blank tab no longer shows up as a tab. Outputs stored by the old tools still load; their tool names are gone. The GPUI cards show screenshots for navigate, computer, batch and login; the other tools render inline.
The tools now act by accessibility refs and screenshot coordinates, so Tab drops what only the old browser_act/browser_read used: the element-discovery script and observe, CSS and text=/role=/aria= selectors with click, type_text, fill, clear, press_key and wait_for, selector-based scroll, eval and the full-page screenshot. So do the session-level verbs that forwarded to the active tab. The web tests move to the tab API. Tests that only covered the removed verbs are gone; their behavior is covered by the AX, mouse, keyboard and form tests.
docs/browser-tools.md lists the tools and explains refs, the screenshot coordinate frame, dialogs, timeouts and tabs. The agency plan points to it where it describes the replaced tools.
a_hung_page_fails_fast_instead_of_hanging clicked a button whose handler started an endless loop 50 ms later. Under a full parallel test run the loop could start before the click's own CDP acknowledgment arrived, so the setup click timed out under the test's 1 s limit. The loop now starts through a script with a 100 ms margin.
On Linux CI the drag logged its mousedown but no mouseup. Pressing on selected text starts a native drag-and-drop, which ends without a mouseup, and what the preceding double click leaves selected differs between platforms. The test now clears the selection before dragging, and the page also logs dragstart, so a drag-and-drop shows up in the failure message if it happens anyway.
Browser work is many small calls, so each now reads like the other inline tools: "Click ref_4", "Find "Sign in"", "Browser steps (3 steps)". code_assistant_core::tools::impls::browser::describe holds the wording, so GPUI and the terminal describe calls the same way. - GPUI: one inline renderer for all browser tools. The header tag shows a named profile and the tab. Expanding a call shows its result: trees and logs in monospace, capped at 40 lines, and screenshots at up to 380 px high. The browser cards and their starts_collapsed override are gone. - Terminal: browser tools had no renderer, so the fallback dumped every parameter and the whole output (a page tree, the page text) into the scrollback. Each call now shows its description and the first line of its result; a batch lists its steps.
A game reads "keep walking" as a key held down, and drag targets want a button held through a move. Tab gets key_down / key_up (keys stay down across calls, released last first), hold_keys (down, wait, up, as trusted CDP events), and mouse_down / mouse_up at a point or where the mouse is. The tab tracks the mouse position and held buttons, so moves while a button is down carry it. Key events no longer set nativeVirtualKeyCode. The key table's codes are Windows virtual key codes; on macOS they name other keys (W's 87 is Keypad 5), and a held W made Chrome auto-repeat hundreds of Numpad5 keydowns.
browser_computer gets hold_key (`text` held for `duration` seconds), key_down / key_up (held across steps, e.g. walk while jumping) and left_mouse_down / left_mouse_up. Without them a model drove a game by dispatching synthetic KeyboardEvents through browser_javascript, which only works on pages that accept untrusted events. The descriptions of browser_computer and browser_batch now say that the page keeps running between calls, also while the model thinks, and that timing-sensitive input belongs in one batch. In a game session the model held a key across a separate wait call and its thinking, and misjudged how far a "0.5 s" push had carried it.
A key pressed with key_down stays down until key_up, also across turns.
A model that forgets to release it leaves the page receiving a held
key. Every browser action's result now notes what is still held down
("Still held down: w, left mouse button"), until it is released.
Held keys are tracked by key code, so a release matches whatever name
or case the press used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The browser tools returned a screenshot, up to 4,000 characters of text and up to 200 CSS-selector elements after every step. That cost 3–7k tokens per call, and the
:nth-of-typeselectors broke on re-renders. They also had no console, no network view, no tabs and no form helpers. This PR moves them to the shape of the computer-use style browser tools models are trained on, such as the built-in browser in Claude Code. Old sessions are allowed to break.The tools
browser_navigateback/forward; returns URL and titlebrowser_read_page/browser_find- role "name" [ref_N] state, whole, interactive-only, as a subtree or by querybrowser_computerbrowser_form_inputbrowser_get_page_text,browser_javascriptbrowser_read_console_messages,browser_read_network_requestsbrowser_tabs_context/create/select/close,browser_resize_windowbrowser_batchbrowser_close,browser_login,browser_profilesActions return one line, plus notes for dialogs answered, tabs the page opened, or a navigation the action caused. The model asks for a screenshot or the tree when it needs one. Coordinates refer to the frame of the latest screenshot, which reports its size. The tool definitions grow from about 7.2k to 12.8k characters (~3.3k tokens), while the output of each call shrinks.
Engine (
web)BrowserSessionholds the browser and its tabs.Tabholds one page and its state: refs, the screenshot frame, dialogs, and the console/network log.The page keeps running between calls, also while the model thinks. The descriptions say so, and point timing-sensitive input (games, animations) to
browser_batch, where steps run back to back andwait/hold_keydurations are exact.UI
Each browser call renders as one line, described the same way in both frontends through the shared
browser::describemodule: "Click ref_4", "Find "Sign in"", "Browser steps (3 steps)".ToolKind::Other, with titles from the tools'title_template.Notes for reviewers
Key events no longer set
nativeVirtualKeyCode. The key table holds Windows virtual key codes, which name other keys on macOS (W's 87 is Keypad 5). A held W made Chrome auto-repeat hundreds ofNumpad5keydowns.Persisted outputs of the old tools still deserialize. Records of the removed tool names (
browser_read,browser_act) are skipped when a session is restored, the same path already used for removed MCP tools.browser_computerandbrowser_form_inputare write tools. An embedder that gates outward actions can tag them (seedocs/browser-agency-plan.md).test_web_searchneeds network access to DuckDuckGo. It fails locally only because DuckDuckGo is unreachable from this machine.docs/browser-tools.mddescribes the tools and concepts.Testing
web: unit tests for the AX renderer. Browser tests cover tabs and popups, refs, mouse and keyboard, the screenshot frame (also on a scrolled page), console and network, REPL eval, form input, emulation, dialogs, timeouts and cookies.code_assistant_core: tool tests per group:ui_gpui: descriptions of the cards.cargo test --no-fail-fastpasses apart fromtest_web_search.cargo clippy --all-targets --all-features -- -D warningsandcargo fmt --checkare clean.