Skip to content

Agent's browser live in the right panel - #221

Merged
stippi merged 16 commits into
mainfrom
sidebar-browser
Oct 5, 2026
Merged

stippi merged 16 commits into
mainfrom
sidebar-browser

Conversation

@stippi

@stippi stippi commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Shows the agent's browser live in the right panel, next to the review view (steps 1–4 of docs/browser-panel-plan.md). Chrome keeps running headless in its own process; the panel mirrors it over a CDP screencast. The GPUI part sits behind the Cargo feature browser-panel, which is on by default.

What the panel does

  • Review | Browser switch in the panel header, remembered per session. When the agent opens its first browser, an open panel switches to it.
  • Live view of the agent's active tab, or of a tab the user picked. Browsers of running sub-agents are listed as well. A short pulse marks where the agent clicks.
  • Take control:
    • The user's mouse, wheel and keyboard go to the page: macOS editing shortcuts carry their command, input methods compose through EntityInputHandler, and copy, cut and paste use the system clipboard.
    • The toolbar gives back, forward, reload and an editable address.
    • While the user has control, the agent's browser tools fail at once with a clear message. After the hand-back, the next result tells the agent where the tab is now.
  • Lifetime: a throwaway browser that is being watched survives the end of the turn and closes when the last viewer lets go.

Layers

  • web:
    • one screencast pump per tab, shared by live views and record, running only while someone watches;
    • a watchable tab state: address, title, loading; popups are adopted without a tool call;
    • raw user input;
    • view guards on the manager;
    • an instance number per launched browser.
  • code_assistant_core:
    • SessionBrowsers covers the agent's browsers plus those of running sub-agents;
    • UiEvent::BrowsersChanged carries metadata only, never frames;
    • SessionService::{watch_browsers, browser_view, set_browser_control};
    • BrowserView with an ordered input queue.
  • ui_gpui: right_panel/browser/ (view, toolbar, input, key mapping, geometry). The view switch of the left sidebar became the shared segmented_switch.

Fixes found on the way

  • main is no longer #[tokio::main]. GPUI's event loop ran inside one never-ending poll of tokio's block_on, so tokio's cooperation budget on the main thread was never renewed. After about 128 operations, every tokio watch/broadcast awaited in a GPUI foreground task stayed pending. The panel froze while the screencast went on. This may explain other stalls seen on the main thread.
  • A screencast start that fails because the page navigates under it (Cannot find context with specified id) is retried.

Testing

  • web: tests against real Chrome for live frames, recording during a live view, a start during navigation, tab state and popups, viewer lifetime, user control and raw input.
  • core: listing and events, sub-agent registration, views keeping a browser, input only under user control, reopened browsers.
  • ui_gpui: tests for the geometry mapping, the key mapping and address schemes.
  • Manually, in a release build: a GPT-6.1 run driving a WebGL page (lunar-walk). The panel keeps up with 60 fps at about 4 ms decode per frame.
  • cargo clippy --all-targets --all-features -D warnings is clean, and so is a build with --no-default-features --features gpui-frontend.

Not in this PR

  • The <select> overlay and file chooser (step 5) wait for the CEF decision.
  • a_hung_page_fails_fast failed twice under load early on this branch and could not be reproduced afterwards.

stippi added 16 commits October 4, 2026 12:51
…ser-panel feature

Review | Browser switch in the panel header, remembered per session. The
view lists the session's browsers from BrowsersChanged events, watches the
active tab (or a picked one) through a BrowserView, decodes frames off the
main thread and pulses where the agent clicks. The panel switches to the
browser when the agent opens its first one.
While the user has control the panel forwards mouse, wheel and keys to the
page (macOS editing shortcuts carry their command, an input method composes
through EntityInputHandler, copy/cut/paste use the system clipboard), and
the toolbar navigates: back, forward, reload and an editable address.
Hand back returns the browser to the agent.
GPUI's event loop ran inside the #[tokio::main] future, in one poll that
never returns, so tokio's cooperation budget on the main thread was never
renewed. After about 128 tokio channel operations there, every tokio
watch or broadcast awaited on the main thread stayed pending: the browser
panel stopped showing frames while the screencast went on. main is
synchronous now; the async modes run on a runtime of their own.
Each launched browser carries an instance number; the listing reports it
and the panel watches key, instance and tab, so a browser closed and
opened again (with tab t1 again) no longer leaves the panel on the dead
tab.
@stippi
stippi merged commit b4988f3 into main Oct 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant