Skip to content

Browser: record action – motion as one contact sheet - #219

Merged
stippi merged 4 commits into
mainfrom
feature/browser-record
Oct 3, 2026
Merged

stippi merged 4 commits into
mainfrom
feature/browser-record

Conversation

@stippi

@stippi stippi commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Adds a record action to browser_computer so the agent can judge motion: animations, UI transitions, a character walking in a game.

What it does

{"action": "record", "duration": 1, "frames": 9, "scale": 0.5} films duration seconds of real time (max 5) and returns one image: a contact sheet of frames cells (2–16, default 9 → 3×3), evenly spread from start to end, left to right, top to bottom. Each cell is labelled #n +0.25s (built-in 5×7 bitmap font, no font dependency). The text says:

9 frames over 1s (3×3, left to right, top to bottom), one every 0.125s
Unchanged from the frame before: #6, #7 (the page painted 12 frames in 1s)
Contact sheet 1567×1039 px (click coordinates still refer to the last screenshot)

It works in browser_batch (e.g. key_down w, record, key_up w) and gets the usual notes ("Still held down: w …").

How

  • web::Tab::record (tab.rs) and web::recording (cell layout, frame picking, sheet drawing; pure, unit-tested). web now depends on image 0.25, which was already in the lockfile via tools_core.
  • Frames come from a CDP screencast (Page.startScreencast / screencastFrame / screencastFrameAck / stopScreencast, typed in chromiumoxide 0.9). The paint timestamps in metadata match the wall clock. A cell shows the last frame painted by its moment.
  • The sheet starts from a screenshot of the page as the recording begins. A screencast sends nothing for a still page, and a late first frame would otherwise stand in for earlier moments.
  • "Unchanged" compares pixels (with a JPEG noise tolerance), so a repaint that left the picture as it was still counts as unchanged.
  • Bounded like every other verb (command × 2 + duration). The sheet stays within MAX_SCREENSHOT_EDGE; scale only shrinks it.

Window size (requires Chrome 140+)

A screencast frame shows the browser window, not the emulated viewport. Headless Chrome's default window is 800×600, and --window-size sets the outer size: headless Chrome reserves part of it for browser UI (143 px on macOS, so a 1280×800 window has a 1280×657 inner area). Without a fix, a moving box right of x≈800 produced no frames at all.

record therefore first sets the window contents to the viewport with Browser.setContentsSize. That command exists from Chrome 140 on (devtools-protocol 0.0.1484773, between the 139 and 140 branch points); on older Chrome, record fails with a clear error. Launch and set_viewport are unchanged. Frames are still cut to the viewport using their metadata, and any from before the fit are dropped. The tests cover a deliberately shrunken window; without the fit, no frames arrive.

Tests

  • web: recording_makes_a_contact_sheet_of_the_motion uses a rAF-driven box against real Chrome. It checks the box position per cell, that every fresh cell shows it somewhere new, and that the box is square in every cell (cut, not stretched). It also covers a viewport taller than the window, a deliberately shrunken window, and a still page (all cells unchanged). Load-tolerant: it allows a couple of skipped cells, and the frame rate in the tall viewport is compared relative to the first recording. Plus unit tests for grid, cell size, frame picking and the noise tolerance.
  • code_assistant_core: record via browser_computer, the 5 s limit, the result text, a browser_batch with key_down/record/key_up, and describe.rs ("Record 2s").

Tool definitions grow by about 250 characters (12.57k → 12.82k).

Not included, as agreed: virtual time, and Animation.setPlaybackRate.

stippi added 4 commits October 3, 2026 11:17
Tab::record collects Page.screencastFrame frames for a stretch of real
time and lays evenly spread moments out as one PNG contact sheet, each
cell labelled with its offset (built-in 5x7 bitmap font). Cells with
no repaint since the one before repeat it and are marked.

Headless Chrome only reports repaints inside its window to a
screencast, and that window was the default 800x600 while the viewport
is emulated at 1280x800: launch with a matching window, and grow it
when a larger viewport is emulated.
A screencast frame shows the browser window, not the emulated viewport,
and headless Chrome reserves 143 px of the window height for browser
UI: a 1280x800 window gave 1280x657 frames, which were stretched to
the cell. Frames are now cut to the viewport; the window gets 200 px
on top of the viewport, and a frame showing less than the viewport
grows the window by what is missing.

The sheet starts from a screenshot of the page as the recording begins
(a still page paints nothing), and a cell counts as unchanged when it
looks like the one before rather than when no frame arrived: a repaint
can leave the picture as it was. Frames are requested at about cell
size, which keeps decoding cheap.
record films `duration` seconds (max 5) and returns one contact sheet
of `frames` evenly spaced frames (default 9), with the layout, timing
and unchanged cells in the text. It runs in browser_batch between
key_down/key_up like any other step and gets the same notes.
describe.rs names it "Record 1s"; docs updated.
…ording

Instead of a guessed 200 px allowance for headless Chrome's reserved
browser UI and growing the window by the shortfall frames report,
record sets the window contents to the viewport with
Browser.setContentsSize (Chrome 140+, a clear error otherwise). The
launch config and set_viewport are back as on main; frames are again
requested at cell size.
@stippi
stippi merged commit 952a52b into main Oct 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant