Repository navigation
Browser: record action – motion as one contact sheet - #219
Merged
Merged
Conversation
Tab::record collects Page.screencastFrame frames for a stretch of real time and lays evenly spread moments out as one PNG contact sheet, each cell labelled with its offset (built-in 5x7 bitmap font). Cells with no repaint since the one before repeat it and are marked. Headless Chrome only reports repaints inside its window to a screencast, and that window was the default 800x600 while the viewport is emulated at 1280x800: launch with a matching window, and grow it when a larger viewport is emulated.
A screencast frame shows the browser window, not the emulated viewport, and headless Chrome reserves 143 px of the window height for browser UI: a 1280x800 window gave 1280x657 frames, which were stretched to the cell. Frames are now cut to the viewport; the window gets 200 px on top of the viewport, and a frame showing less than the viewport grows the window by what is missing. The sheet starts from a screenshot of the page as the recording begins (a still page paints nothing), and a cell counts as unchanged when it looks like the one before rather than when no frame arrived: a repaint can leave the picture as it was. Frames are requested at about cell size, which keeps decoding cheap.
record films `duration` seconds (max 5) and returns one contact sheet of `frames` evenly spaced frames (default 9), with the layout, timing and unchanged cells in the text. It runs in browser_batch between key_down/key_up like any other step and gets the same notes. describe.rs names it "Record 1s"; docs updated.
…ording Instead of a guessed 200 px allowance for headless Chrome's reserved browser UI and growing the window by the shortfall frames report, record sets the window contents to the viewport with Browser.setContentsSize (Chrome 140+, a clear error otherwise). The launch config and set_viewport are back as on main; frames are again requested at cell size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
recordaction tobrowser_computerso the agent can judge motion: animations, UI transitions, a character walking in a game.What it does
{"action": "record", "duration": 1, "frames": 9, "scale": 0.5}filmsdurationseconds of real time (max 5) and returns one image: a contact sheet offramescells (2–16, default 9 → 3×3), evenly spread from start to end, left to right, top to bottom. Each cell is labelled#n +0.25s(built-in 5×7 bitmap font, no font dependency). The text says:It works in
browser_batch(e.g.key_down w,record,key_up w) and gets the usual notes ("Still held down: w …").How
web::Tab::record(tab.rs) andweb::recording(cell layout, frame picking, sheet drawing; pure, unit-tested).webnow depends onimage0.25, which was already in the lockfile viatools_core.Page.startScreencast/screencastFrame/screencastFrameAck/stopScreencast, typed in chromiumoxide 0.9). The paint timestamps inmetadatamatch the wall clock. A cell shows the last frame painted by its moment.command × 2 + duration). The sheet stays withinMAX_SCREENSHOT_EDGE;scaleonly shrinks it.Window size (requires Chrome 140+)
A screencast frame shows the browser window, not the emulated viewport. Headless Chrome's default window is 800×600, and
--window-sizesets the outer size: headless Chrome reserves part of it for browser UI (143 px on macOS, so a 1280×800 window has a 1280×657 inner area). Without a fix, a moving box right of x≈800 produced no frames at all.recordtherefore first sets the window contents to the viewport withBrowser.setContentsSize. That command exists from Chrome 140 on (devtools-protocol0.0.1484773, between the 139 and 140 branch points); on older Chrome,recordfails with a clear error. Launch andset_viewportare unchanged. Frames are still cut to the viewport using their metadata, and any from before the fit are dropped. The tests cover a deliberately shrunken window; without the fit, no frames arrive.Tests
web:recording_makes_a_contact_sheet_of_the_motionuses a rAF-driven box against real Chrome. It checks the box position per cell, that every fresh cell shows it somewhere new, and that the box is square in every cell (cut, not stretched). It also covers a viewport taller than the window, a deliberately shrunken window, and a still page (all cells unchanged). Load-tolerant: it allows a couple of skipped cells, and the frame rate in the tall viewport is compared relative to the first recording. Plus unit tests for grid, cell size, frame picking and the noise tolerance.code_assistant_core: record viabrowser_computer, the 5 s limit, the result text, abrowser_batchwithkey_down/record/key_up, anddescribe.rs("Record 2s").Tool definitions grow by about 250 characters (12.57k → 12.82k).
Not included, as agreed: virtual time, and
Animation.setPlaybackRate.