Repository navigation
The OpenBot Recorder and the OpenBot Browser: teach a Bot by doing it in your own browser - #858
Open
davidmckayv wants to merge 468 commits into
Open
davidmckayv wants to merge 468 commits into
davidmckayv wants to merge 468 commits into
Conversation
…han the host's running-app check and its own request for a connect code
… the demonstration URL check, the binding store's type alias and the Learning check the Recorder wiring handed out, and keep private what only its own module reads
…and where it goes: no screen, sound or transcription
…ivate to their modules
…cut at 2 MiB key by key, so no password, PIN, card number or session token in a form post, a URLSearchParams body or an oversized JSON body reaches Intelligence on either capture path; the page hook records the content type the browser adds, and a password manager's untrusted fills are hashed too
…oes, so a live connect code from OpenBot's own API never reaches Intelligence when capture falls back to the page hook
…ed content script: a post or flush that throws on a port Chrome has not yet reported dead is queued and sent after the reconnect, so a sensitive-value digest is never lost; once chrome.runtime.id is gone after an extension update the port closes for good and capture stops instead of retrying every 4 s
…ete a connect code when it is used or expired, clear the session when a connection is revoked or replaced, and refuse a person more than ten unexpired codes at once
…che as new ones arrive, and tie each trajectory binding to its Bot so deleting the Bot deletes its bindings
…says so: request-actor now uses the gate's isTrajectoryConnectPath, which also refuses an id that does not decode, and the guard's bearer prefix instead of its own copies
…wser in and handing out capture passes, while what is recorded goes to Intelligence, not to OpenBot
…tension, the only place that reads them, and delete the network body size the server no longer uses
…nterface rules go back to the app's styles.css so the Recorder's pages do not import them, the id-from-public-key derivation moves into the one test that uses it, the dead gesture re-exports go, and a test name says recording
…tests fail on 1.3
…gence operator take: drop the Intelligence source paths, commits, gaps list and open-bug section, add how to let OpenBot Desktop's native messaging host through browser policy, and say to confirm the Recorder's policy shows OK in chrome://policy and that a value of the wrong type is refused; the masking and fake gateway comments stop citing Intelligence source files too
…ettings > Browsers, the capture pass, OPENBOT_RECORDER_EXTRA_EXTENSION_IDS and the boot failure on a bad value, and the runtime 1.79.1 move to Chat Completions behind a non-OpenAI OPENAI_BASE_URL with COPILOTKIT_OPENAI_API=responses to keep the Responses API; both variables are in .env.example
…E on Linux, keep debug builds out of the developer's own browsers, and remove the host registrations on uninstall On Windows every browser's host key is now written whether or not the browser is there yet, since the manifest lives in OpenBot's own folder; on macOS and Linux the app checks again every minute and writes only a manifest that is missing or different. On Linux each browser's folder follows an absolute $XDG_CONFIG_HOME, and $CHROME_CONFIG_HOME for Chrome and Chromium, as Chrome's own user data directory does. A debug build no longer installs at launch, its --install-recorder-host writes only under OPENBOT_RECORDER_HOST_TEST_HOME and never the registry, and the integration tests point it at a temporary folder. The NSIS uninstaller removes the three host keys and the manifest, except during an update. The host name and extension id are read from the shared file once, in one place.
…table code, each message within a deadline below the extension's wait The ping no longer runs the handoff's process ownership checks (lsof, or PowerShell and netstat on Windows): it reads whether a deployment is selected, asks the app's port for /api/capabilities, and in organization mode reads whether Desktop holds a sign-in for the organization, without the network. Every reply now carries ok, and a failed one carries error, one of origin-not-allowed, not-installed, not-running, not-signed-in, organization-unavailable, handoff-failed, timed-out, host-failed or unknown-message. Each message runs on its own thread and is answered timed-out at 2 s for a ping and 7 s for a handoff, inside the extension's 3 s and 8 s. On this machine a release ping now takes about 3 ms (it took about 175 ms) and a handoff about 175 ms. The host reads the host name and extension id from the install module's one copy, and a test runs a full organization handoff and checks that the host's stderr holds no connect code and no part of a cookie.
…ment Desktop never builds or runs the extension; the frozen bun install needs only its workspace manifest, so its source, tests and assets now stay out of every deployment. A simulated deployment with only that file still installs with bun install --frozen-lockfile.
…its host registrations, macOS and Linux keep the per-person host file until someone deletes it
…Recorder records no screen, so nothing may call tabCapture
…utcome naming why: its live events are already in Intelligence, so it never reads as a recording the person finished
…olicy pin elsewhere both end the live recording, halt the document's streams for its token and keep everything for its person; a pin also revokes the token, a new gateway list reaches running streams, and Connect and Record re-check the policy
… connecting revokes the old token, so the live recording, its stored session and the document's stream take the new one, and a stream the old token already signed out resumes from the outbox
… once its port stays down past its reconnect window, reopen it for a Stop that finds it gone, and refuse Start over while it cannot be reached, so events are not dropped and Stop does not wait for a browser restart
…sage, so the extension typechecks
…s it and counts the dropped events, and Stop reports that nothing reached Intelligence (a resumed recording keeps what it stored, parked) instead of Sent to Intelligence
…ne and the service worker told, so closed records no longer pile up and a discard's walk stays short; a recover that finds a recording's records gone sends nothing again
…m before the rework left (its other stores, other meta keys and demonstration trajectories with their events), and read stored recordings through a shape check so an old build's demonstration is never sent
…how a plain sentence for it in the popup (a general one for a code the Recorder does not know), since a good ping no longer promises a good handoff; the host-deadline comments now cite its 2 s ping and 7 s handoff
The store's disclosure sat below "Allow the microphone" and its explanation. The Chrome Web Store asks for it to be seen before anything is collected, so it is now the page's own heading, and the microphone is a section under it.
…estart, and the setup page opens with what the Recorder collects
…wser When OpenBot is open in a browser that holds the Recorder, its Browser button works with that browser: Record and Stop from OpenBot, the red dot read from the Recorder, and only the connected OpenBot may ask. Connecting skips the Connect page only for an OpenBot the organization pins or OpenBot Desktop.
…Stop, and a trusted connect with no Connect page
When OpenBot is open in a browser that holds the Recorder, its Browser button works with that browser. The Recorder
now answers openbot.state, openbot.record and openbot.stop over externally_connectable (shared/recorder-page.ts),
validated with recorderPageMessageSchema, and only for the OpenBot it is connected to, by the origin Chrome reports
for the sender. Any other page gets { connected: false } for state and the same refusal for record and stop, whether
the Recorder is connected or not, so it learns nothing. Record starts a recording named title on the tab that sent it
(OpenBot's own top frame only, so a frame inside another site cannot name that site's tab) through the popup's own
Record, so capture is unchanged and the recording follows every tab the person uses; Stop goes through the popup's
Stop. State reports the phase, title, start, action count and the last stopped recording with its actions and
phrases, read as the launcher API reads them.
openbot.connect from a trusted OpenBot (David: "only for trusted OpenBots") connects with no Connect page: the
OpenBot the organization's policy pins, or OpenBot Desktop's loopback address (http on a loopback IP, the only http
the Recorder accepts). It takes an approved Connect's steps, as OpenBot Desktop's own connect does: the same limit,
the code exchanged at the reported origin under this copy's install id and the browser's name, the connection stored
in a queued job that checks the policy again (a token it keeps out is revoked), an open Connect page closed, the
signed-out notice gone, and the router's approved-Connect follow-up. Its answer says connected: true. Every other
origin, https on a loopback IP included, keeps the Connect page exactly as before.
The connection tests' web OpenBot was the fake OpenBot's loopback address, which now connects with no Connect page,
so it is an https site reached through fetch at the same fake, and OpenBot Desktop's handoffs name the loopback
address.
…Connect page only for trusted OpenBots
…wser button Where OpenBot is open in Chrome or Edge, the sidebar's Browser button now works with that browser instead of ignoring it. With the Recorder there and connected to this OpenBot, one click opens a popover from the button: a task name and Record, then the name, elapsed time, action count and Stop, then what Stop saved, said as the Browser page says it. Record and Stop go to the Recorder over openbot.record and openbot.stop (shared/recorder-page.ts), and the red dot reads the Recorder's own openbot.state, every second while it records and every 30 s otherwise, so it lights at once. With the Recorder there but not connected here, the click connects it as Settings, Browsers does (a minted code and openbot.connect), and the popover waits for the Connect click in the Recorder's tab, or goes straight to Record for a trusted OpenBot. With no Recorder, the button links to its store page in a new tab, with a short note, once the Recorder is published there. Refusals show the Recorder's own sentence in the popover. A page has chrome.runtime.sendMessage only while an extension lists it in externally_connectable, so its absence in Chrome or Edge means no Recorder yet, not a browser that cannot hold one; the store page follows from the brands the browser reports. Adding the Recorder gives an open page the API at once, so the Recorder is asked again whenever the person comes back to the tab. OpenBot Desktop's own app window (WebView2 on Windows, a WebKit view elsewhere), Safari, Firefox, a phone, and a Recorder from before openbot.state keep today's button. The popover is shadcn's Base UI popover, as wide as the button and opening above it like the account menu. Base UI gives the button aria-expanded and aria-haspopup and moves focus in and back; when Record gives way to Stop, Stop to the task name, or the connect message to Record, focus moves to the control that took the place of the one in use.
… connect needs no Connect page The Recorder now connects OpenBot Desktop's loopback address, or the OpenBot its organization pins, with no Connect page, and answers openbot.connect with connected: true. Settings, Browsers took any accepted connect as waiting on that page, so it asked the person to confirm in a tab that never opened until its next 2 s read found the connection. It now reads the new connection at once and says Connected. Its refusal sentences come from the bridge, shared with the Browser button, which also says the two refusals a trusted connect can end in: OpenBot refused the Recorder's code, or the Recorder could not reach OpenBot.
Brings in OpenBot #824, #826, #827, #833, #836, #838 and #847. Conflicts: - Migrations: main added 0061 to 0065, and this branch had its own 0061. The Recorder's tables are regenerated as 0066_recorder_extension from the merged schema, statement for statement the same as the old 0061; the CHANGELOG and the two tests that name it follow. - page-shell.tsx and styles.css: this branch had moved the page frame into page-parts.tsx and the theme tokens into theme.css, which the Recorder shares. Main's layout pass (768px note, page rhythm, 56px rows with inset dividers, PageCard, the empty-state PageEmpty, the 15/16 type scale on a 16px root, 0.5rem radius, body text-base) is carried into those files, and main's @-menu rule into styles.css. - CHANGELOG: both "Before upgrading" lists, eleven items.
…wser branch Brings in OpenBot #824, #826, #827, #833, #836, #838 and #847 by way of the Recorder branch's merge of main. Conflicts: - Migrations: this branch's 0062_recorder_activity is regenerated as 0067_recorder_activity after main's 0061 to 0065 and the Recorder's 0066, statement for statement the same; the CHANGELOG follows. - server/src/index.ts: main's computerGuidance (renamed from COMPUTER_GUIDANCE) beside this branch's isPersonBrowserId import. - ci.yml: both new test files in the list. - docs/configuration.md: main's COMPUTER_SEPARATE_WORKSPACES row beside this branch's browser rows. - CHANGELOG: both "Before upgrading" lists, twelve items.
The notice on a recorded page carries its own copy of the app's theme tokens, and its test holds the two together; main's layout pass moved the radius from 0.55rem to 0.5rem in the app's theme, so the notice follows.
davidmckayv
requested review from
MikeRyanDev,
guidovizoso,
mxmzb and
tylerslaton
as code owners
October 27, 2026 21:39
The OpenBot Browser's environment test wrote its expected folders with /, and on Windows Path::join puts \ between the parts, so the test failed there and only there. The expected folders are now joined as the code joins them.
guidovizoso
requested changes
Oct 27, 2026
guidovizoso
left a comment
guidovizoso
left a comment
Collaborator
There was a problem hiding this comment.
The core is careful work:
- Connect codes are 256-bit, hashed, single use and expire after 5 minutes.
- Every route is scoped to its owner.
- The migrations are additive only, with cascading FKs.
- Password and card fields are masked before anything leaves the browser.
- Only the public key is in
extension/keys. - Chrome is driven over
--remote-debugging-pipe, so no debugging port is open. - The app follows the PageShell and data-access conventions.
Requesting changes for three blockers and a set of privacy and UX gaps.
Blockers
- Any local web page can take over the Recorder's connection without a click (
extension/src/background/connection.ts:419,trustedConnectRule). Anyhttp://127.0.0.1:<port>or[::1]origin is treated asloopback-desktop.trustedConnectthen exchanges the code and overwrites the stored connection, with no Connect page and no revoke of the old one. Combined withexternally_connectable: <all_urls>and an unpinned gateway policy, any local dev server, notebook or local app with an XSS can re-point the Recorder and receive the person's next recordings. Please trust loopback only when the native host confirms the origin, and never replace a live connection silently. - Personal and opened skills never reach a run.
AcquireLearnedSkillstakesuserId(server/src/learning/runtime.ts:32), but neither real caller passes it (server/src/copilot.ts:1037,server/src/learning/remote.ts:57). Only unit tests do. Settings → Learning "open" is saved and audited aslearning.sharing_changed, but has no effect. Please wire it through and add an integration test across the run boundary, or hide the UI until it is wired. Once it works, "open" puts one person's recorded and narrated text into every Bot in the deployment, which is a cross-person prompt-injection path. It needs an admin-level gate before it ships. - In shared mode, a Bot's shell can read the person's browser profile. With
OPENBOT_BROWSER_URLunset, person browsers run on the shared computer (server/src/browser/host.ts). There they sit under/profiles, in the same root container where/execis allowed (agent-computer/src/browser-runtime.ts:49). A prompt-injected Bot can read/profiles/person-<hex>/Default/Cookies. Please refuse person browsers on a shared computer, or give each person their own.
Privacy and consent
- The disclosure understates what is collected.
- "Only what you do is recorded" (
browser-page.tsx:187) andWHAT_IT_RECORDSlist clicks, typing and pages. - Recordings also carry network request and response bodies, AG-UI events on any AG-UI page (including the person's own chat messages), and narration.
- The setup page says the browser "sends only the text". Chrome's Web Speech API usually streams the audio to Google's speech service, and
offscreen/voice.tsdoes not request on-device recognition. - Please fix the copy or require on-device recognition, and say all this before the first Connect in Settings → Browsers.
- "Only what you do is recorded" (
- A recording follows every tab the person activates. Request and response bodies get only heuristic redaction, there is no site exclusion list, and the in-tab notice appears only after 50 actions. A glance at a bank or webmail tab mid-recording gets uploaded.
- The microphone is on by default when Record starts (
use-narration.ts:37). - There is no way to delete a recording from the app or the server.
externally_connectable: <all_urls>:- every site can
pingto see whether the Recorder is installed; - every site can open a Connect page that replaces the real connection in one click;
- server error text from that origin is shown inside the extension's UI.
- Please say "This replaces your connection to X", and show only fixed error strings.
- every site can
- Lifetimes are unbounded:
- Recorder tokens never expire, and signing out doesn't end them.
- Organization mode stores an encrypted copy of the person's full session cookie for the life of the connection.
- Recordings queued for another person sit in IndexedDB indefinitely.
- Trajectories in Intelligence outlive user deletion.
- The logged-in local Chrome profile and the native host manifests survive uninstall. On Windows the
RMDiris non-recursive.
Should fix
- Some refusals write no audit row: unknown, expired or used codes, and the 429s on
/connect-codesandconnectNow.recorderAuditswallows failed writes. Please call these out against the "refusals write a row" rule. - Two concurrent exchanges for the same person and install can hit
recorder_connections_live_install_idxand return a 500 across replicas. Catch23505, or retry. - The 1,838-line
@copilotkit/learning@0.0.1patch carries security-relevant masking. It stops applying silently on a version bump. Please upstream it, or add a test that fails when the patched exports are missing. - Windows force-kills Chrome (
taskkill /T /F,stack.rs:2085). Unix uses SIGTERM and waits, so Windows risks the profile damage the comment atstack.rs:1596says this avoids. - Cloud person browsers are exempt from idle close (
profiles.ts:479). An abandoned tab keeps a root--no-sandboxChromium running, and the cap evicts Bots' browsers first. - On Linux only
/opt/google/chromeand/opt/microsoft/msedgeare found. Otherwise the Browser button just disappears, with the reason only on stderr. --disable-blink-features=AutomationControlledand--enable-unsafe-extension-debuggingneed an explicit sign-off. The latter shows Chrome's unsupported-flag bar, which the UX never explains.- UX: "This browser" has no install action when the Recorder is missing; reuse
StorePageLink. Disconnect has no confirmation, and one pending revoke disables every row.
Scope
Please move the CopilotKit 1.79.1 runtime bump and COPILOTKIT_OPENAI_API into their own PR, together with the tests tied to 1.79. They change how OpenAI-compatible Bots are called and should be revertable on their own.
Nits
<PageEmpty>Opening your browser…</PageEmpty>is loading text, which the data-access skill says not to use.- The tab strip uses
role="tablist"with no arrow keys and notabpanel. recorder_install.rssays the install check runs every minute, but it runs once.- The Recorder is built with
bun run buildon the user's machine during install. Ship it prebuilt. mintCodecounts and then inserts, so the cap isn't atomic.PUT /skills/:nameaccepts a name the person doesn't have.
Anthropic's SDK 2.3.0, which CI now installs for the AG2 harness, sends the key as Authorization: Bearer instead of x-api-key. The agent's key still reaches Anthropic; the test only looked for the old header, so it failed on every pull request. It now accepts either, and passes with Anthropic's SDK 1.17.0 and 2.3.0 alike. (cherry picked from commit a1391e7)
…owser # Conflicts: # server/Dockerfile
…owser # Conflicts: # server/src/computer/routes.ts
Blockers: - A loopback page connects the Recorder with no click only at OpenBot Desktop's own address, which Desktop's native host confirms; a live connection is never replaced silently, the Connect page names the one it replaces, and the replaced token is revoked. - A person's Skills reaching their Bots is pinned by an integration test across the run boundary; opening Skills to others needs Admin, Enterprise controls, Skill sharing (off by default); only a Skill in the person's own space can be opened. - A person's own browser runs only on a computer of its own: the server and the computer both refuse it on a computer Bots share. Privacy and consent: - The disclosure says everything collected and where it goes, and what is masked, before the first Connect. - Narration is off until the person chooses, and on-device recognition only. - Every recorded tab shows a recording mark from its first step, with one click to leave a site out; a page only looked at is held while the person stays on its site, so nothing of a left-out site is sent, its address included. - Recorder connections end at sign-out and after going unused for the session lifetime; kept recordings for another person are deleted unsent after 7 days; uninstall on Windows and Linux removes the profile and the native host manifests. Should fix and nits: - Every refusal writes its audit row, and connections and revocations are audited in the same transaction; concurrent exchanges take turns; mintCode is atomic. - A test fails CI when the @copilotkit/learning patch stops applying. - Windows asks Chrome to close before forcing it; idle cloud person browsers close; Linux finds Chrome and Edge on the PATH; the Recorder ships prebuilt. - Settings, Browsers: the store install action, Disconnect asks first, each row has its own pending state; the tab strip follows the WAI-ARIA tabs pattern. Found in end-to-end runs: - The live screen takes keys only while it has focus, and forwarded keys carry no native key code, so no key reaches Chrome's own shortcuts. - A browser that stops answering gets a plain sentence within seconds, and every browser request has a per-request timeout covering its own waits. - Opening the browser keeps a live, valid Recorder connection instead of minting a new one each time. - Cancelling or closing the Connect page is reported back at once. - The CI fixes for anthropic 2.x and the Chromium build CI installs.
…the language, not one ICU's
…ything it reaches recorder-browsers.test.tsx timed out on GitHub's runner (8.8 s) while it took about 3 s here. Its waitFor checks expect(dialog).toBeNull() while the dialog is still open, and Bun printed the failed matcher's element by walking everything it reached: happy-dom's window and document, and the React fiber and props on every rendered element. That walk is synchronous, so the event loop stood still for 3.3 s here, and the fake DELETE the test had just released could not answer until it was over. The app tests' preload now gives happy-dom's Node a Bun.inspect.custom that prints its markup, cut short, which Bun's printer asks for first; Node is one class for every window a file registers, so it holds in every file. The test now takes 13 ms, and a failed check on an element prints the element. Not a product delay: the app's dialog and request ran in milliseconds throughout.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the OpenBot Recorder and the OpenBot Browser into OpenBot. Both were built and reviewed in a private repository as two stacked pull requests; this lands them here as one, on top of current main.
The Recorder. A Chrome extension a person uses to teach a Bot by doing: they press Record, do a task in their own browser while saying what they are doing, and the clicks, typing, pages, network calls and narration go straight into their own Learning space in Intelligence as a trajectory, where Learning turns it into a skill a Bot can use. Settings → Browsers connects a browser; each person gets their own Learning space and chooses how their skills are shared.
The OpenBot Browser. A button in OpenBot opens a real Chrome window the person uses as their own, with the Recorder already installed and connected, and the Browser page records a named task in it. It records the person, never the Bot. OpenBot Desktop opens and connects it on macOS, Windows and Linux.
Before upgrading. Two migrations add the Recorder's tables, after main's 0065:
0066_recorder_extension: six new tables (recorder_connect_codes,recorder_connections,recorder_trajectory_bindings,learning_person_spaces,learning_skill_sharing,learning_open_skills). Nothing existing changes.0067_recorder_activity: two nullable columns onrecorder_connections.The CopilotKit runtime moves to 1.79.1, with
COPILOTKIT_OPENAI_APIchoosing the API an OpenAI-compatible endpoint is called with; the CHANGELOG's Unreleased section has every item.Reviewing. Most of the size is the extension (its source, and recorded agent traffic it is tested against in
extension/tests/fixtures) and the two drizzle snapshots. The server's part is the Recorder and Browser routes, connect codes and the per-person Learning spaces; the app's part is Settings → Browsers and the Browser page.Checked before publishing. No keys or tokens in the added lines, no customer names or internal links in the code or the commit messages, every dependency public (including the patched
@copilotkit/learning@0.0.1), and no new CI secret. The extension's signing key is not in the repository.