fix: answers from the question card reach Claude Code, and stay in the card when they can't

.claude/skills/verify-berthd/SKILL.md

@@ -5,7 +5,7 @@ description: Prove a change to the Shipyard app works the way a person sees it -
 
 # Verify the Shipyard app
 
-The app (`app/`) is what people touch. Its production build runs in a browser on the mock fixtures (`?mock=1`, `src/lib/mock*.ts`, synthetic "acme" data), with no laptop agent and no box. That is the surface this skill drives. The Go daemons (`berthd`, `berth`) are verified by `go test -race ./...`; a box-side change also needs a run on a real box, which this skill doesn't cover.
+The app (`app/`) is what people touch. Its production build runs in a browser on the mock fixtures (`?mock=1`, `src/lib/mock*.ts`, synthetic "acme" data), with no laptop agent and no box. That is the surface this skill drives. The Go daemons (`berthd`, `berth`) are verified by `go test -race ./...`; a box-side change also needs a run on a real box: `scripts/dev-box.sh up` starts a box and an agent of this checkout beside whatever Shipyard already runs on the machine, and `drive.mjs --agent` drives the same build against it (see [features/chat-questions.md](features/chat-questions.md)).
 
 The e2e suite (`app/e2e`, see `.claude/skills/e2e-tests`) asserts. This skill **shows**: it drives the same build through a scenario and keeps the evidence for a PR or a person.
 
@@ -37,7 +37,7 @@ Read only. It checks the port answers with the app, that the PID in `/tmp/verify
 node .claude/skills/verify-berthd/scripts/drive.mjs <scenario> --port $PORT [--out DIR]
 ```
 
-`<scenario>` is a file in `scripts/scenarios/` (by name, without `.mjs`) or a path. The driver:
+`<scenario>` is a file in `scripts/scenarios/` (by name, without `.mjs`) or a path. With `--agent http://127.0.0.1:9378` and `VERIFY_BERTH_TOKEN` in the environment it opens the app on that laptop agent instead of the mock (a scenario that needs it exports `live = true`). The driver:
 
 - opens `http://localhost:$PORT/?mock=1` at 1440×900 in dark mode, and waits until the sidebar's `nav-home` is there;
 - refuses every request to the laptop agent's ports (1377-1379), as the e2e fixtures do;

.claude/skills/verify-berthd/features/README.md

@@ -15,8 +15,9 @@ The maintained list of what a person can do in the app, and how to prove each on
 | Sidebar places: Pinned, More, Hidden, counts on rows (Home's waiting, Review's, a plugin's) | [sidebar-nav.md](sidebar-nav.md) | `plugin-nav-badges` |
 | Plugins: built-in and your own, what they add (screens, sidebar items, widgets, panels) | [plugins.md](plugins.md) | `plugin-nav-badges` (sidebar items) |
 | Home's widget grid | [home-widgets.md](home-widgets.md) | none yet; `app/e2e/home-widgets.spec.ts` asserts it |
+| Questions in chat: an agent's form of questions as a card, answered on a real box; kept with Retry and Copy when it can't be | [chat-questions.md](chat-questions.md) | `chat-questions-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` on the mock |
 
 ## Proof and skip reporting
 
 - Every proof names the feature file and the scenario it ran, and attaches the screenshots from the evidence folder.
-- Something the mock can't show (a real box, the Mac app's window and updater, a plugin's helper on a box) is reported as not verified here, with what would verify it.
+- Something the mock can't show (the Mac app's window and updater, a plugin's helper on a box) is reported as not verified here, with what would verify it. A real box and agent of this checkout run on the same machine with `../scripts/dev-box.sh` (see [chat-questions.md](chat-questions.md)).

.claude/skills/verify-berthd/features/chat-questions.md

@@ -0,0 +1,40 @@
+# Questions in chat
+
+When an agent asks with a form of its own (Claude Code's AskUserQuestion: up to four questions, each with options, some taking several picks, each with room for the person's own words), the chat draws it as a card: a step per question, then a review with **Submit answers**. The box fills in the agent's own form with keys, checking its screen after every key (`internal/box/answer.go`, POST `/v1/sessions/{name}/answer`). The card is `app/src/components/conversation/question-form.tsx`.
+
+## Sub-features
+
+- `questions-card`: the card's steps, the person's own words, the review, Submit; once answered it folds to what was answered (`app/e2e/chat.spec.ts`, mock).
+- `questions-deliver`: on a real box, long own words (wrapped by Claude Code), questions with lists in them, on an agent nobody has open (its pane at tmux's 80x24): every answer arrives word for word and the agent goes on. The box makes the pane tall enough while it answers and puts it back.
+- `questions-kept`: when the box can't fill the form in, the card keeps every answer and says why, with **Retry** and **Copy answers** (each answer on a line under its question's name), and the agent's own screen opens under it to finish there. A retry that goes closes both.
+- `questions-log`: every answer that doesn't go is in the box's log with why, and the screen it stopped at.
+
+## How to get to it (user POV)
+
+- Open an agent in chat and have it ask (or ask it to call AskUserQuestion). The card is in the conversation, and the reply box points at it.
+
+## Driving it with drive.mjs
+
+- Mock: `app/e2e/chat.spec.ts` covers the card and, with `?scene=answer-stuck`, a first submit that fails then a Retry.
+- Real box: `scenarios/chat-questions-live.mjs`, on a dev box of this checkout (`../scripts/dev-box.sh`):
+
+```sh
+.claude/skills/verify-berthd/scripts/dev-box.sh up
+eval "$(.claude/skills/verify-berthd/scripts/dev-box.sh env)"   # the token stays in env
+# An agent nobody has open sits at 80x24:
+tmux -L berth resize-window -t ask -x 80 -y 24 && tmux -L berth set-option -w -u -t ask window-size
+node .claude/skills/verify-berthd/scripts/drive.mjs chat-questions-live --port $PORT --agent http://127.0.0.1:9378
+# The failure path: Claude's own-words row holds something else when the box gets there.
+VERIFY_SURPRISE="tmux -L berth send-keys -t ask Down Down Down && sleep 0.6 && tmux -L berth send-keys -t ask -l 'typed in the terminal' && sleep 0.8" \
+VERIFY_UNSURPRISE="tmux -L berth send-keys -t ask -N 21 BSpace && sleep 0.8" \
+node .claude/skills/verify-berthd/scripts/drive.mjs chat-questions-live --port $PORT --agent http://127.0.0.1:9378
+.claude/skills/verify-berthd/scripts/dev-box.sh down
+```
+
+The preview must be opened as `http://localhost:<1420-1439>`: the agent's API only takes those origins. Second view: Claude's own record (`~/.claude/projects/*acme-shop*/*.jsonl`) has the tool result with every answer, and the dev box's log (`/tmp/verify-berthd-box/berthd.log`) has `answer ask: answered 3 questions`, or why not with the screen.
+
+## Gotchas
+
+- The scenario sends its prompt from the chat, so the session must be at its prompt: one left at a form from a failed run needs Esc first.
+- A box started from the app starts agents structured (Claude through its SDK, which answers through a hook, not keys); `dev-box.sh` sets the box's defaults so they start in their terminals, the path this covers.
+- `dev-box.sh up` refuses if the dev box rewrote the agent hooks in your home (it does that only when they are older than its own); put them back from your backup before going on.

.claude/skills/verify-berthd/scripts/dev-box.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# dev-box.sh up|env|down: a box and a laptop agent of this checkout's
+# build, on this machine, apart from the Shipyard already running here:
+# their own state, tmux server and ports. `up` builds bin/, starts berthd
+# on 127.0.0.1:7544 and the agent on 9377/9378 (never 1377-1379), pairs
+# them as box "devq", adds a synthetic repo "acme-shop" and starts Claude
+# Code in its terminal there, session "ask". `env` prints what drive.mjs
+# and tmux need; `down` stops it all and deletes the state.
+#
+# HOME stays real: Claude Code needs its sign-in. The box refreshes
+# Claude's hooks in ~ only when they are out of date; `up` stops if it
+# did, so a dev box never changes the real one's hooks unnoticed.
+set -euo pipefail
+root=$(git -C "$(dirname "$0")" rev-parse --show-toplevel)
+# Short: a unix socket's path can't be longer than 108 bytes.
+dir=${VERIFY_BOX_DIR:-/tmp/verify-berthd-box}
+box_env=(BERTH_HOME="$dir/box-home" BERTH_USER_DIR="$dir/box-user" TMUX_TMPDIR="$dir/tmux")
+lap_env=(BERTH_HOME="$dir/lap-home" BERTH_USER_DIR="$dir/lap-user" BERTH_UI_ADDR=127.0.0.1:9378 BERTH_PROXY_ADDR=127.0.0.1:9377)
+box() { env "${box_env[@]}" "$root/bin/berthd" "$@"; }
+lap() { env "${lap_env[@]}" "$root/bin/berth" "$@"; }
+tm() { env TMUX_TMPDIR="$dir/tmux" tmux -L berth "$@"; }
+
+case ${1:-} in
+up)
+  for p in 7544 9377 9378; do
+    if ss -ltn | grep -q ":$p "; then echo "port $p is taken: is a dev box up already? ($0 down)" >&2; exit 1; fi
+  done
+  mkdir -p "$dir"/{box-home,box-user,lap-home,lap-user,tmux} && chmod 700 "$dir/tmux"
+  make -C "$root" build >/dev/null
+  env "${box_env[@]}" nohup "$root/bin/berthd" serve --listen 127.0.0.1:7544 >"$dir/berthd.log" 2>&1 &
+  echo $! >"$dir/berthd.pid"
+  until grep -q "berthd serving" "$dir/berthd.log"; do sleep 0.5; done
+  if grep -q "updated berth's hooks" "$dir/berthd.log"; then
+    echo "the dev box rewrote agent hooks in $HOME (berthd.log): put them back before going on" >&2
+    exit 1
+  fi
+  lap pair "$(box pair --address 127.0.0.1:7544 --json | jq -r .link)" --name devq >/dev/null
+  token=$(lap ui-token | jq -r .token)
+  # Agents in their terminals, as the bug reports' sessions run.
+  curl -sf -X PUT -H "Authorization: Bearer $token" -H 'content-type: application/json' \
+    http://127.0.0.1:9378/v1/boxes/devq/api/agents/defaults -d '{"structured":false,"mode":"bypassPermissions"}' >/dev/null
+  repo=$dir/acme-shop
+  if [[ ! -d $repo/.git ]]; then
+    git init -q -b main "$repo" && echo "# acme shop" >"$repo/README.md"
+    git -C "$repo" add . && git -C "$repo" -c user.email=dev@acme.test -c user.name=acme commit -qm init
+  fi
+  box location add acme-shop "$repo" >/dev/null
+  box session new acme-shop --name ask -- claude --dangerously-skip-permissions >/dev/null
+  # A new folder asks to be trusted first.
+  sleep 8
+  tm send-keys -t ask Up && sleep 0.5 && tm send-keys -t ask Enter
+  echo "up: box devq (127.0.0.1:7544), agent http://127.0.0.1:9378, worktree devq/acme-shop, session ask"
+  ;;
+env)
+  # VERIFY_BERTH_TOKEN is a secret: export it, don't print it anywhere else.
+  echo "export VERIFY_BERTH_TOKEN=$(lap ui-token | jq -r .token) VERIFY_WORKTREE=devq/acme-shop TMUX_TMPDIR=$dir/tmux"
+  ;;
+down)
+  [[ -f $dir/berthd.pid ]] && kill "$(cat "$dir/berthd.pid")" 2>/dev/null || true
+  tm kill-server 2>/dev/null || true
+  # The agent the CLI started: the one on 9378 with this state, no other.
+  for pid in $(ss -ltnpH "sport = :9378" 2>/dev/null | grep -o 'pid=[0-9]*' | cut -d= -f2 | sort -u); do
+    tr '\0' '\n' <"/proc/$pid/environ" | grep -qx "BERTH_HOME=$dir/lap-home" && kill "$pid"
+  done
+  rm -rf "$dir"
+  echo "down: $dir removed"
+  ;;
+*)
+  echo "usage: dev-box.sh up|env|down" >&2
+  exit 2
+  ;;
+esac

.claude/skills/verify-berthd/scripts/drive.mjs

@@ -1,7 +1,9 @@
 #!/usr/bin/env node
-// drive.mjs <scenario> --port N [--out DIR]: drive the app's preview build
-// on its mock fixtures through a scenario (scenarios/*.mjs), with the
-// scenario's test plugins loaded as built-ins, and keep the evidence.
+// drive.mjs <scenario> --port N [--out DIR] [--agent URL]: drive the
+// app's preview build on its mock fixtures through a scenario
+// (scenarios/*.mjs), with the scenario's test plugins loaded as built-ins,
+// and keep the evidence. --agent drives a real laptop agent instead (a dev
+// one, never 1377-1379), with its UI token in VERIFY_BERTH_TOKEN.
 // See ../SKILL.md.
 import { mkdirSync, writeFileSync } from "node:fs";
 import { createRequire } from "node:module";
@@ -20,12 +22,18 @@ const flag = (name) => {
 };
 const name = args.find((a, i) => !a.startsWith("--") && !args[i - 1]?.startsWith("--"));
 const port = flag("--port");
-if (!name || !port) {
-  console.error("usage: drive.mjs <scenario> --port N [--out DIR]");
+const agent = flag("--agent");
+const token = process.env.VERIFY_BERTH_TOKEN;
+if (!name || !port || (agent && !token)) {
+  console.error("usage: drive.mjs <scenario> --port N [--out DIR] [--agent URL, with VERIFY_BERTH_TOKEN set]");
   process.exit(2);
 }
 const file = name.endsWith(".mjs") || name.includes("/") ? resolve(name) : join(here, "scenarios", `${name}.mjs`);
 const scenario = await import(pathToFileURL(isAbsolute(file) ? file : resolve(file)).href);
+if (scenario.live && !agent) {
+  console.error(`${name} drives a real box: give it --agent (see ../features/chat-questions.md)`);
+  process.exit(2);
+}
 const stamp = new Date().toISOString().replace(/[:.]/g, "-");
 const out = resolve(flag("--out") ?? `/tmp/verify-berthd/${name.replace(/.*\//, "").replace(/\.mjs$/, "")}-${stamp}`);
 mkdirSync(out, { recursive: true });
@@ -34,23 +42,28 @@ const log = [];
 const steps = [];
 const failures = [];
 const allow = scenario.allowErrors ?? [];
+// The live agent's token is in the page's address: a browser build takes
+// it only from ?token= (lib/api.ts endpoint, as e2e/live.mjs does). It is
+// a dev agent's on loopback (dev-box.sh), and nothing the driver writes
+// keeps it.
+const redact = (text) => (token ? String(text).replaceAll(token, "<token>") : String(text));
 const step = (text) => {
   steps.push(text);
   console.log(`- ${text}`);
 };
 
 const browser = await chromium.launch();
 const context = await browser.newContext({ viewport: { width: 1440, height: 900 }, colorScheme: "dark" });
-// The laptop agent's ports: a mock run must never reach them.
+// The laptop agent's ports: a run must never reach this machine's own.
 await context.route(/^https?:\/\/(?:[^/]*\.)?(?:localhost|127\.0\.0\.1|\[::1\]):(?:1377|1378|1379)(?:\/|$)/, (r) => r.abort());
 const page = await context.newPage();
 page.on("console", (m) => {
-  log.push(`[${m.type()}] ${m.text()}`);
-  if (m.type() === "error" && !allow.some((re) => re.test(m.text()))) failures.push(`console error: ${m.text()}`);
+  log.push(redact(`[${m.type()}] ${m.text()}`));
+  if (m.type() === "error" && !allow.some((re) => re.test(m.text()))) failures.push(redact(`console error: ${m.text()}`));
 });
 page.on("pageerror", (e) => {
-  log.push(`[pageerror] ${e.message}`);
-  failures.push(`page error: ${e.message}`);
+  log.push(redact(`[pageerror] ${e.message}`));
+  failures.push(redact(`page error: ${e.message}`));
 });
 
 const plugins = scenario.plugins ?? [];
@@ -65,19 +78,22 @@ if (plugins.length) {
 
 const shot = async (label) => {
   await page.screenshot({ path: join(out, `${label}.png`) });
-  writeFileSync(join(out, `${label}.aria.txt`), await page.locator("body").ariaSnapshot());
+  writeFileSync(join(out, `${label}.aria.txt`), redact(await page.locator("body").ariaSnapshot()));
   console.log(`  saved ${label}.png`);
 };
 
 let ok = true;
 try {
-  step(`open http://localhost:${port}/?mock=1`);
-  await page.goto(`http://localhost:${port}/?mock=1`);
+  const params = new URLSearchParams(agent ? { token, agent } : { mock: "1" });
+  for (const [k, v] of Object.entries(scenario.params ?? {})) params.set(k, v);
+  // The token is in the page's address, as the app takes it; never in the log.
+  step(agent ? `open http://localhost:${port}/ on the agent at ${agent}` : `open http://localhost:${port}/?${params}`);
+  await page.goto(`http://localhost:${port}/?${params}`);
   await expect(page.getByTestId("nav-home")).toBeVisible({ timeout: 20_000 });
   await scenario.run({ page, shot, step, expect });
 } catch (err) {
   ok = false;
-  failures.push(`scenario failed: ${err.message?.split("\n").slice(0, 6).join("\n")}`);
+  failures.push(redact(`scenario failed: ${err.message?.split("\n").slice(0, 6).join("\n")}`));
   await page.screenshot({ path: join(out, "failed.png") }).catch(() => {});
 }
 await browser.close();