Locators inside a WebView, and find answering from the page (CHALLENGES 312)

docs/CHALLENGES.md

@@ -6690,6 +6690,29 @@ for an element that is there but off screen times out saying so and naming
 Table Demo: Item 25 at y 5307, `--for visible` refused "off screen",
 `--for hidden` passed, and after `scroll-to` visible passed.
 
+### 312. `find` inside a WebView answered from the native screen
+
+**Found by:** giving WebViews locators (ROADMAP, locators item 4) and
+checking them on MobiumApp's Actionability and Frames pages, each tap
+judged by what the page said it reached. With a WebView's context
+selected, every action named the page's element, but `find role=button`
+listed the app's own Back button first and then the page's buttons as
+the platform's accessibility saw them: `app_find` never asked which
+context was selected, read the native screen, and gave its refs. The
+inspector's Find goes through `app_find`, so its count of matches was
+the native screen's too, and a ref find handed out was not the one `map`
+would. **Fixed:** in a WebView `find` matches the page's elements by the
+rules an action uses and gives each the ref `map` gives it; a ref from
+`find` reached the same button on the emulator and the simulator.
+
+Two refusals there said less than they knew. Three "Tap me" buttons, one
+per frame, were refused as `"Tap me" (button)` three times, and now each
+names its frame. And `css=#sameBtn`, a selector written for a frame's own
+document, was "no element … app_map lists what it has". The button is
+there, in `iframe#sameFrame`. A selector without its frame names only
+the page's own elements, so the refusal, and `find`'s answer, now give
+the path that reaches it: `css=iframe#sameFrame >> #sameBtn`.
+
 ## Findings that were not defects
 
 Worth recording because each one closed off an approach that looked obvious.

docs/ROADMAP.md

@@ -544,10 +544,14 @@ this is what is not.
 
   All three are asserted by `docs/checks/locators.sh` on MobiumApp's Table
   Demo, on both platforms.
-  4. **CSS selectors, and text, inside a WebView**, where only refs work
-     today: CSS is the page's own way to name an element, `map` already
-     derives a selector for each, and matching would go through the same
-     checks.
+  4. ~~**CSS selectors, and text, inside a WebView**~~ Done 2026-10-12:
+     `css=` with an optional frame path, and text, label, testid, role and
+     class matched against the page map, quotes and `,nth=` included; the
+     element found goes through every web action's checks, and `find`
+     answers from the page (CHALLENGES 312). Checked on MobiumApp's
+     Actionability and Frames pages on the Android 15 emulator and the
+     iPhone 17 Pro simulator, cross-origin frames included. `wait` inside
+     a WebView is still refused — the next step for a test file in a page.
   5. **An XPath over the platform's own tree, only once an app needs
      it** — evaluated by Mobium, so the checks still apply, and refused on
      a platform it was not written for. XPath is what the neutral kinds

docs/guides/cli.md

@@ -150,6 +150,15 @@ not move with rows kept off screen; past the last, nothing matches. It comes
 last, after any `,role=`. It is brittle, since a row added above moves it:
 prefer `within` where a row says which.
 
+Inside a WebView, after `mobium context WEBVIEW_…`, a locator matches what
+`map` reads of the page: `text=` and `label=` an element's accessible name,
+`testid=` its `data-testid` or `id`, `role=` its role — quotes and `,nth=` as
+above, counted down the page. `css=` takes a CSS selector: `mobium tap
+'css=button.save'`, with a frame's path before it as `map` prints one,
+`'css=iframe#f >> button'`. Only what `map` lists can be named, since that is
+what an action can touch. `within` is for native screens; a selector says
+what is inside what.
+
 A locator that matches more than one element is refused, never guessed at;
 append a role — `label=Apps,role=button` — or use a ref. Every action waits
 for its target to be ready before it touches it — [auto-wait](autowait.md) is

internal/agent/handlers.go

@@ -918,14 +918,18 @@ func (h *Handlers) text(ctx context.Context, args map[string]interface{}) (*Tool
 }
 
 func (h *Handlers) find(ctx context.Context, args map[string]interface{}) (*ToolsCallResult, error) {
-	dev, driver, err := h.session(ctx, args)
+	s, err := h.sessionFor(ctx, args)
 	if err != nil {
 		return nil, err
 	}
+	dev, driver := s.dev, s.driver
 	raw := stringArg(args, "locator")
 	if raw == "" {
 		return nil, mobiumerr.New(mobiumerr.InvalidArgument, "app_find needs a locator, e.g. \"text=Sign In\"")
 	}
+	if s.web != nil {
+		return h.findInWeb(ctx, s, raw)
+	}
 	loc, err := uitree.ParseLocator(raw)
 	if err != nil {
 		return nil, err

internal/agent/schema.go

@@ -357,7 +357,10 @@ func GetToolSchemas() []Tool {
 			Description: "Switch between the native shell and a WebView, or report the current " +
 				"context when called with no argument. While a WebView context is active, " +
 				"app_map and app_tap operate on the page's elements; taps are still delivered " +
-				"as real touches on the device.",
+				"as real touches on the device. A locator there matches what app_map reads of " +
+				"the page — text and label an element's accessible name, testid its " +
+				"data-testid or id, role its role — and css= takes a CSS selector, with a " +
+				"frame's path before it as app_map prints one (\"iframe#f >> button\").",
 			InputSchema: map[string]interface{}{
 				"type": "object",
 				"properties": withDevice(map[string]interface{}{

internal/agent/weblocator_test.go

@@ -0,0 +1,85 @@
+package agent
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/mobiumdev/mobium/internal/mobiumerr"
+	"github.com/mobiumdev/mobium/internal/webview"
+)
+
+// A locator inside a WebView matches what the page map reads of each
+// element, by a native locator's rules — it was refused outright, so a test
+// file, which has no refs, could not touch a page.
+func TestALocatorFindsAPageElement(t *testing.T) {
+	els := []webview.Element{
+		{Tag: "button", Label: "Save", X: 10, Y: 20, W: 100, H: 20},
+		{Tag: "button", Label: "Save draft", X: 10, Y: 60, W: 100, H: 20},
+		{Tag: "a", Label: "Learn more", TestID: "more", X: 10, Y: 100, W: 100, H: 20},
+		{Tag: "button", Label: "Delete", X: 10, Y: 140, W: 100, H: 20},
+		{Tag: "button", Label: "Delete", X: 10, Y: 180, W: 100, H: 20},
+	}
+	h := NewHandlers()
+	s := &session{dev: fakeDevice(), driver: &hostDriver{}, backend: BackendUIA2,
+		web: &layoutPage{els: els}, webCtx: "WEBVIEW_test"}
+	find := func(target string) (webview.Element, error) {
+		_, e, _, err := h.findWeb(context.Background(), s, target)
+		return e, err
+	}
+	for target, want := range map[string]string{
+		"text=Save":             "Save", // the whole match among a part
+		`text="Save draft"`:     "Save draft",
+		"testid=more":           "Learn more",
+		"role=link":             "Learn more",
+		"text=Delete,nth=2":     "Delete",
+		"label=Learn,role=link": "Learn more",
+	} {
+		e, err := find(target)
+		if err != nil || e.Label != want {
+			t.Errorf("%s: got %q, %v; want %q", target, e.Label, err, want)
+		}
+	}
+	if e, _ := find("text=Delete,nth=2"); e.Y != 180 {
+		t.Errorf("nth=2 counted down the page to y %v, want 180", e.Y)
+	}
+	if _, err := find("text=Delete"); mobiumerr.CodeOf(err) != mobiumerr.AmbiguousLocator || !strings.Contains(err.Error(), "2 elements") {
+		t.Errorf("two Deletes: %v", err)
+	}
+	if _, err := find("text=Nowhere"); mobiumerr.CodeOf(err) != mobiumerr.NoSuchElement {
+		t.Errorf("nothing: %v", err)
+	}
+	if _, err := find("label=Delete within text=Save"); mobiumerr.CodeOf(err) != mobiumerr.Unsupported ||
+		!strings.Contains(err.Error(), "css=") {
+		t.Errorf("within in a page: %v", err)
+	}
+}
+
+// Look-alikes in frames are told apart by their frame in a refusal, and a
+// selector written for a frame's own document, without the frame, is
+// answered with the path that reaches what it matched — MobiumApp's
+// three frames each hold a "Tap me".
+func TestAFramedMatchIsNamedByItsFrame(t *testing.T) {
+	els := []webview.Element{
+		{Tag: "button", Label: "Tap me", Frame: "iframe#a", X: 10, Y: 20, W: 100, H: 20},
+		{Tag: "button", Label: "Tap me", Frame: "iframe#a >> iframe#b", X: 10, Y: 60, W: 100, H: 20},
+	}
+	h := NewHandlers()
+	s := &session{dev: fakeDevice(), driver: &hostDriver{}, backend: BackendUIA2,
+		web: &layoutPage{els: els}, webCtx: "WEBVIEW_test"}
+	_, _, _, err := h.findWeb(context.Background(), s, `text="Tap me"`)
+	if mobiumerr.CodeOf(err) != mobiumerr.AmbiguousLocator ||
+		!strings.Contains(err.Error(), `"Tap me" (button) in iframe#a >> iframe#b`) {
+		t.Errorf("two framed look-alikes: %v", err)
+	}
+	got := framedHint(els, []int{1}, "#tap")
+	if !strings.Contains(got, "css=iframe#a >> iframe#b >> #tap") {
+		t.Errorf("a selector without its frame: %q", got)
+	}
+	if got := framedHint(els, []int{0}, "iframe#x >> #tap"); !strings.Contains(got, "css=iframe#a >> #tap") {
+		t.Errorf("a selector with the wrong frame: %q", got)
+	}
+	if framedHint(els, nil, "#tap") != "" {
+		t.Error("a hint with nothing elsewhere")
+	}
+}