From 0b144354bb6ca680af10b8a6a985653225a74357 Mon Sep 17 00:00:00 2001
From: pras-ops <prashantjacob.bilaspur@gmail.com>
Date: Wed, 7 Oct 2026 09:28:06 +0530
Subject: [PATCH] docs: add competitive research, workflow analysis and product
 direction

Snapshot of the UI/UX testing product (Studio, sandbox, checks, recorder) before the repository is narrowed to the scraper. This branch is the source for the separate testing repository.
---
 COMPETITIVE_RESEARCH.md      | 237 +++++++++++++++++++++++++++++
 PRODUCT_DIRECTION.md         | 234 ++++++++++++++++++++++++++++
 PRODUCT_WORKFLOW_ANALYSIS.md | 287 +++++++++++++++++++++++++++++++++++
 3 files changed, 758 insertions(+)
 create mode 100644 COMPETITIVE_RESEARCH.md
 create mode 100644 PRODUCT_DIRECTION.md
 create mode 100644 PRODUCT_WORKFLOW_ANALYSIS.md

diff --git a/COMPETITIVE_RESEARCH.md b/COMPETITIVE_RESEARCH.md
new file mode 100644
index 0000000..a7dc1fe
--- /dev/null
+++ b/COMPETITIVE_RESEARCH.md
@@ -0,0 +1,237 @@
+# 🔍 Competitive & Community Research — How to Make ScrapeWizard Better
+
+> Deep-dive conducted 2026-09-30 across GitHub, Hacker News, Ministry of Testing (The Club),
+> official Playwright docs, vendor docs, and practitioner reviews. Findings are checked against
+> the **current** codebase (post-sync, commit `6003e10`).
+>
+> ⚠️ **Reddit could not be accessed** — reddit.com blocks automated fetching. Ministry of Testing
+> and Hacker News were used as the practitioner-sentiment substitute. Recommended manual check:
+> r/QualityAssurance and r/softwaretesting, searching "self-healing", "Playwright healer", "mabl".
+
+---
+
+## 1. TL;DR — the five findings that matter most
+
+1. **Playwright now ships its own "Healer" agent (free, official).** `npx playwright init-agents`
+   gives Planner / Generator / Healer. Microsoft cites roughly **75%+ success on selector
+   failures**. It needs an AI loop (VS Code, Claude Code, Codex or OpenCode) and **edits your
+   source files**. This is now the baseline every self-healing tool gets compared against.
+2. **A near-identical open-source competitor exists: `qa-core-heal`.** It's a deterministic,
+   no-LLM healer that proposes diffs you approve, then re-runs to verify. Its published evals:
+   **53/53 healable breaks fixed, 21/21 unhealable refused, 0 wrong heals, deterministic across
+   runs.** That number is now the credibility bar for "safe healing." (It's tiny: 6 stars, CLI
+   only, with no recorder and no dashboard. Those are your openings.)
+3. **Practitioners distrust "self-healing" as a marketing term.** The Ministry of Testing thread
+   calls these tools mostly "marketing/propaganda", and one tester sums it up as *"flaky is still
+   flaky."* Their number-one fear is a **false heal that masks a real bug**: the tool clicks a
+   similar-but-wrong element and the test goes green. Trust comes from **refusal, audit logs and
+   human review**, not from heal rates.
+4. **Locators are not the main cause of flakiness.** Async **timing accounts for about 45%** of
+   flakiness (ContextQA root-cause analysis). Healing only fixes one slice of failures, so a tool
+   that **classifies why a test failed** before touching it beats one that heals blindly.
+5. **The market has settled on "AI plans once, then cheap deterministic replay, re-plan only on
+   change."** Stagehand (action cache keyed by page fingerprint), Midscene (plan + locate cache)
+   and Magnitude (planner/executor split with plan caching) all do this. **Your architecture
+   already is this pattern.** Say so explicitly and use the fingerprints you already capture as
+   the cache key.
+
+---
+
+## 2. Landscape map
+
+| Tool | Type | Approach | Relevance to ScrapeWizard |
+|---|---|---|---|
+| **Playwright Test Agents** (Planner/Generator/Healer) | Free, official | LLM agents via an IDE loop; Healer replays, inspects UI, patches source, re-runs | **Primary threat.** Free and built-in. Reviewers say the Healer is useful, the Generator is *worse than Codegen*, and the Planner hallucinates steps |
+| **Playwright Codegen** | Free, official | Recorder with an **assertion toolbar** (visibility/text/value), a **locator picker**, and role → text/label → testid priority | **Your recorder's real benchmark.** If yours records worse locators than free Codegen, the pitch collapses |
+| **qa-core-heal** | OSS (MIT, TS) | Deterministic ladder (role→label→placeholder→text→alt→title→testid→CSS/XPath), kind guard, named refusals, diff approval, verify+revert, `heal-log.jsonl`, published evals | **Closest competitor to your healing pitch.** Copy its safety discipline; beat it on recorder, portal and evidence |
+| **Healenium** (EPAM) | OSS | Runtime healing via tree-comparison / **LCS** over the stored DOM; Postgres backend stores locator history plus screenshots | The proven "history-based" model. You already store fingerprint history, so lean on it |
+| **Stagehand** (Browserbase) | OSS | NL actions, **action cache keyed by page fingerprint**: replay if it matches, fall back to AI and update the cache if not | Validates your "AI once" economics. Their cache-key idea maps directly onto your fingerprints |
+| **Midscene.js** (ByteDance) | OSS | Vision-model automation; caches plans and matched elements (51s → 28s on a cache hit) | Same pattern. Their visual replay report is a UX reference |
+| **Magnitude** | OSS | Planner LLM plus a tiny 2B vision executor (Moondream); cached plans | HN critique: pure vision "lets them off the hook" on accessibility. **Your built-in a11y checks are a counter-position** |
+| **Stagehand / Browser Use / Skyvern / AgentQL** | OSS agents | General browser agents | Adjacent, not direct competitors. Agents are "slow, expensive, and inconsistent" for regression testing (HN) |
+| **mabl / Testim / Functionize / Tosca** | Commercial | Codeless plus AI maintenance | Roughly **$450–500+/month**, quote-only pricing, annual contracts, per-user pricing, lock-in (G2). Your local/free/no-lock-in angle is valid |
+| **QA Wolf / Octomind / Momentic / Checksum / Meticulous** | Commercial (AI QA) | Managed or hosted generation plus maintenance | QA Wolf found a single agent "overwhelmed" and moved to a **multi-agent triage that separates intentional change / flake / real bug**, with **history** as the key input |
+| **Currents / ReportPortal / Allure** | Reporting | Run history, **flaky detection**, error grouping, trace/video | **Your portal competes here.** Currents is paid and cloud-only; Allure is static with no history; ReportPortal is heavy. A local dashboard with history and flaky detection is a real gap |
+| **Argos / Lost Pixel / BackstopJS / Percy** | Visual | Baselines, masking, review UI | The bar for your visual check (see §5) |
+
+---
+
+## 3. What practitioners actually say
+
+- **Skepticism is the default.** "None of these fancy expensive tools are as good as they try to
+  make them seem" (MoT). Self-healing helps mainly where "IDs are dynamic or locators are too
+  hard to define."
+- **Masking bugs is the core objection.** The recommended safeguards (MoT insights, SD Times):
+  - Heal **locators only, never assertions.**
+  - A healed locator must resolve to **exactly one** displayed, actionable element.
+  - Use a **high confidence threshold** for silent heals.
+  - **Log every heal:** the original locator, the new locator, the signals used and the outcome.
+  - **Fail the build if the heal rate spikes.**
+  - Offer graduated autonomy: **runtime-only → human-reviewed PR → auto-merge.**
+- **The Playwright Healer in real use** (Bondar Academy, BrowserStack, Currents): about 10 minutes
+  per fix. It sometimes **weakens assertions** (e.g. `toHaveText` → `toBeVisible`), which hides
+  problems. It "cannot prove the product is correct"; a passing re-run only supports the repair.
+  It over-edits when its scope isn't constrained.
+- **On determinism (HN, Magnitude thread):** developers want "LLMs as an intermediate workflow
+  for producing deterministic Playwright code," not LLMs driving every run.
+- **On distribution (HN):** bootstrapped AI-QA teams struggle to choose between SDETs and manual
+  testers, and between OSS, SaaS and managed service. Generic tools miss "complex,
+  product-specific test cases."
+
+---
+
+## 4. Gap analysis — ScrapeWizard vs. market standard (verified in code)
+
+| Capability | Market standard | ScrapeWizard today | Gap |
+|---|---|---|---|
+| Locator priority | Role → label → placeholder → text → alt → title → testid → CSS/XPath (Playwright docs, Codegen, qa-core-heal) | testid → `tag[aria-label]` CSS → classes → XPath (`selector_engine.py`) | 🔴 **No `getByRole` / `getByLabel` / `getByText`.** Recorded locators are weaker than free Codegen |
+| Recorder events | click, fill (input), select, check, press, hover, upload, dialogs, navigation | `click` and `change` only (`recorder.py:132-142`) | 🔴 Many real flows can't be recorded (Enter key, dropdowns, checkboxes) |
+| Recorded assertions | Toolbar: assert visibility / text / value | Auto `visible` plus URL only | 🟠 Users can't express intent while recording |
+| Heal confidence + ambiguity | Threshold plus "refuse when ambiguous" | ✅ 0.85 threshold + 0.10 ambiguity margin (`healing.py`) | ✅ Good foundation |
+| Kind guard | Candidate must match the expected element kind/role | Filters by same tag only | 🟡 Add role/intent check |
+| Named refusals | Explain *why* a heal was refused | Log line only | 🟠 Surface refusal reasons in the report/UI |
+| Heal audit + approval | `heal-log`, approve-to-persist, verify, auto-revert | No persisted heal events, no approve/reject flow | 🔴 **The trust mechanism practitioners ask for is missing** |
+| Never heal assertions | Rule | Not enforced or documented | 🟠 Make it an explicit invariant with a test |
+| Failure classification | Locator / assertion / timing / network / app error / env **before** healing | None | 🔴 Healing runs without knowing *why* the step failed |
+| Published evals | Mutation suite with public numbers | Mutation tests exist (`test_demo_app_mutations.py`), no published efficacy/safety numbers | 🟠 Publish them: this is your credibility proof |
+| Debug artifacts | **Playwright trace (`trace.zip`) + Trace Viewer** | Screenshots only; no tracing | 🟠 Cheap, large win |
+| Timing robustness | Web-first auto-waiting assertions | Fixed sleeps in places (recorder `sleep(0.3)`, navigation 1s waits) | 🟠 About 45% of flakiness is timing |
+| Visual regression | Masking dynamic regions, disabled animations, per-OS baselines, anti-aliasing tolerance | Raw PIL pixel diff, 5% threshold | 🟠 Will be flaky across machines/OS |
+| Auth / secrets | `storageState`, setup login, secrets from env | Passwords masked as `***MASKED***`, so login flows can't replay | 🔴 Blocks most real apps |
+| Run history analytics | Flaky detection (pass/fail flip rate), error grouping, heal-rate trend | Runs listed; no flake/heal analytics | 🟠 Your dashboard differentiator vs Allure/Currents |
+| Export targets | Playwright **TS** (the ecosystem default), JUnit XML for CI | Python pytest only | 🟡 Add TS export + JUnit |
+
+---
+
+## 5. Recommendations (prioritized)
+
+### P0 — Credibility (without these, reviewers dismiss the tool)
+
+1. **Adopt Playwright's locator ladder in the selector engine and recorder.**
+   - Generate `role+name` (the accessible name), `label`, `placeholder`, `text`, `alt`, `title`
+     and `testid` candidates **before** CSS/XPath.
+   - Verify each candidate is **strict** (exactly one match) at record time, and chain/filter to
+     make it unique, the way Codegen does.
+   - Store the ladder in the fingerprint as it's stored today. Only the ordering and the kinds change.
+   - Acceptance: on the demo app, the primary recorded locator is `getByRole`/`getByLabel` for
+     every interactive element that has an accessible name.
+2. **Ship the safety model practitioners are asking for.**
+   - **Invariant:** heal locators only, never assertions (enforced plus a regression test).
+   - **Kind guard:** a candidate must match the expected role/tag intent.
+   - **Named refusals:** "ambiguous: 2 candidates within 0.10", "element removed", "closed
+     shadow root" and similar, shown in the run report.
+   - **HealEvent persistence:** original locator, healed locator, tier, score, signals, verified,
+     approved/rejected, reverted.
+   - **Graduated autonomy setting:** `runtime-only` (default) → `approve-to-persist` →
+     `auto-persist ≥ X`.
+   - **Heal-rate spike guard:** mark the run ⚠️ or failed if heals exceed N% of steps.
+3. **Classify failures before acting.** For each failed step, classify it as
+   `locator_not_found` / `assertion_failed` / `timeout` / `network_error` (4xx/5xx on the critical
+   request) / `app_error` (console error) / `navigation_mismatch`. **Only `locator_not_found`
+   enters healing.** Everything else is reported with evidence. (This is how QA Wolf and
+   qa-core-heal avoid masking bugs.)
+4. **Publish your eval numbers.** Extend the mutation suite to the qa-core-heal format: counts of
+   healable fixed, unhealable refused, valid locators wrongly touched, and wrong heals, plus a
+   determinism check across two runs. Put the table in the README. That's the single strongest
+   trust signal available.
+
+### P1 — Recorder parity with free Codegen
+
+5. **Record more event types.** Use `input` (not only `change`), plus `select`, `check`/uncheck,
+   `press` (Enter/Tab/Escape), hover (opt-in), file upload, dialogs, and new-tab/navigation.
+6. **Assertion recording.** Add a floating toolbar in the recorded page to assert visible / text /
+   value on a clicked element, matching Codegen's UX.
+7. **Auth that replays.** Support a login setup step or captured `storageState`; resolve masked
+   values from env/keyring at run time (`${LOGIN_PASSWORD}`). Never persist the secret.
+8. **Remove fixed sleeps.** Replace them with Playwright auto-waiting / web-first assertions
+   (`expect(locator).toBeVisible()` style). This targets the ~45% timing class.
+9. **Capture Playwright traces.** Call `context.tracing.start(screenshots=True, snapshots=True)`
+   per run, save `trace.zip` in run artifacts, and add an "Open in Trace Viewer" action in the
+   portal (`npx playwright show-trace` or trace.playwright.dev).
+
+### P1 — Visual checks that don't flake
+
+10. **Masking plus stability.** Allow mask selectors per step, inject CSS to disable animations
+    and caret, wait for fonts/images, keep baselines per OS/viewport/browser, and use an
+    anti-aliasing-aware comparator (pixelmatch-style) with a per-step threshold. In exported
+    tests, use Playwright's native `toHaveScreenshot` with `mask`.
+
+### P2 — Dashboard differentiation (vs Currents / Allure / ReportPortal)
+
+11. **Flaky detection:** a per-test flip rate across the last N runs, "flaky" badges, and optional
+    retry-on-failure that marks results as *flaky* rather than *passed*.
+12. **Error grouping:** cluster failures by normalized error plus step, so one broken button shows
+    as one issue, not 40 red runs.
+13. **Heal analytics:** heal rate over time, per-tier distribution, and pending approvals. These
+    turn "self-healing" from a claim into a visible, auditable metric.
+14. **CI outputs:** JUnit XML plus exit codes, and a **Playwright TypeScript export** alongside
+    pytest (the Playwright ecosystem is TS-first).
+
+### P2 — AI layer, positioned against Playwright's Healer
+
+15. **Make fingerprints the cache key (the Stagehand pattern).** If the page/element fingerprint
+    matches, replay deterministically. If it drifted, run deterministic tiers. Call AI only when
+    those fail, then update the ladder. This is already your design; name it and measure it
+    (cache-hit rate, AI calls per run).
+16. **Don't rebuild what Playwright gives away.** Consider "export to Playwright project + agent
+    definitions" so teams already using `init-agents` can adopt you, while your moat stays in
+    **safe deterministic healing, the recorder, and the evidence dashboard.**
+
+---
+
+## 6. Positioning — what to say (and what to stop saying)
+
+- **Stop leading with "AI self-healing."** It triggers the practitioner skepticism documented
+  above, and Playwright now gives it away.
+- **Lead with safety and evidence:** *"Record a flow. Get Playwright tests with accessible
+  locators. When the UI drifts, ScrapeWizard heals locators deterministically, **refuses rather
+  than guesses**, and logs every heal for review. Zero AI calls by default. Runs locally, no
+  subscription."*
+- **Own the differentiators competitors lack together:**
+  1. Recorder **plus** portal **plus** run history (qa-core-heal and Playwright's Healer have none of these).
+  2. A11y, console and network checks **in the same run** (vision agents skip accessibility).
+  3. Published safety evals (0 wrong heals).
+  4. Local-first with no seat pricing (vs mabl/Testim at $450–500+/month).
+- **The honest threat:** if Playwright's Healer becomes deterministic and qa-core-heal adds a
+  UI, the window narrows. Speed on P0 matters more than breadth on Wave 2 features.
+
+---
+
+## 7. How this changes the existing plans
+
+- **MARKET_READY_PLAN.md:** insert P0 items 1–4 **before** the frontend refactor phases. They
+  change the product's core value, while the UI refactor changes its polish.
+- **PLATFORM_PLAN.md §9 (healing):** add the kind guard, never-heal-assertions invariant, failure
+  classification, and graduated autonomy. Replace the "tiers 0–6" pitch language with
+  "safe healing + refusals."
+- **FRONTEND_PLAN.md:** add a Heal Review queue, flaky badges, error grouping, and an
+  "Open trace" action to Run Detail.
+
+---
+
+## Sources
+
+- Playwright Test Agents (official): https://playwright.dev/docs/test-agents
+- Playwright locators (official priority order): https://playwright.dev/docs/locators
+- Playwright Codegen (assertion toolbar, locator picker): https://playwright.dev/docs/codegen
+- Playwright agents review (Bondar Academy): https://bondaracademy.com/blog/playwright-ai-agents-review
+- Playwright agents limitations (BrowserStack): https://www.browserstack.com/guide/playwright-agent
+- Playwright agents strategies (Currents): https://currents.dev/posts/9-strategies-to-get-the-most-out-of-playwright-test-agents
+- qa-core-heal (GitHub): https://github.com/sardar-usman/qa-core-heal
+- qa-core-heal launch thread (Ministry of Testing): https://club.ministryoftesting.com/t/looking-for-5-testers-open-source-self-healing-tool-for-broken-playwright-locators/87557
+- "Self-healing locators: useful or marketing?" (MoT): https://club.ministryoftesting.com/t/ai-ml-self-healing-locators-useful-or-just-marketing/52681
+- Designing safe self-healing automation (MoT): https://www.ministryoftesting.com/insights/when-ai-maintains-your-tests-designing-safe-self-healing-automation
+- Hidden risk in self-healing (SD Times): https://sdtimes.com/software-testing/the-hidden-risk-in-self-healing-test-automation-a-governance-blueprint-for-digital-banking/
+- Stagehand caching: https://docs.stagehand.dev/examples/caching · https://github.com/browserbase/stagehand
+- Midscene caching: https://midscenejs.com/caching
+- Magnitude (Show HN): https://news.ycombinator.com/item?id=43796003
+- Octomind (Show HN): https://news.ycombinator.com/item?id=37645376
+- AI-native QA growth question (HN): https://news.ycombinator.com/item?id=43898735
+- Ask HN: agent-driven QA: https://news.ycombinator.com/item?id=48069781
+- QA Wolf multi-agent maintenance: https://www.qawolf.com/blog/three-principles-for-building-multi-agent-ai-systems
+- Healenium (LCS / tree comparison): https://www.happiestminds.com/wp-content/uploads/2024/02/Adaptive-Automation-Empowering-Seamless-Testing-Through-Self-Healing-Locators-and-LCS-Algorithm.pdf
+- awesome-ai-testing (landscape): https://github.com/tugkanboz/awesome-ai-testing
+- Flaky test root causes (ContextQA): https://contextqa.com/blog/flaky-tests-automated-testing/
+- Visual regression production guide: https://testquality.com/playwright-visual-regression-guide/
+- Reporting tools comparison: https://testdino.com/blog/flaky-test-detection
+- mabl vs Testim pricing (G2 / Bug0): https://www.g2.com/compare/testim-io-vs-mabl · https://bug0.com/knowledge-base/mabl-pricing
diff --git a/PRODUCT_DIRECTION.md b/PRODUCT_DIRECTION.md
new file mode 100644
index 0000000..98aab3a
--- /dev/null
+++ b/PRODUCT_DIRECTION.md
@@ -0,0 +1,234 @@
+# 🧭 Product Direction — One Engine for Flows: Test, Extract, Monitor
+
+> Follows [PRODUCT_WORKFLOW_ANALYSIS.md](PRODUCT_WORKFLOW_ANALYSIS.md) and
+> [COMPETITIVE_RESEARCH.md](COMPETITIVE_RESEARCH.md). This document records the decisions made
+> and designs the product around them.
+
+## 0. Decisions (from the founder)
+
+| Question | Decision |
+|---|---|
+| Scraper's role | **It is the base engine.** Scraping and testing do the same work |
+| Target market | **Small firms and projects that need testing but can't staff QA**, and want all their user flows managed in one space, not isolated individual tests |
+| Primary user | **The "user tester"**: whoever does the testing (founder, PM, developer, support person), not a QA specialist |
+| Sharing | **Shareable reports** |
+| Code | **Needed sometimes**, because code gives control that replaying steps can't |
+
+---
+
+## 1. Why the scraper is the right base engine
+
+Scraping and testing are the same pipeline with a different last step:
+
+```
+                 NAVIGATE → LOCATE → ACT → EXTRACT → ┬─ EXPORT  = scraping
+                                                     ├─ ASSERT  = testing
+                                                     └─ ALERT   = monitoring
+```
+
+A test is a scraper that asserts. A scraper is a test that exports. A monitor is a test on a timer.
+
+**Evidence from the code:** the scraper side already has the page-understanding capabilities the
+tester lacks.
+
+| Capability | Scraper side (exists) | Tester side (today) |
+|---|---|---|
+| Network intelligence | `Scanner`: API endpoint discovery (`/api/`, GraphQL, JSON), JSON response status, WebSocket detection (`recon/scanner.py:182-222`) | Logs responses ≥ 400 only |
+| Stability / waiting | `Runtime.smart_wait`, `wait_for_idle`, mutation and stability tracking | Fixed sleeps |
+| Structured data | `DOMAnalyzer`: repeating-section detection, field detection | None |
+| Pagination | `PaginationDetector`, runtime pagination loop | None |
+| Sessions / auth | Guided access, `storage_state` capture and reuse | Password masked, can't replay |
+| Actions | Navigation steps: click, fill, wait, scroll, press | `navigate`, `click`, `fill` only |
+| Resilience | — | Selector ladder, fingerprints, healing tiers (tester is ahead here) |
+
+So the plan isn't "fold the scraper into the tester." It's **merge both into one flow engine**:
+the scraper supplies page understanding, and the tester supplies fingerprints and healing.
+
+---
+
+## 2. Why code is needed (and why replay-only tools fail)
+
+Replay-only tools (Codegen, mabl, Testim, Selenium IDE) repeat a **fixed path**. Research puts
+the no-code ceiling at roughly **85–90% of cases**. The remaining 10–15% are exactly the cases
+that catch real bugs:
+
+| Real scenario | Why step-replay fails | What it needs |
+|---|---|---|
+| "Add the **cheapest in-stock** item to the cart" | The target changes every run | **Extract a list → pick by rule** |
+| "Price on the product page = price in cart = checkout total" | Replay clicks but never compares | **Extract to a variable → assert later** |
+| "Every row in the orders table has a valid date and status" | Can't loop | **For each extracted item → assert** |
+| "Search results are sorted by price across **all pages**" | Stops at page 1 | **Pagination loop + extraction** |
+| "Cookie banner *may* appear" / A/B variants | Fails on the optional step | **Condition: if visible → act** |
+| "Sign up with a **new unique email** each run" | Replays the same email, then "already exists" | **Generated data / variables** |
+| "Create the test user via **API**, then test the UI" | UI setup is slow and flaky | **API call step** |
+| "Order total = Σ items + tax" | Business rule | **Computed assertion / code** |
+| "After *Pay*, `POST /api/orders` returns 201 with an `orderId`" | Doesn't see the network | **Wait-for-response + JSON assertion** |
+
+Most of these need **extraction plus variables plus logic**, which is scraper territory. That's
+why the scraper-as-core idea is right, and it's a genuine competitive edge: other test tools
+treat data extraction as an afterthought.
+
+### How to add code without Katalon's trap
+
+Katalon's dual "Manual view ↔ Script view" converts code back and forth. The conversion is lossy:
+code gets rewritten or lost when you switch, and users are told to never switch views. **Don't
+build two-way conversion.** Use a layered step model instead:
+
+| Layer | For | What it is |
+|---|---|---|
+| **1. Recorded steps** | Everyone | Click, fill, select, press… captured by doing the task |
+| **2. Smart steps** (still visual) | The user tester | Extract, Assert, Condition, Loop, Paginate, API call, Wait-for-response, Data/variables. Configured with forms, no code |
+| **3. Code step** | When forms aren't enough | A **self-contained code block inside the flow**, with a documented context (`page`, `vars`, `extract()`, `expect()`, `request`). Shown as one card in the visual flow and edited in a code editor. **Never converted back into visual steps** |
+| **Eject** | Teams that outgrow it | **One-way** export of the whole flow to a Playwright project (pytest or TS). The exit door, which removes lock-in fear |
+
+The flow (a list of typed steps) is always the source of truth. Code blocks are just one step type
+inside it, so nothing is ever lossy.
+
+---
+
+## 3. The product for small firms: "all your user flows in one space"
+
+A small firm doesn't want "a test." It wants to know **whether signup, login, search, checkout
+and billing still work**, without hiring anyone.
+
+### 3.1 Flow Map (the home screen)
+A board of the app's **journeys** (Signup, Login, Search, Checkout, Password reset…), each with
+live health across every dimension: functional ✅, network ⚠️, a11y ❌, performance ✅, visual ✅.
+**Shared steps** such as "Login" are reusable **components**: record once, reused by every flow,
+healed once. This is the "whole-flow view instead of individual tests" the target market wants.
+
+### 3.2 Runs happen without a QA person
+- **On demand:** "Run all flows" before a release.
+- **Scheduled monitoring:** every hour or night, locally, or in CI via the CLI. Small firms pay
+  Checkly $24–64+/month plus per-run fees for exactly this; locally it costs nothing per run.
+- **Alerts:** email, Slack or webhook when a flow breaks, with the report link attached.
+
+### 3.3 One run, every dimension, one issue list
+As designed in the workflow analysis: capture everything (actions, HAR, console, trace), replay
+once, evaluate every check pack, and output **deduplicated findings** with status
+(new / known / accepted / fixed).
+
+### 3.4 Shareable reports (Jam-style)
+- **Single-file HTML report**: self-contained with embedded screenshots and findings. Email it,
+  attach it to a ticket, or host it anywhere (GitHub Pages, S3). Works with no server.
+- **Per-finding link**: a direct anchor to one issue, with its evidence (screenshot, failing
+  request, console line, trace step).
+- **Optional later:** a hosted share service (possible paid tier; see §6).
+- ⚠️ **Redaction is mandatory:** HAR files and console logs contain auth tokens, cookies and
+  personal data. Scrub `Authorization`, `Cookie`, `Set-Cookie`, tokens in URLs and bodies, and
+  masked form values **before** anything is exported or shared.
+
+---
+
+## 4. Why this beats the competition
+
+| | Playwright Codegen / Agents | mabl / Testim | Katalon | Checkly | Jam | **This product** |
+|---|---|---|---|---|---|---|
+| Start with no code | ⚠️ records to code | ✅ | ✅ | ❌ code | ✅ capture | ✅ record |
+| Data extraction + variables + logic | code only | limited / JS | ✅ (dev-heavy) | code only | ❌ | ✅ **native smart steps (scraper core)** |
+| Code when needed, without lossy conversion | code-only | bolted-on JS | ❌ lossy dual view | code-only | ❌ | ✅ **code step + one-way eject** |
+| All dimensions in one run (UI/API/a11y/visual/perf) | UI | partial | partial | UI/API | capture only | ✅ |
+| Flow map / whole-app view | ❌ | ⚠️ | ⚠️ | ⚠️ | ❌ | ✅ |
+| Scheduled monitoring + alerts | ❌ | ✅ (paid) | ✅ (paid) | ✅ per-run fees | ❌ | ✅ **local, no per-run cost** |
+| Shareable reports | HTML report | cloud | cloud | cloud | ✅ | ✅ single-file + per-finding link |
+| Safe self-healing | Healer (LLM, edits code) | ✅ (black box) | ✅ | ❌ | ❌ | ✅ **deterministic, refuses rather than guesses, audited** |
+| Cost for a small firm | free (needs devs) | ~$450–500+/mo | enterprise pricing | $24–64+/mo + runs | freemium | **free/local; optional paid sharing** |
+| Lock-in | none | high | high (Testsigma similar) | medium | n/a | **none (eject to Playwright)** |
+
+**Positioning:**
+> *"Record your app's user flows once. Check they work (clicks, data, APIs, accessibility,
+> speed) on every release or every hour. Share one clear report. Drop into code when you need
+> control. No QA team, no per-run bill, no lock-in."*
+
+---
+
+## 5. Target architecture — one flow engine, three modes
+
+```
+                         ┌──────────────── FLOW (source of truth) ─────────────────┐
+                         │ typed steps: recorded · smart (extract/assert/if/loop/   │
+                         │ paginate/api/wait-response/data) · code · component-ref  │
+                         └───────────────────────────┬──────────────────────────────┘
+                                                     │
+  ┌──────────────────────────── CORE ENGINE (merged scraper + tester) ─────────────────────────┐
+  │ Session     Runtime (browser/context), storage_state auth, env + secrets                    │
+  │ Locate      selector ladder (role-first) + fingerprints + safe healing                      │
+  │ Act         click/fill/select/press/upload/hover/scroll                                     │
+  │ Extract     DOMAnalyzer (repeating lists, fields) → typed values/tables → variables         │
+  │ Paginate    PaginationDetector + loop                                                       │
+  │ Observe     network (Scanner → full HAR + API catalog), console, trace, screenshots         │
+  │ Stabilize   smart_wait / wait_for_idle / mutation tracking (replaces fixed sleeps)          │
+  │ Code        sandboxed code-step context: page, vars, extract(), expect(), request           │
+  └────────────────────────────────────────────┬────────────────────────────────────────────────┘
+                  ┌─────────────────────────────┼──────────────────────────────┐
+             TEST mode                     EXTRACT mode                   MONITOR mode
+        check packs → findings        export CSV/XLSX/JSON           schedule → run → alert
+        (functional, network/API,     (today's scraper output)       (email/Slack/webhook)
+         a11y, visual, perf, console)
+                  └─────────────────────────────┼──────────────────────────────┘
+                                  REPORTS (single-file HTML, per-finding links, JUnit)
+                                  STUDIO (flow map, recorder, findings inbox, heal review)
+```
+
+### Where today's code goes
+
+| Today | Becomes |
+|---|---|
+| `scrapewizard_runtime/` (Runtime, BaseScraper, NavigationExecutor) | Core **Session / Act / Stabilize**; `BaseScraper`'s `get_items` / `parse_item` becomes an Extract step, and pagination becomes a Paginate step |
+| `recon/scanner.py` | Core **Observe**, extended from endpoint discovery to full HAR + API catalog |
+| `recon/dom_analyzer.py`, `pagination.py` | Core **Extract / Paginate**, also powering "suggest assertions" |
+| `engine/selector_engine.py`, `fingerprint.py`, `healing.py` | Core **Locate** (make role-first) |
+| `engine/recorder.py` | Recorder on the core, plus capture of network/console/trace and more events |
+| `engine/sandbox.py`, `checks/` | **Test mode** runner + check packs, rebuilt as a flow executor |
+| `core/orchestrator.py` (scraper state machine) | **Retired over time**; its guided-access and assessment logic moves into core Session |
+| `llm/understanding`, `codegen`, `repair` | **Optional AI assists**: propose extractions/assertions, write a code step from plain English ("check total = sum of items"), tier-6 heal. Always logged, never on the hot path |
+| CLI `build` | `extract` mode (keep the command as an alias for compatibility) |
+| Studio | Flow map, recorder, flow editor (visual + code step), findings inbox, reports, schedules |
+
+---
+
+## 6. Business model fit (small firms)
+
+- **Free / open-source core:** local engine, Studio, recorder, all check packs, scheduling via
+  CLI, single-file reports. This drives adoption.
+- **Optional paid convenience** (later, only if there's traction):
+  - hosted share links for reports and findings
+  - hosted scheduled runs, for small firms without an always-on machine or CI
+  - team workspace
+  - alerts/integrations hub
+
+  This matches the open-core idea in PLATFORM_PLAN §28 and never meters the local engine.
+
+---
+
+## 7. Roadmap (replaces the order in earlier plans)
+
+| Phase | Deliverable | Why first |
+|---|---|---|
+| **0. Flow schema** | A versioned JSON schema of step types (recorded, smart, code, component-ref) plus the variables model | **The contract everything builds on**: recorder writes it, engine runs it, Studio edits it, eject exports it |
+| **1. Merge the engines** | One core package (Session / Locate / Act / Extract / Paginate / Observe / Stabilize); flow executor runs today's 3 actions + Extract + Assert + variables | Removes the two-products problem at the code level |
+| **2. Smart steps** | Extract (value/list/table), Assert (compare, count, regex, sum), Condition, Loop, Paginate, API call, Wait-for-response, Data (generated / CSV) | Delivers the "control beyond replay" promise without code |
+| **3. Code step + eject** | Code block with context API + editor; one-way export to a Playwright project | The escape hatch; kills lock-in fear |
+| **4. Capture + findings** | Recorder captures HAR/console/trace, role-first locators, `storage_state` auth; findings dedup; check packs incl. network/API + performance | From the workflow analysis, phases 1–4 |
+| **5. Flow map + components + monitoring** | Journey board, reusable components (Login), schedules, alerts | The small-firm value: all flows in one space, running without a QA person |
+| **6. Shareable reports** | Single-file HTML, per-finding anchors, **redaction** | The chosen sharing model |
+| **7. Review-first Studio UX** | Proposals after recording, findings inbox, heal review; engineer panel as "advanced" | Built for the user tester |
+
+Frontend polish from MARKET_READY_PLAN (React Query, primitives, TS strict) happens **inside
+Phase 5–7**, on the new screens, rather than polishing the current per-step screens.
+
+---
+
+## 8. Risks
+
+1. **Scope.** This is a big product. Guard it: Phases 0–3 plus a minimal Phase 4 (HAR + findings)
+   is already a strong, differentiated v1. Performance, visual polish and hosted sharing can follow.
+2. **Code-step safety.** Code runs locally, which is fine for your own flows. Shared or imported
+   flows containing code must show a clear warning before running.
+3. **Secrets in shared reports.** Redaction must be on by default, tested, and impossible to
+   forget (§3.4).
+4. **The engine merge is a refactor with regression risk.** Keep the existing CLI `build` and the
+   `test` path working through Phase 1, and gate it with the existing golden tests plus new
+   flow-executor tests.
+5. **Naming.** "ScrapeWizard" undersells a flow-quality product. Decide before any public launch
+   (PLATFORM_PLAN §18).
diff --git a/PRODUCT_WORKFLOW_ANALYSIS.md b/PRODUCT_WORKFLOW_ANALYSIS.md
new file mode 100644
index 0000000..ee1a48e
--- /dev/null
+++ b/PRODUCT_WORKFLOW_ANALYSIS.md
@@ -0,0 +1,287 @@
+# 🧭 Product & Workflow Analysis — "Test Everything in One Place"
+
+> **Update:** the §6 decisions have been made. The scraper is the **base engine**, the target is
+> **small firms without QA staff**, the primary user is the **user tester**, and reports are
+> **shareable**. The resulting design is in [PRODUCT_DIRECTION.md](PRODUCT_DIRECTION.md), which
+> supersedes §3 and §5 below where they differ.
+
+> **Question analysed:** the goal is one place where a user loads their app and tests *everything*
+> (UI, UX, network and more), with scraping as only the starting point. The user workflow differs
+> from normal UI testing and code testing. How should the project change, and what's wrong today?
+>
+> Inputs: the current codebase (commit `6003e10`, verified by grep and execution), plus
+> [COMPETITIVE_RESEARCH.md](COMPETITIVE_RESEARCH.md) and new research on all-in-one platforms,
+> capture tools, network-from-traffic testing and performance-in-journey tools (sources at the end).
+
+---
+
+## 1. The core insight: your workflow really is different
+
+Every mainstream testing workflow starts from **specification**: someone decides in advance what
+to check, then writes or records a test for that one thing. Your workflow starts from **behaviour**:
+a person does the task once, and the tool works out everything worth checking from that one
+session.
+
+| Workflow | Who | How a test is born | What one session produces | Where results live |
+|---|---|---|---|---|
+| **Code-first** (Playwright, Cypress) | SDET / developer | Writes assertions in code upfront | One functional test | CI logs, HTML report |
+| **Codeless record/playback** (mabl, Testim, Katalon, Codegen) | QA / manual tester | Records clicks, adds assertions | One functional test (plus healing) | Vendor cloud |
+| **Specialist tools** (Postman, Lighthouse, axe, Percy) | Separate specialists | A separate session per tool | One dimension each | Five tools, five reports |
+| **Capture tools** (Jam.dev) | Anyone | Clicks "capture" when a bug happens | Video + console + network + steps + device | A shareable bug report, **no replay/regression** |
+| **Your vision** | Anyone who can use the app | **Does the task once** | **Functional + network/API + a11y + visual + performance + console, from one recording** | **One local place, one findings list** |
+
+**The short version: record once, check everything, see one list of issues.** That's the product.
+It's a real gap. Each close analogue covers only a slice:
+
+- **Jam.dev** captures everything, but only for one-off bug reports. You can't replay it as a regression test.
+- **Chrome DevTools Recorder** records, replays and measures performance, but has no checks, no history and no dashboard.
+- **Lighthouse user flows** measure performance and a11y inside a journey, but you have to script them in Puppeteer.
+- **Keploy / HAR tools** turn browser traffic into API tests, but test nothing in the UI.
+- **Playwright Codegen / Healer** cover UI only, and are code-first.
+- **Katalon / mabl / Testsigma** are "all-in-one", but code- or cloud-heavy, expensive, and criticised for lock-in and slowness.
+
+Nobody does **one recording → every quality dimension → one deduplicated issue list, locally.**
+
+> **The current code does not implement this workflow.** It implements a
+> *record/playback functional tester* with a few side-checks attached. The rest of this document
+> explains the gap.
+
+---
+
+## 2. Issues — deep analysis
+
+### A. Identity: two products share one body
+
+| # | Issue | Evidence |
+|---|---|---|
+| A1 | **The scraper and the tester are separate products that happen to share a repo.** The Studio (the testing product) never calls the scraper pipeline. | `grep Orchestrator|DOMAnalyzer|CodeGenerator` in `studio/` returns nothing |
+| A2 | **About 4,700 lines of scraper-only code** (`core/` orchestrator state machine, `llm/` understanding, codegen and repair agents, `recon/`, `interactive/`) serve a product the vision calls "just the starting part." | Line counts by package |
+| A3 | **The README headline still leads with scraping:** "Self-Healing Web Scraper Builder & UI/UX Test Automation Studio", "Two Products, One Unified Engine." Visitors can't tell what the product *is*. | `README.md` lines 1–20 |
+| A4 | **The CLI mixes both worlds:** `build`/`resume`/`list` (scraper) next to `start`/`record`/`test` (tester), under a name that says "Scrape." | `cli/main.py` |
+| A5 | **"UX testing" means something else in the market.** UX testing means usability research with real people (task completion, satisfaction; Maze, UserTesting). What this tool does is **UI, functional and technical-quality testing of user journeys**. Using the word "UX" attracts the wrong buyers and invites a comparison you'll lose. | Industry definitions (Contentsquare, Testsigma, Digivante) |
+
+**What it means:** until the identity is settled, every design decision gets pulled two ways, and
+marketing can't make one claim.
+
+### B. The recording, your most valuable asset, is thin
+
+In your workflow **the recording is the source of truth.** Everything should be derived from it.
+Right now it captures almost nothing beyond the DOM:
+
+| # | Issue | Evidence |
+|---|---|---|
+| B1 | The recorder captures only **`click` and `change` DOM events**. No `input`, select, check, keypress, upload or dialog events. | `recorder.py:132-142` |
+| B2 | **No network capture during recording.** No HAR, no request/response bodies, no timings. So **network/API tests can't be derived from the recording**, and the whole "network testing" pillar has nothing to work from. | `grep har|on("request"|on("response"|tracing` in `recorder.py`: none |
+| B3 | **No console capture during recording.** Baseline console noise is unknown, so every run can't tell "new error" from "always there." | Same grep |
+| B4 | **No Playwright trace during recording or replay.** No DOM snapshots over time and no timeline evidence. | Same grep |
+| B5 | Login secrets are replaced with `***MASKED***` and **can't be replayed**, which blocks every authenticated app. | `recorder.py` masking |
+
+### C. The network pillar is a side-effect, not a test type
+
+| # | Issue | Evidence |
+|---|---|---|
+| C1 | The only network feature is a listener that logs responses **≥ 400**. | `checks/console_network.py:20-22` |
+| C2 | You can't assert a **specific API call**: "POST /api/checkout returns 200 in under 800 ms and has an `orderId`." | No API assertion model |
+| C3 | No request/response schema or contract checks, and no latency budgets. | — |
+| C4 | No mocking or replay of the network (Playwright `routeFromHAR`), which you'd need for stable UI tests independent of a flaky backend. | — |
+
+**What it means:** "network testing" is currently a warning count per step, not a capability a
+user can design, review or rely on.
+
+### D. Results are raw logs, not findings
+
+| # | Issue | Evidence |
+|---|---|---|
+| D1 | **A11y runs full-page axe on every step**, so the same violation is reported N times per run. The earlier demo showed the same 2 violations under each step. | `sandbox.py:207` inside the step loop; `test` CLI output |
+| D2 | `StepResult` stores **per-step lists** of console, network and a11y. There's no deduplication, severity, first-seen or occurrence count. | `models.py` `StepResult` |
+| D3 | **No `Finding`/`Issue` entity.** Nothing can be marked new / known / accepted / fixed across runs, so every run re-reports everything. | `models.py`: no such class |
+| D4 | **Mixed pass/fail semantics.** Functional failures fail the run; a11y and visual are silently informational. The user can't set per-dimension policy (block / warn / info). | `sandbox.py` status logic |
+
+**What it means:** "one place" only has value if the one place is **a clean, deduplicated list of
+issues across every dimension.** Today it's a pile of per-step logs, which is the problem Jam,
+Sentry and Currents exist to solve.
+
+### E. Missing dimensions the vision promises
+
+| Dimension | Status | Best-in-class engine to reuse (don't rebuild) |
+|---|---|---|
+| Functional flow | ✅ replay + assertions (visible, url) | Playwright |
+| Accessibility | ◑ axe per step (duplicated) | axe-core, ideally once per unique page state |
+| Visual | ◑ raw PIL pixel diff | pixelmatch-style diff + masking, or Playwright `toHaveScreenshot` |
+| Console health | ◑ errors only, no baseline | Playwright console events + a recorded baseline |
+| **Network / API** | ❌ 4xx/5xx listener only | HAR capture → derived API assertions (Keploy/HAR pattern) |
+| **Performance** | ❌ none | **Lighthouse user flows** (navigation / timespan / snapshot) or Web Vitals via `PerformanceObserver` |
+| **Data correctness** (the scraping heritage) | ❌ not in the tester | Your `DOMAnalyzer` repurposed: "extract this list, assert ≥ N items / price format" |
+| Security basics (later) | ❌ | Mixed content, cookie flags, security headers from HAR |
+
+Lighthouse's three modes line up exactly with a recorded journey: **navigation** for `goto`
+steps, **timespan** for interaction sequences, and **snapshot** after a step. That's a natural
+performance pack.
+
+### F. Data model: no home for "everything"
+
+The current schema is `Test → Step → Run → StepResult` (plus `Setting`). It has no place for:
+
+- **Project / App under test**: the thing the user "loads."
+- **Environment**: dev, staging and prod base URLs, credentials and variables. The only `base_url`
+  in the backend belongs to the *LLM runtime*, not the app (`routes_settings.py:23`).
+- **Journey / Recording as an asset**: the raw capture (actions + HAR + console + trace), kept
+  separate from any single test derived from it.
+- **Check packs and policies**: which dimensions run, and what counts as failure.
+- **Findings**: the deduplicated issues with status (§D).
+- **Suite / schedule**: groups of journeys run together.
+
+### G. The UI is built for the wrong persona
+
+- The **Step Manager** puts CSS/XPath selector ladders front and centre. That's an engineer's editor.
+  The person your workflow targets ("anyone who can use the app") shouldn't have to read XPath.
+- In a do-the-task workflow **users don't know what to assert upfront.** The natural UX is
+  **review, not write**: after recording, the tool proposes checks ("we saw `POST /api/login`
+  return 200, keep checking it?", "3 a11y issues found, set as baseline?", "this timestamp changes
+  every run, mask it?") and the user accepts or rejects them. The current UI has no proposal/review
+  queue at all.
+- **No findings inbox** (see §D), no heal-review queue, no per-dimension summary on a run.
+
+### H. Competitive exposure (from the research)
+
+- Playwright's free Healer and Codegen, and qa-core-heal's safety bar, make **"self-healing" a weak
+  headline.** It's table stakes now, not the story.
+- The **all-in-one criticism is real**: Katalon is called slow and bloated; Testsigma is called
+  locked-in. Breadth without depth fails. **Mitigation:** one shared capture layer, with pluggable
+  check packs that each delegate to a best-in-class engine (axe, Lighthouse, Playwright trace,
+  pixelmatch) instead of homegrown approximations.
+- **Your defensible story:** *record once → full quality report → one issue list → local, no
+  subscription, no lock-in (exports to Playwright).*
+
+---
+
+## 3. Proposed product model — "one execution, many lenses"
+
+```
+PROJECT (the app under test)
+ ├─ Environments: dev / staging / prod → base URL, auth (storageState), variables, secrets
+ └─ JOURNEYS (recorded once by a human doing the task)
+      Capture = actions + fingerprints + HAR (network) + console + trace + screenshots + timings
+      │
+      ├─ CHECK PACKS derived from the capture (each has a policy: block / warn / info)
+      │    • Functional   — steps + assertions (proposed, then reviewed)
+      │    • Network/API  — key calls: status, latency budget, schema, "contains"
+      │    • Accessibility— axe once per unique page state, baseline known issues
+      │    • Visual       — per-state baselines with masks
+      │    • Performance  — Web Vitals / Lighthouse flow per navigation and interaction
+      │    • Console      — new errors vs recorded baseline
+      │    • Data         — extraction assertions (scraping engine reused here)
+      │
+      └─ RUN = replay the journey ONCE; every enabled pack evaluates the SAME execution
+             (one trace, one HAR, one set of screenshots → consistent and cheap)
+                    │
+                    ▼
+            FINDINGS (deduplicated across steps and runs)
+              severity · dimension · evidence (trace/screenshot/request) · first seen
+              status: new → known → accepted / fixed · regression flag
+                    │
+                    ▼
+            Findings inbox · run report · CI exit code / JUnit · Playwright export
+```
+
+**Principles:**
+1. **Capture maximally, check selectively.** Record everything once; packs decide what matters.
+2. **Replay once, evaluate many.** Never run the browser six times for six dimensions.
+3. **Findings, not logs.** Deduplicate, track status across runs, and surface regressions.
+4. **Review, don't write.** The tool proposes checks and the human accepts them.
+5. **Delegate to best-in-class engines.** Your value is orchestration, evidence and workflow, not re-implementing axe or Lighthouse.
+6. **Deterministic by default, AI optional.** AI can name things, propose assertions and heal as a last resort, always logged. (Your existing philosophy, kept.)
+
+**Where scraping fits:** as the **Data pack** ("extract the product list, assert 20 items with
+valid prices") and as the engine that *proposes* data assertions from a page. The standalone CLI
+scraper (`build`) is either kept as a separate legacy tool or split into its own package (see §6).
+
+---
+
+## 4. Redesigned user workflow (in the app)
+
+| Step | New workflow | Today |
+|---|---|---|
+| 1 | **Create project**: app URL, environments, log in once (captured as `storageState`) | Enter a URL per test; no auth replay |
+| 2 | **Record journey**: do the task; capture actions, network, console, trace, screenshots | Clicks and field changes only |
+| 3 | **Review proposals**: API calls seen, text/visibility assertions, a11y baseline, suggested visual masks, perf baseline. Accept or reject | None; land in a selector editor |
+| 4 | **Journey saved** with its enabled check packs and policies | Test = steps |
+| 5 | **Run** manually, on a schedule, or in CI: one replay, all packs evaluated | One replay, side-checks per step |
+| 6 | **Findings inbox**: new/regression issues across all dimensions, deduplicated, with evidence | Per-step lists, repeated every run |
+| 7 | **Triage**: accept known issues, approve or reject heals, export to Playwright/JUnit | Heal approval not implemented |
+
+The engineer view (selector ladder, raw steps) stays available as an **advanced** panel, not the default.
+
+---
+
+## 5. Re-sequenced roadmap
+
+| Phase | Goal | Key work | Replaces / updates |
+|---|---|---|---|
+| **0. Identity** (days) | One product, one story | Decide the scraper's fate (§6); rename the positioning away from "scraper builder" and "UX testing"; rewrite the README headline | README, PLATFORM_PLAN §1/§18 |
+| **1. Capture layer** | The recording becomes the source of truth | Recorder: more event types, **HAR + console + trace capture**, role-first locators, auth via `storageState` + env secrets | BUILD_GUIDE Step 1.5, COMPETITIVE_RESEARCH P0.1/P1.5-9 |
+| **2. Domain model** | A home for "everything" | `Project`, `Environment`, `Journey` (+ capture artifacts), `CheckPack`/policy, **`Finding`** with dedup and status | PLATFORM_PLAN §12 |
+| **3. Findings engine** | Logs → issues | Dedup across steps and runs; a11y once per page state; per-pack policies; failure classification (locator / assertion / timeout / network / app); regression detection | Fixes §D |
+| **4. New check packs** | Deliver the promised dimensions | **Network/API pack** (from HAR), **Performance pack** (Web Vitals / Lighthouse flow), visual masking, console baseline, Data pack | Fixes §C, §E |
+| **5. Review-first UX** | Build for the actual persona | Proposal review queue after recording, findings inbox, heal-review queue, per-dimension run summary, engineer panel as advanced | FRONTEND_PLAN, MARKET_READY_PLAN P3 |
+| **6. Outputs & CI** | Fit into real pipelines | JUnit XML, exit codes by policy, Playwright TS/pytest export, trace links | MARKET_READY_PLAN P5 |
+
+The frontend refactor in MARKET_READY_PLAN (React Query, primitives, TS strict) still applies. Do
+it **alongside Phase 5**, not before Phases 1–3. The UI should be rebuilt around findings and
+proposals, not polished around the current per-step model.
+
+**Defer or cut:** drag-drop flow designer, RBAC/teams, scheduling UI (a CLI cron is enough at
+first), embedded browser, Electron packaging.
+
+---
+
+## 6. Decisions you need to make
+
+1. **What happens to the scraper?**
+   - **(a) Absorb it (recommended):** it becomes the Data pack plus the "propose data assertions"
+     helper. Retire `build`'s separate state machine over time.
+   - **(b) Split it:** move the scraper to its own package or repo; this one becomes testing-only.
+   - **(c) Keep both as is:** not recommended, since it keeps the identity problem (§A).
+2. **Who is the primary user?** Pick one for v1:
+   - **Manual testers / product people** ("anyone who can use the app"): favours review-first UX,
+     findings inbox, no code visible. **Best fit for your "different workflow".**
+   - **SDETs / developers:** favours code export, CLI and CI first. Crowded, and Playwright's
+     free tools cover it.
+3. **Naming:** drop "UX testing" (it means usability research) and "scraper." Something that says
+   *record once, check everything* (journey quality, web quality, experience checks) fits better.
+   This ties to the existing product-name decision (PLATFORM_PLAN §18).
+4. **Local-only or shareable?** Findings are more valuable when shared (a link to a finding with
+   its evidence, like Jam). Decide whether v1 stays single-user local or exports shareable reports.
+
+---
+
+## 7. Summary
+
+- **The idea is sound and the gap is real:** nobody offers *record once → every quality
+  dimension → one deduplicated issue list, locally.*
+- **The current build doesn't implement that idea yet.** It's a record/playback functional tester
+  with side-checks, sharing a repo with an unrelated scraper.
+- **The biggest structural issues:**
+  - (A) split identity
+  - (B) a recording that captures no network, console or trace
+  - (D) raw per-step logs instead of findings
+  - (F) no project, environment or findings data model
+  - (G) an engineer-centric UI for a "just do the task" persona
+- **Fix order:** identity → capture → model → findings → new check packs → review-first UX → CI.
+
+---
+
+## Sources
+
+- Jam.dev (one-click capture of console, network, steps): https://jam.dev · https://dev.to/namnguyenthanhwork/jamdev-an-effective-tool-for-bug-reproduction-1979
+- Chrome DevTools Recorder (record, replay, measure performance, export): https://developer.chrome.com/docs/devtools/recorder/overview
+- Lighthouse user flows (navigation / timespan / snapshot): https://web.dev/articles/lighthouse-user-flows · https://github.com/GoogleChrome/lighthouse/blob/main/docs/user-flows.md
+- HAR-based API testing: https://dev.tools/blog/har-file-api-testing/ · https://github.com/slve/har.test
+- Keploy API test recorder: https://keploy.io/docs/running-keploy/api-testing-chrome-extension/
+- Playwright HAR replay: https://qaskills.sh/blog/playwright-network-har-replay-testing
+- Generating API tests from HAR with AI (Bondar Academy): https://bondaracademy.com/blog/generate-api-tests-with-ai-playwright
+- Katalon vs Testsigma (all-in-one tradeoffs): https://www.testsprite.com/use-cases/en/compare/katalon-vs-testsigma
+- Katalon alternatives (criticisms): https://bugbug.io/blog/test-automation-tools/katalon-alternatives/
+- UX vs UI testing definitions: https://contentsquare.com/guides/ui-design/testing/ · https://testsigma.com/blog/ui-ux-testing/ · https://www.digivante.com/?p=28391
+- Manual testers moving to automation: https://smartbear.com/blog/no-code-no-problem-ai-powered-automation-for-manual-testers/ · https://www.shiplight.ai/blog/low-code-platforms-manual-testers
+- Prior research (Playwright agents, qa-core-heal, MoT, HN, Stagehand, Magnitude): see [COMPETITIVE_RESEARCH.md](COMPETITIVE_RESEARCH.md)