Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ CLI (src/index.ts) ─── commander routes to 4 commands:
src/scraper/scrape-location.ts ── shared orchestrator, accepts ParsedUrl (used by both commands); delegates final payload construction to exported pure helper `assembleScrapeResult()` which reconciles `business.totalReviews` with `reviews.length`
│
├── browser.ts ── persistent Chromium context (~/.revcli/chrome-profile/), anti-detection, SIGINT cleanup
├── auth.ts ── isSignedIn(), hasLimitedView(), waitForUserAuth(), waitForSignIn() – Google auth detection; decideSignedIn()/GOOGLE_AUTH_COOKIES make the cookie check authoritative
├── auth.ts ── isSignedIn(), hasLimitedView(), waitForUserAuth(), waitForSignIn(), waitForBrowserClose() – Google auth detection; decideSignedIn()/GOOGLE_AUTH_COOKIES make the cookie check authoritative
├── navigator.ts ── orchestrates state eviction → consent → locale → limited-view check → tab → sort; verifies loaded placeId against parsed.placeId
├── consent.ts ── Google consent handling, hl=en enforcement, g_ep/entry stripping
├── business-extractor.ts ── business name/rating/totalReviews/address extraction; page.evaluate() returns raw DOM strings only, parsing happens Node-side via parseReviewCount + parseRatingText; PII-scrubbed debug logging of candidate strings
Expand Down Expand Up @@ -138,8 +138,8 @@ URL input → `parseGoogleMapsInput()` validates → `scrapeLocation(parsed)` la
### Patterns to know

- **Persistent browser profile**: `launchBrowser()` uses `chromium.launchPersistentContext()` with `~/.revcli/chrome-profile/`. Google auth cookies survive between CLI runs. The `PROFILE_DIR` constant is exported from `browser.ts`. `ensureProfileDir()` in `browser.ts` creates the directory with owner-only `0700` permissions – it is a **credential store** holding your Google session cookies, so it must never be shared, backed up, or committed. The profile's spoofed user-agent and `navigator.webdriver` masking are ToS-sensitive (see the anti-automation note in README's Legal notice); removing them would break unauthenticated scraping.
- **Authentication flow**: Google Maps shows a "limited view" (no Reviews tab) to unauthenticated EEA users. `revcli auth` opens a browser for manual sign-in. The scrape command also detects limited view inline and can prompt for auth in headed mode.
- **Cookie-authoritative sign-in detection**: `auth.ts` defines `GOOGLE_AUTH_COOKIES` (the definitive set of Google session cookies: SID/HSID/SSID/APISID/SAPISID/`__Secure-*PSID`). `decideSignedIn(cookieCheckSucceeded, hasAuthCookies, signInButtonVisible)` treats a successful cookie check as authoritative – when `cookieCheckSucceeded` is true, `hasAuthCookies` decides, and the DOM "Sign in" button is only a fallback when the cookie check failed. `hasGoogleAuthSession()` checks the cookie set directly; `waitForSignIn()` polls and throws a typed `UnrecoverableError` with kind `"AUTH_POLL"` on timeout.
- **Authentication flow**: Google Maps shows a "limited view" (no Reviews tab) to unauthenticated EEA users. `revcli auth` opens a browser for manual sign-in and never short-circuits on an existing session – the window stays open (until the user closes it) so the profile can be switched to a different account. The scrape command also detects limited view inline and can prompt for auth in headed mode.
- **Cookie-authoritative sign-in detection**: `auth.ts` defines `GOOGLE_AUTH_COOKIES` (the definitive set of Google session cookies: SID/HSID/SSID/APISID/SAPISID/`__Secure-*PSID`). `decideSignedIn(cookieCheckSucceeded, hasAuthCookies, signInButtonVisible)` treats a successful cookie check as authoritative – when `cookieCheckSucceeded` is true, `hasAuthCookies` decides, and the DOM "Sign in" button is only a fallback when the cookie check failed. `hasGoogleAuthSession()` checks the cookie set directly; `waitForSignIn()` polls and throws a typed `UnrecoverableError` with kind `"AUTH_POLL"` on timeout, returning `SignInOutcome` (`"signed-in"` | `"browser-closed"`) otherwise – `waitForUserAuth()` treats a closed window as fatal for the scrape, while `revcli auth` reports it as a normal end. `waitForBrowserClose()` resolves once the window is gone, polling `pages()` as well as the `close` event because closing the last window on macOS leaves the process (and context) alive.
- **Default headed mode**: The browser shows by default so users can observe scraping. Use `--headless` to hide it. This is the opposite of most scrapers – it's intentional because auth requires a visible browser.
- **URL normalization**: `appendHlParam()` in `consent.ts` always forces `hl=en` and strips `g_ep`/`entry` tracking params that trigger Google's limited view. This runs before every navigation.
- **Centralized selectors**: All Google Maps CSS class selectors live in `src/scraper/selectors.ts`. Google obfuscates these names and changes them periodically. When scraping breaks, check selectors first. See [docs/selector-maintenance.md](docs/selector-maintenance.md) for the full update procedure.
Expand Down Expand Up @@ -170,7 +170,7 @@ URL input → `parseGoogleMapsInput()` validates → `scrapeLocation(parsed)` la
- ESM-only (`"type": "module"`, `.js` extensions in imports)
- Node 22+ required
- Strict TypeScript, zero `any` types
- Tests use vitest (287 tests across 17 files) – pure-function tests for parser, schema, URL, CSV, JSON, retry, rate-limiter, consent, unrecoverable, batch-utils, validate, scroller, storage-types, scrape-location, extractor-selectors, extractor, auth; Playwright-dependent modules are not unit tested
- Tests use vitest (297 tests across 17 files) – pure-function tests for parser, schema, URL, CSV, JSON, retry, rate-limiter, consent, unrecoverable, batch-utils, validate, scroller, storage-types, scrape-location, extractor-selectors, extractor, auth; browser-coupled code is otherwise not unit tested, except where a narrow structural type makes it fakes-only (`ClosableBrowser` for `waitForBrowserClose`)
- `parseInputFile()`, `slugify()`, and `deduplicateFilename()` in batch.ts are exported for testability
- `parseReview()`, `parseReviewCount()`, and `detectLanguage()` in parser.ts are exported for testability
- `parseRatingText()` in business-extractor.ts is exported for testability (called Node-side after `page.evaluate()` returns the raw aria-label string)
Expand All @@ -179,4 +179,5 @@ URL input → `parseGoogleMapsInput()` validates → `scrapeLocation(parsed)` la
- `extractPlaceIdFromUrl()`, `placeIdsMatch()`, and `canVerifyPlaceIdFormat()` in url.ts are exported for testability (direct tests cover priority, encoding, edge cases, null handling, case-insensitivity, and ChIJ vs 0x format discrimination)
- `VOLATILE_STORAGE_TYPES` in `src/scraper/storage-types.ts` is a pure constant kept separate from `browser.ts` so tests can pin the cookies-excluded invariant without pulling in Playwright
- `calculateStaleDelay()` and `shouldContinueScrolling()` in scroller.ts are exported for testability
- `waitForBrowserClose()` in auth.ts takes the narrow structural `ClosableBrowser` (not `BrowserContext`) so its poll/close-event behaviour is testable against a fake, and `waitForSignIn()` returns the `SignInOutcome` union instead of `void` so each caller's closed-window policy is explicit and pinned by tests
- `extractOriginalLanguageFromButtonText()` in extractor.ts is exported for testability – pure helper that parses the human-readable language name out of the translated-review toggle button text ("See original (Polish)" → "Polish", handles nested parens like "Chinese (Simplified)")
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed
- `revcli auth` no longer shuts the browser down as soon as Google Maps renders. It used to exit
immediately whenever the profile already held a signed-in session, which made it impossible to
sign in or switch accounts. The window now stays open – for credential entry, or for account
switching when a session already exists – and closes when sign-in is detected or when you close
it. The 5-minute timeout also reports a message instead of an unhandled-rejection stack trace.

## [0.1.4] - 2026-09-18

### Changed
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,8 @@ revcli auth logout # Clear session

**How it works:** revcli uses a persistent Chrome profile at `~/.revcli/chrome-profile/`. Sign in once with `revcli auth`, and all subsequent scrapes reuse that session. Google auth cookies persist between CLI runs. The profile is a **credential store** holding your Google session cookies — it is created with owner-only (`0700`) permissions. Do not share it or back it up; anyone with read access to it can act as your signed-in session.

`revcli auth` always leaves the browser window open: it closes once sign-in is detected, or when you close it yourself. If the profile is already signed in, the window stays open anyway so you can sign out and sign in with a different account — close it when you're done.

## Output schema

<details>
Expand Down
45 changes: 37 additions & 8 deletions src/commands/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ import {
isSignedIn,
hasLimitedView,
waitForSignIn,
waitForBrowserClose,
} from "../scraper/auth.js";
import { handleConsent } from "../scraper/consent.js";
import { UnrecoverableError } from "../core/errors.js";

export async function authLoginCommand(): Promise<void> {
logger.info("Opening Google Maps in Chrome...");
Expand All @@ -28,10 +30,7 @@ export async function authLoginCommand(): Promise<void> {
await handleConsent(page);
await page.waitForTimeout(3000);

if (await isSignedIn(page)) {
logger.success("Already signed in to Google Maps. No action needed.");
return;
}
const alreadySignedIn = await isSignedIn(page);

console.log(`
Sign in to your Google account:
Expand All @@ -41,13 +40,43 @@ export async function authLoginCommand(): Promise<void> {
3. Complete any 2FA prompts if required
4. Wait until Google Maps fully loads

The browser will close automatically once sign-in is detected.
The session is saved to ${PROFILE_DIR} and reused by every later run.
`);

logger.info("Waiting for sign-in... (timeout: 5 minutes)");
// This command is the credential-entry step, so it must never short-circuit
// on an existing session: the window has to stay open for the user to sign
// in, switch accounts, or sign in again.
if (alreadySignedIn) {
logger.info(
"Already signed in. The browser stays open so you can switch accounts – close the window when you're done.",
);
await waitForBrowserClose(context);
logger.success("Browser closed. Google session saved for later runs.");
return;
}

logger.info(
"Waiting for sign-in... The browser closes automatically once sign-in is detected (timeout: 5 minutes).",
);

await waitForSignIn(context, page);
logger.success("Signed in to Google Maps. Session saved.");
const outcome = await waitForSignIn(context, page);
if (outcome === "browser-closed") {
logger.warn(
"Browser closed before sign-in was detected. Run `revcli auth status` to check the saved session.",
);
return;
}
logger.success("Signed in to Google Maps. Session saved for later runs.");
} catch (err) {
// Commander does not await async action handlers, so an escaping
// UnrecoverableError (the 5-minute timeout) would surface as an unhandled
// rejection with a raw stack trace.
if (err instanceof UnrecoverableError) {
logger.error(err.message);
process.exitCode = 1;
return;
}
throw err;
} finally {
await closeBrowser(context);
}
Expand Down
86 changes: 74 additions & 12 deletions src/scraper/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -120,25 +120,36 @@ export interface SignInWaitOptions {
pollIntervalMs?: number;
}

/** How a `waitForSignIn` poll loop ended. */
export type SignInOutcome = "signed-in" | "browser-closed";

/**
* Poll until the user has signed in, re-resolving the active page each tick
* (sign-in can open a new tab/window or close the original page) and using the
* authoritative `isSignedIn`/`isGoogleAuthUrl` checks. Shared by
* `waitForUserAuth` and the `auth login` command so the two poll loops never
* diverge. Throws a typed `UnrecoverableError("AUTH_POLL", …)` on a closed
* context or timeout instead of leaking a raw "Target closed" error.
* diverge.
*
* Resolves `"signed-in"` once an authenticated Maps page is seen, or
* `"browser-closed"` when the user closes the window first – the caller
* decides whether that is a failure (the scrape flow, which cannot continue)
* or a normal end (the `auth` command). Throws a typed
* `UnrecoverableError("AUTH_POLL", …)` on timeout instead of leaking a raw
* "Target closed" error.
*/
export async function waitForSignIn(
context: BrowserContext,
initialPage: Page,
options: SignInWaitOptions = {},
): Promise<void> {
): Promise<SignInOutcome> {
const { timeoutMs = 300000, pollIntervalMs = 2000 } = options;
const startTime = Date.now();

while (Date.now() - startTime < timeoutMs) {
try {
const pages = context.pages();
// The user closed the browser window instead of signing in.
if (pages.length === 0) return "browser-closed";
const activePage = pages[pages.length - 1] ?? initialPage;
await activePage.waitForTimeout(pollIntervalMs);

Expand All @@ -151,7 +162,7 @@ export async function waitForSignIn(
url.includes("myaccount.google.com")
) {
if (url.includes("google.com/maps")) {
if (await isSignedIn(activePage)) return;
if (await isSignedIn(activePage)) return "signed-in";
continue;
}
// Sign-in completed on another Google page – return to Maps to confirm.
Expand All @@ -161,15 +172,10 @@ export async function waitForSignIn(
});
await handleConsent(activePage);
await activePage.waitForTimeout(3000);
if (await isSignedIn(activePage)) return;
if (await isSignedIn(activePage)) return "signed-in";
}
} catch (err) {
if (isClosedContext(err)) {
throw new UnrecoverableError(
"AUTH_POLL",
`Sign-in poll aborted: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (isClosedContext(err)) return "browser-closed";
logger.debug(
`Sign-in poll error (will retry): ${err instanceof Error ? err.message : String(err)}`,
);
Expand All @@ -182,6 +188,54 @@ export async function waitForSignIn(
);
}

/**
* Minimal structural view of a browser context. `waitForBrowserClose` only
* needs to observe whether any page is still open and to react to the
* context's `close` event; narrowing the parameter keeps the helper testable
* without a real browser. `BrowserContext` satisfies it structurally.
*/
export interface ClosableBrowser {
pages(): readonly unknown[];
once(event: "close", listener: () => void): unknown;
off(event: "close", listener: () => void): unknown;
}

/**
* Resolves once the user has closed the browser window.
*
* Used by `revcli auth` when the profile is already authenticated: there is no
* sign-in to detect, so the window stays open for switching accounts or
* signing in again, and the user closing it is the only end state. Polls
* `pages()` in addition to the `close` event because closing the last window
* on macOS leaves the browser process – and therefore the context – alive.
*/
export async function waitForBrowserClose(
browser: ClosableBrowser,
pollIntervalMs = 1000,
): Promise<void> {
let closed = false;
const onClose = () => {
closed = true;
};
browser.once("close", onClose);

try {
while (!closed && !hasNoPages(browser)) {
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
} finally {
browser.off("close", onClose);
}
}

function hasNoPages(browser: ClosableBrowser): boolean {
try {
return browser.pages().length === 0;
} catch {
return true;
}
}

/**
* Waits for the user to sign in to Google Maps interactively.
* Opens the Google sign-in page and polls until authentication is detected.
Expand All @@ -203,6 +257,14 @@ export async function waitForUserAuth(page: Page): Promise<void> {
timeout: 30000,
});

await waitForSignIn(page.context(), page);
const outcome = await waitForSignIn(page.context(), page);
// A scrape cannot continue without a session, so a closed window is fatal
// here even though `revcli auth` treats it as a normal end.
if (outcome === "browser-closed") {
throw new UnrecoverableError(
"AUTH_POLL",
"Browser closed before sign-in was detected.",
);
}
logger.success("Sign-in detected – continuing scrape.");
}
Loading
Loading