[oBusk]

Speculative — for discussion.

Why

Diff pages are an unbounded URL space (every package × version pair), and each uncached one runs libnpmdiff (up to 40–70s CPU for large packages). Catalog pages link to "Suggested Diffs", so crawlers spider the whole graph.

What

• robots.txt disallows /*... (every diff URL) for *, and allows a curated list.
• Curated list = latest major and latest minor bump (same generateComparisons logic as the catalog page) for 12 popular, small packages (CRAWLABLE_PACKAGES in src/lib/crawlableDiffs.ts) → 24 URLs.
• Cached with cacheLife("days"); build shows /robots.txt as static with 1d revalidate.
• Catalog (package) pages stay crawlable — they're cheap and are the main SEO surface.
• Google/Bing use the most specific match, so the longer Allow lines win over Disallow: /*....

Generated output at build time:

User-Agent: *
Allow: /
Allow: /react@18.3.1...react@19.0.0
Allow: /react@19.2.9...react@19.3.0
Allow: /express@4.22.3...express@5.0.0
...
Disallow: /api/
Disallow: /*...
Crawl-delay: 10

Open questions

• Should any diffs be crawlable at all? Check Search Console for diff URLs that actually get clicks; if none, drop the allow list.
• Package list is hand-picked (avoids heavy packages like typescript/react-dom). Could instead use top-N by downloads from the Algolia npm-search index.
• Only compliant crawlers are affected; the firewall handles the rest.
• %40-encoded diff URLs still contain ... so they're covered by the disallow.

lint, typecheck, test-ci (315/315, incl. new crawlableDiffs.test.ts), build pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xk9y1TZXZcrUoUrCUaLGKp