Skip to content

fix(data): backfill 37 confirmed GSMArena URLs via verify tool - #339

Merged
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/gsmarena-backfill-retry
Sep 29, 2026
Merged

Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/gsmarena-backfill-retry

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What

Retry of the GSMArena backfill (the 2026-09-23 rate limit had long expired).
Upgrades kaggle-only smartphone records (Tier 3, variant.source_category == "gsmarena-kaggle") to a confirmed GSMArena Tier-2 spec-page URL, using the
existing app.verify.gsmarena_backfill tool in TechEngine. The tool's strict
multi-signal gate (sitemap identity + heading match + liveness + spec agreement
on battery/RAM/display/weight) decides every write; AMBIGUOUS / NOTFOUND /
CONTRADICT are reported, never written. Matches were not hand-authored and the
gate was not loosened.

37 records received a confirmed source_urls upgrade before the run was
stopped by a fresh rate limit (see below). Commits:

  1. fix(data): backfill 37 confirmed GSMArena URLs via verify tool — the
    data/smartphone/** source_urls edits.
  2. chore(dump): refresh smartphone dump pages for GSMArena URL backfill — the
    mandatory dump refresh (python -m app.dump from a TechEngine checkout, fresh
    scratch DATABASE_URL, TECHAPI_DATA_DIR pointed at this worktree). Only the
    37 affected site/public/v1/smartphones/<slug>/index.json detail pages
    changed; unrelated score-percentile/schema churn from a fresh full dump was
    excluded so this PR carries only the backfill's effect.

Rate limit (run stopped early, not retried)

The apply run made real progress (the 37 confirmed writes above) and then hit a
GSMArena 429 with Retry-After: 36000s (10 hours). The tool stopped cleanly
before sending any further requests — no retry spinning — and left the remaining
kaggle-only records eligible for a later invocation. A dry-run of 20 immediately
beforehand was clean (19 CONFIRM / 1 NOTFOUND, no 429), so the limit was tripped
during this apply run rather than being a pre-existing block.

Unresolved this run (correctly not written): Caterpillar B15 (NOTFOUND,
no-index-hit), Lenovo A60+ (AMBIGUOUS, ambiguous-index), Sharp Aquos Crystal 2 (NOTFOUND, not-live).

Tool summary

# GSMArena backfill

- records: 57 across 57 brands
- index phones: 13849
- http requests this process: 37
- sleep between requests: 1.5s
- resumed from cache: 20
- dry-run: False

## Decisions

| decision   | count | ratio |
| ---------- | ----: | ----: |
| CONFIRM    |    54 | 94.7% |
| AMBIGUOUS  |     1 |  1.8% |
| NOTFOUND   |     2 |  3.5% |
| CONTRADICT |     0 |  0.0% |

(54 CONFIRM proposed across this run's window; 37 resulted in a new working-tree
write — the remainder already carried the confirmed URL from a prior session's
cache, so the idempotent write guard left them untouched.)

Verification

  • python -m app.validate (bundled validator) — ✅ Data validation passed.
  • Dump regenerated with python -m app.dump; the 37 committed detail pages are
    clean source_urls appends matching the data edits.

Closes #296

Upgrades 37 kaggle-only smartphone records (Tier 3, source_category gsmarena-kaggle) to a confirmed GSMArena Tier-2 spec-page URL. Each URL was confirmed by app.verify.gsmarena_backfill's strict multi-signal gate (sitemap identity + heading match + liveness + spec agreement); AMBIGUOUS/NOTFOUND/CONTRADICT results were reported, not written.

Refs #296
Regenerates the published static dump (site/public/v1/smartphones) for the 37 records upgraded in the previous commit, via python -m app.dump from a TechEngine checkout against this worktree's data. Only the source_urls of the affected detail pages changed; unrelated dump churn was excluded.

Refs #296
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026
@Seungpyo1007
Seungpyo1007 merged commit 2b4639b into develop Sep 29, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the Seungpyo1007/gsmarena-backfill-retry branch September 29, 2026 14:48
@Seungpyo1007 Seungpyo1007 removed the enhancement New feature or request label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

1 participant