Skip to content

fix(data): merge 28 confirmed PDA near-duplicate records - #335

Merged
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/pda-dedup-scan
Sep 29, 2026
Merged

Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/pda-dedup-scan

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What

Near-duplicate scan of the pda category (140 records — small enough to review
every match by hand). Merged 28 confirmed duplicate records into 21 keepers
(140 → 112 files), keeping the better-sourced side field by field and unioning
source_urls. Anything ambiguous was left unmerged and is documented below.

Both gates pass on the merged tree:

  • python -m app.validate → passed
  • integrity_check.py <data> --strict → 0 hard anomalies (structural section empty)

Per-pattern breakdown

The same three patterns the tablet/watch (#324) and smartphone (#329/#330) passes
turned up all appear here.

Pattern 1 — identical scrape imported twice under different row-id slugs

Two cigarplug/scrape-gsma rows for the same device, byte-identical except
source_row. Kept the lower row-id, deleted the other (no field change needed):

  • motorola/2002/accompli-388 — kept …-scrapegsma-344, deleted …-1227
  • mitac/2005/mio-a701 — kept …-scrapegsma-278, deleted …-1161

Pattern 2 — same device under a brand-prefixed slug and a bare slug (sibling dirs)

Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle
record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the
brand-inclusive form, unioned source_urls to retain the phonedb reference, kept
the keeper's soc/release_date, left verified as-is, deleted the phonedb twin:

Keeper (kept) Deleted twin New name
treo-680 palm-treo-680-palm-lowrider Palm Treo 680
treo-750v palm-treo-750v-htc-cheetah Palm Treo 750v
treo-750 palm-treo-750-htc-cheetah Palm Treo 750
treo-500v palm-treo-500v-palm-otto Palm Treo 500v

Treo 750 vs 750v and Treo 500 vs 500v are distinct SKUs — only the
matching-model twins were merged. palm-treo-500-palm-otto (2008 Treo 500) is a
different device and was left untouched.

Pattern 3 — cross-source duplicate (clean short-slug record + scrape-gsma row, same base dir)

Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin,
and repaired the keeper's ram_gb from the twin's cited memory_internal — the
clean side carried the fabricated ram_gb = 1 placeholder while the scrape side had
the real figure. github.com/cigarplug/scrape-gsma was unioned so the repaired RAM
is sourced. No value invented: in every pair the deleted twin already held the real
value.

  • 16 HP iPAQ pairs — h6310/h6315/h6320/h6325/hw6510/hw6515/hw6910/hw6915/rw6815/
    rw6818/rw6828/514 → 0.0625 GB (64 MB); 610c/910c/data-messenger/voice-messenger
    → 0.125 GB (128 MB).
  • i-mate — pda2 (two identical scrape twins -1004/-121) → 0.125 GB;
    pda2k → 0.125 GB; pdal → 0.062 GB.
  • palm treo-pro (3-way: bare Wikipedia-cited + …-scrapegsma-7066 + phonedb
    palm-treo-pro) — kept the bare record, ram_gb 1 → 0.125 GB (128 MB), name →
    "Palm Treo Pro", unioned all four source_urls.

Left unmerged (ambiguous — documented, not guessed)

  • palm-pre-scrapegsma-7065 (GSM webOS Pre, 2009-10) and palm-pre-plus-scrapegsma-7063
    (GSM Pre Plus, 2010-03) overlap a region-split family — palm-pre[-plus]-palm-castle-c
    (CDMA) and palm-pre[-plus]-gsm-{eu,na}-palm-castle-g — but do not map 1:1, so
    no merge was made.
  • The EU/NA regional variant pairs (palm-pixi-gsm, palm-pixi-plus-gsm,
    palm-pre-2-gsm, palm-pre[-plus]-gsm-…-castle-g) are legitimate separate SKUs
    (different release dates + regions) and were left alone.

Systemic finding (reported, not mass-edited)

The same fabricated-defaults signature the earlier passes flagged is present here:
nearly every clean pda record carries ram_gb = 1 (real values are 16–256 MB),
and many Palm records carry os = "Feature phone" for what are Palm OS / webOS
devices. I only repaired ram_gb inside the confirmed merges (where the deleted
twin supplied a cited real value); the remaining placeholder records were left
untouched rather than mass-edited, since fixing them needs a per-record source and
is out of scope for a dedup pass (per the "never invent values" rule).

Ownership / dump

data/pda/ only, plus the mandatory dump refresh as the last commit (re-dumped the
pdas collection from a fresh scratch DB with TECHAPI_DATA_DIR on this worktree;
pdas 140 → 112, no other collection touched).

Closes #296 — this clears the pda slice of the "near-duplicates in other categories
not yet scanned" item, but that item stays open (smartphone still has 322
documented-unresolved pairs per #330, and other categories are not yet scanned), so
this is Refs, not Closes.

Scanned the pda category (140 records, reviewed every match by hand) for the
three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330)
passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112),
keeping the better-sourced side field by field and unioning source_urls.

Pattern 1 - identical scrape imported twice under different row-id slugs
(100% identical specs, differ only by source_row):
  - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227.
  - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161.

Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling
dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare
GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia),
upgraded name to the brand-inclusive form, unioned source_urls to retain the
phonedb reference, kept the keeper's soc/release_date, left verified as-is,
deleted the phonedb twin:
  - treo-680  <- palm-treo-680-palm-lowrider   ("Palm Treo 680")
  - treo-750v <- palm-treo-750v-htc-cheetah    ("Palm Treo 750v")
  - treo-750  <- palm-treo-750-htc-cheetah     ("Palm Treo 750")
  - treo-500v <- palm-treo-500v-palm-otto      ("Palm Treo 500v")
Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the
matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a
different device and was left alone.

Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug
record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean
record (correct soc FK, canonical slug), deleted the scrape twin, and repaired
the keeper's ram_gb from the twin's cited memory_internal (the clean side
carried the fabricated ram_gb=1 placeholder; the scrape side had the real
value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired
RAM is sourced:
  - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB);
    hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/
    voice-messenger -> 0.125 GB (128 MB).
  - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB.
  - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB.
  - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro):
    kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name ->
    "Palm Treo Pro", unioned all four source_urls.

Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the
GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a
region-split family (palm-pre[-plus]-palm-castle-c CDMA and
palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge.
The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm,
pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone.

No value was invented: every ram_gb repair took the real figure already present
in the deleted twin's cited memory_internal. verified flags unchanged.

Both gates pass: `python -m app.validate` and `integrity_check.py <data>
--strict` (0 hard anomalies).

Refs #296
Re-dumped the pdas collection from the deduped data tree (fresh scratch DB, TECHAPI_DATA_DIR on this worktree). pdas 140 -> 112: 28 merged-away detail pages removed, 21 survivor pages updated (repaired ram_gb, brand-inclusive names, unioned source_urls), and the top-level manifest pda count updated. No other collection touched.
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026
@Seungpyo1007
Seungpyo1007 merged commit e5e18a1 into develop Sep 29, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the Seungpyo1007/pda-dedup-scan branch September 29, 2026 12:47
@Seungpyo1007 Seungpyo1007 removed the enhancement New feature or request label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

1 participant