Skip to content

fix(data): resolve 364 smartphone brand-prefix conflicts - #330

Merged
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/smartphone-prefix-conflicts
Sep 29, 2026
Merged

Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/smartphone-prefix-conflicts

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

Summary

Follow-up to #329. That PR merged the 50 brand-prefix pairs whose core specs were
already identical and left 717 pairs that disagreed on a core field
(soc / release_date / ram_gb / battery_mah / weight_g / os) documented
but unresolved. This PR resolves the largest, highest-confidence slice.

How the 717 break down

Every pair is a bare slug (dx650, from the Kaggle gsmarena-mobile-devices dump,
carrying a raw_gsmarena block) vs a brand-prefixed twin (acer-dx650). Conflict
fields across the 717: soc 572, release_date 498, weight_g 159, battery_mah 83,
ram_gb 62, os 14.

The soc conflict is almost never a real spec difference — both slugs resolve to a
real soc record, and in 414 pairs they are the same physical chip under a
different slug spelling (qualcomm-msm7227-snapdragon-s1 vs qualcomm-msm7227,
qualcomm-msm8260a-snapdragon-s4-plus vs qualcomm-snapdragon-s4-msm8260a). The
remaining conflicts on those pairs are the fabricated bulk-import placeholders
(weight_g=150.0, battery_mah=1500, generic YYYY-01-01 date) that #329 flagged.

What this PR changes: 364 merges

The 364 same-chip pairs sourced Kaggle-GSMArena (bare) vs phonedb.net (prefixed).
Keeper = the GSMArena record (real specs, richer, has the raw_gsmarena provenance
block). For each:

  • soc: kept the keeper's slug — confirmed FK-valid for all 364, same physical chip.
  • weight_g / battery_mah: kept the keeper's GSMArena value; the phonedb twin's
    150.0 g / 1500 mAh are the stamped placeholders. In all 364 the keeper already
    held the real value, so no scalar was rewritten — the placeholder only ever
    lived on the deleted twin. Spot-verified: Acer DX650 = 133 g (GSMArena,
    imei.org, softpedia) not phonedb's 150.0; Acer CloudMobile S500 = 122 g.
  • name: upgraded to the brand-inclusive form (Acer DX650 over DX650).
  • source_urls: unioned, so the phonedb.net reference is retained.
  • verified: left unchanged — deliberately not flipped to true, because the
    phonedb side's scalars proved unreliable, so it isn't a trustworthy source.
  • release_date: left as the keeper's value and not counted as resolved.
    phonedb's specific dates are unreliable (it lists Acer DX650 as 2009-03; GSMArena
    and imei.org say 2009-06), so importing them would be guessing. Reported as
    still-unresolved rather than changed.
  • The redundant phonedb twin file is deleted.

No value was invented.

Counts (this PR)

  • resolved-and-merged: 364 (same-chip Kaggle-vs-phonedb duplicates)
  • confirmed-different-devices-not-duplicates: 0 in this PR (the ~158 different-chip
    pairs, which include the genuine variants, are being triaged separately)
  • still-unresolved-no-source (carried to follow-up): 353 — 37 same-chip pairs
    sourced vs vendor/Wikipedia pages, 157 needing a per-record ram/weight/battery
    source, 158 different-chip pairs (some genuinely different devices to un-pair),
    plus the release_date field on the 364 merged here.

Validation

  • python -m app.validate → passed.
  • integrity_check.py <data> --strict → no hard anomalies (0 duplicate slugs,
    0 slug/file mismatches, 0 verified-without-source).
  • Post-merge: 92,994 smartphone files (−364), zero duplicate slugs.

Dump

Last commit refreshes site/public/v1 from the merged data (python -m app.dump,
no re-scrape): removes the 364 deleted slugs and reconciles score percentiles.

Closes #296

…uplicates)

Follow-up to #329, which merged the 50 identical-core brand-prefix pairs and left
717 pairs that disagreed on a core field (soc / release_date / ram_gb / battery_mah
/ weight_g / os) documented but unresolved. This resolves the largest, highest-
confidence slice of those.

Classification of the 717 (scripted, then spot-verified against sources):
- The soc "conflict" in 572 pairs is two DIFFERENT slugs that both resolve to a
  real soc record. For 414 of those the two slugs are the SAME physical chip with
  a different spelling (e.g. "qualcomm-msm7227-snapdragon-s1" vs "qualcomm-msm7227",
  "qualcomm-msm8260a-snapdragon-s4-plus" vs "qualcomm-snapdragon-s4-msm8260a") — not
  a real spec difference.
- The other conflicts on those same-chip pairs are the fabricated import placeholders
  (weight_g=150.0, battery_mah=1500, and the generic YYYY-01-01 launch date) that
  #329 already flagged as a systemic bulk-import bug.

This commit merges the 364 same-chip pairs whose two sides are a GSMArena scrape
(bare slug, Kaggle `gsmarena-mobile-devices`, carries a `raw_gsmarena` block) and a
phonedb.net twin (brand-prefixed slug). Keeper = the GSMArena record:
- Its specs are the real ones. Spot-verified: Acer DX650 weighs 133 g
  (GSMArena, imei.org, softpedia) — the phonedb twin's 150.0 g is the stamped
  placeholder; Acer CloudMobile S500 is 122 g (GSMArena) vs phonedb's 150.0.
- soc kept as the keeper's slug (confirmed FK-valid for all 364; same physical chip).
- name upgraded to the brand-inclusive form ("Acer DX650" over "DX650").
- source_urls unioned so the phonedb.net reference is retained.
- verified left unchanged — NOT flipped to true, because the phonedb side's scalar
  values proved unreliable (fabricated weights), so it is not a trustworthy source.
- release_date left as the keeper's value and NOT resolved here: the phonedb twin's
  specific dates are unreliable too (phonedb says Acer DX650 = 2009-03 but
  GSMArena/imei say 2009-06), so importing them would be guessing. Dates are
  reported as still-unresolved rather than changed.
The redundant phonedb twin file is deleted. No scalar value was invented; in every
one of these 364 pairs the keeper already held the real GSMArena value and only the
deleted twin carried the placeholder.

Left for follow-up (documented on #296): 37 same-chip pairs sourced against vendor
/Wikipedia pages rather than phonedb, 157 pairs with a genuine ram/weight/battery
question needing per-record sourcing, and 158 pairs whose soc slugs are two
different chips — several of which are genuinely different devices that share a
name (e.g. Honor Magic V2 SD8Gen2 vs Kirin 9000S, Huawei P50 Pro SD888 vs Kirin
9000, HTC One M9 Helio X10 vs Snapdragon 810) and will be un-paired, not merged.

Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies).

Closes #296
…flicts

Regenerates site/public/v1 from the current data with no re-scrape
(python -m app.dump). Removes the 364 deleted phonedb-twin slugs and reconciles
score percentiles across the scored catalog. `app.validate` passes on the dumped
tree.
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026
@Seungpyo1007
Seungpyo1007 merged commit 9ce1979 into develop Sep 29, 2026
5 checks passed
Seungpyo1007 added a commit that referenced this pull request Sep 29, 2026
Continues the brand-prefix dedup from #329/#330. Those resolved 414 pairs and
carried forward a residual of 353. This resolves 170 of that residual, source-
verified, and explicitly un-pairs 3 that are genuinely different devices.

Three sub-slices resolved (Refs #296):

1. 37 vendor/Wikipedia-sourced same-chip pairs. Same device, same physical chip
   (soc slugs differ only in spelling, e.g. `qualcomm-msm8974-snapdragon-800` vs
   `snapdragon-800`, `hisilicon-kirin-980` vs `kirin-980`), with battery/weight/ram
   already matching. Unlike the phonedb twins in #330, the twin here carries a
   Wikipedia or vendor page as its source, so its specific release_date is reliable
   and was adopted over the bare side's generic YYYY-01-01 placeholder (spot-checked:
   Huawei P30 Pro 2019-03-26 = GSMArena; LG G2 September 2013 = Wikipedia). Kept the
   canonical bare slug, unioned source_urls, set verified=true only where the twin
   was a vendor/Wikipedia source.

2. 7 same-model pairs whose two sides named different chips — verified against a
   cited source which side is correct, fixed the soc, then merged:
   - Oppo Find X8 Pro -> Dimensity 9400 (oppo.com launch; GSMArena)
   - Motorola Edge 50 Ultra -> Snapdragon 8s Gen 3 (motorola.com; phonearena)
   - Honor Magic V3 -> Snapdragon 8 Gen 3 (phonearena SM8650-AB; honor.com)
   - OnePlus Nord CE 2 Lite (Snapdragon 695), Nord CE 2 (Dimensity 900),
     Sony Xperia Z (APQ8064 S4 Pro), Honor Magic V2 (Snapdragon 8 Gen 2) —
     the bare side already held the correct chip; merged and cited GSMArena/vendor.

3. 126 pairs where the soc slugs are the same physical chip via a spelling variant,
   or one side's soc was a vague descriptor (`dual-core 1.0 GHz`, `<brand> mobile
   platform`, `n-a`) and the twin named the specific FK-valid part. Kept the
   GSMArena (bare) record, adopted the specific soc when the keeper's was vague,
   rejected the phonedb stamped placeholders (weight 150.0 / battery 1500), left the
   (unreliable) phonedb dates alone, unioned source_urls, deleted the twin.

Un-paired as genuinely different devices (left as separate records, no merge):
- HTC: `one-m9` holds the One M9+ (Helio X10) vs the One M9 (Snapdragon 810).
- Motorola: `razr` holds the 2023 Razr (SD 8+ Gen 1) vs the 2019 Razr (SD 710).
- Sony: `xperia-z3` holds the Xperia Z3+ (SD 810) vs the Xperia Z3 (SD 801).

No value invented; every repaired field cites its evidence URL in source_urls.
Left unresolved for a later pass (documented on #296): 21 same-model pairs whose
correct chip I could not source this round (includes the Huawei P50 Pro, which
genuinely shipped both Kirin 9000 and Snapdragon 888 SKUs), 157 pairs needing a
per-record ram/weight/battery source, and 2 odd/other pairs.

Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies).

Refs #296
Seungpyo1007 added a commit that referenced this pull request Sep 29, 2026
Continues the brand-prefix dedup from #329/#330. Those resolved 414 pairs and
carried forward a residual of 353. This resolves 170 of that residual, source-
verified, and explicitly un-pairs 3 that are genuinely different devices.

Three sub-slices resolved (Refs #296):

1. 37 vendor/Wikipedia-sourced same-chip pairs. Same device, same physical chip
   (soc slugs differ only in spelling, e.g. `qualcomm-msm8974-snapdragon-800` vs
   `snapdragon-800`, `hisilicon-kirin-980` vs `kirin-980`), with battery/weight/ram
   already matching. Unlike the phonedb twins in #330, the twin here carries a
   Wikipedia or vendor page as its source, so its specific release_date is reliable
   and was adopted over the bare side's generic YYYY-01-01 placeholder (spot-checked:
   Huawei P30 Pro 2019-03-26 = GSMArena; LG G2 September 2013 = Wikipedia). Kept the
   canonical bare slug, unioned source_urls, set verified=true only where the twin
   was a vendor/Wikipedia source.

2. 7 same-model pairs whose two sides named different chips — verified against a
   cited source which side is correct, fixed the soc, then merged:
   - Oppo Find X8 Pro -> Dimensity 9400 (oppo.com launch; GSMArena)
   - Motorola Edge 50 Ultra -> Snapdragon 8s Gen 3 (motorola.com; phonearena)
   - Honor Magic V3 -> Snapdragon 8 Gen 3 (phonearena SM8650-AB; honor.com)
   - OnePlus Nord CE 2 Lite (Snapdragon 695), Nord CE 2 (Dimensity 900),
     Sony Xperia Z (APQ8064 S4 Pro), Honor Magic V2 (Snapdragon 8 Gen 2) —
     the bare side already held the correct chip; merged and cited GSMArena/vendor.

3. 126 pairs where the soc slugs are the same physical chip via a spelling variant,
   or one side's soc was a vague descriptor (`dual-core 1.0 GHz`, `<brand> mobile
   platform`, `n-a`) and the twin named the specific FK-valid part. Kept the
   GSMArena (bare) record, adopted the specific soc when the keeper's was vague,
   rejected the phonedb stamped placeholders (weight 150.0 / battery 1500), left the
   (unreliable) phonedb dates alone, unioned source_urls, deleted the twin.

Un-paired as genuinely different devices (left as separate records, no merge):
- HTC: `one-m9` holds the One M9+ (Helio X10) vs the One M9 (Snapdragon 810).
- Motorola: `razr` holds the 2023 Razr (SD 8+ Gen 1) vs the 2019 Razr (SD 710).
- Sony: `xperia-z3` holds the Xperia Z3+ (SD 810) vs the Xperia Z3 (SD 801).

No value invented; every repaired field cites its evidence URL in source_urls.
Left unresolved for a later pass (documented on #296): 21 same-model pairs whose
correct chip I could not source this round (includes the Huawei P50 Pro, which
genuinely shipped both Kirin 9000 and Snapdragon 888 SKUs), 157 pairs needing a
per-record ram/weight/battery source, and 2 odd/other pairs.

Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies).

Refs #296
Seungpyo1007 added a commit that referenced this pull request Sep 29, 2026
Scanned the pda category (140 records, reviewed every match by hand) for the
three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330)
passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112),
keeping the better-sourced side field by field and unioning source_urls.

Pattern 1 - identical scrape imported twice under different row-id slugs
(100% identical specs, differ only by source_row):
  - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227.
  - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161.

Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling
dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare
GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia),
upgraded name to the brand-inclusive form, unioned source_urls to retain the
phonedb reference, kept the keeper's soc/release_date, left verified as-is,
deleted the phonedb twin:
  - treo-680  <- palm-treo-680-palm-lowrider   ("Palm Treo 680")
  - treo-750v <- palm-treo-750v-htc-cheetah    ("Palm Treo 750v")
  - treo-750  <- palm-treo-750-htc-cheetah     ("Palm Treo 750")
  - treo-500v <- palm-treo-500v-palm-otto      ("Palm Treo 500v")
Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the
matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a
different device and was left alone.

Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug
record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean
record (correct soc FK, canonical slug), deleted the scrape twin, and repaired
the keeper's ram_gb from the twin's cited memory_internal (the clean side
carried the fabricated ram_gb=1 placeholder; the scrape side had the real
value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired
RAM is sourced:
  - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB);
    hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/
    voice-messenger -> 0.125 GB (128 MB).
  - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB.
  - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB.
  - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro):
    kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name ->
    "Palm Treo Pro", unioned all four source_urls.

Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the
GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a
region-split family (palm-pre[-plus]-palm-castle-c CDMA and
palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge.
The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm,
pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone.

No value was invented: every ram_gb repair took the real figure already present
in the deleted twin's cited memory_internal. verified flags unchanged.

Both gates pass: `python -m app.validate` and `integrity_check.py <data>
--strict` (0 hard anomalies).

Refs #296
Seungpyo1007 added a commit that referenced this pull request Sep 29, 2026
… base slug

The brand-prefix scan across #329/#330/#334 kept flagging some pairs as duplicates
that are actually two DIFFERENT products which happen to share a base name — the
mechanical slug-matching couldn't tell them apart. A focused pass for that specific
pattern (name divergence, not the "same device / data conflict" or "needs a source"
patterns already handled) found 7, each confirmed against a source as a distinct
real device. Un-paired by giving the mis-slugged variant a distinct slug/filename
/name so it is no longer a false duplicate of the base model. No merges; both
records stay as separate devices.

- HTC U11+ vs U11 — distinct fall-2017 model, smaller bezels, SD835/6GB/3930mAh vs
  the summer U11 (androidcentral). `u11` -> `u11-plus` (name "HTC U11+").
- HTC One M9+ vs One M9 — the M9+ uses a Helio X10 and QHD display, separate from
  the SD810 One M9. `one-m9` -> `one-m9-plus`.
- Huawei P40 Pro+ vs P40 Pro — distinct higher model (pocket-lint, CNET).
  `p40-pro` -> `p40-pro-plus`.
- Sony Xperia Z3+ vs Xperia Z3 — the Z3+ (a.k.a. Xperia Z4) is a distinct 2015
  Snapdragon 810 model with its own Wikipedia page, vs the 2014 Z3 (Snapdragon 801).
  `xperia-z3` -> `xperia-z3-plus`.
- Motorola Razr (2023) vs Razr (2019) — different foldable generations (SD 8+ Gen 1
  vs SD 710). `razr` -> `razr-2023`.
- Fairphone 3+ (2020) vs Fairphone 3 (2019) — successor model. The 3+ record
  -> `fairphone-3-plus-dual-sim-lte-fp3plus`.
- Gigabyte GSmart GS202+ vs GS202 — distinct + variant. The + record
  -> `gsmart-gs202-plus`.

Each renamed record keeps its own specs and sources untouched (nothing merged,
nothing invented); only slug, base_model_slug, name and filename change so the two
devices no longer collide as a false pair. New slugs are collision-free.

Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies). Residual brand-prefix pairs drop from 183 to 176.

Refs #296
@Seungpyo1007
Seungpyo1007 deleted the Seungpyo1007/smartphone-prefix-conflicts branch September 30, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

1 participant