fix(data): merge 50 confirmed smartphone near-duplicate records by Seungpyo1007 · Pull Request #329 · GetTechAPI/TechAPI

Seungpyo1007 · 2026-09-29T07:40:56Z

Summary

Scanned all 93,407 data/smartphone records for the three near-duplicate
patterns the tablet/watch pass (#324) found, and merged the confirmed cases. This
closes the "near-duplicates in other categories not yet scanned" item on #296 for
the smartphone category.

Method mirrors #324: normalize slugs/names (ignore punctuation and brand-prefix
variance), group by (brand, base-slug), then cross-check core specs
(release_date, soc, ram_gb, battery_mah, weight_g, os) to separate
genuine duplicates from similar-sounding but distinct SKUs. Nothing ambiguous was
merged.

Per-pattern breakdown

Brand-prefix siblings — the dominant pattern. Found 767 pairs where the same
brand+model exists both under a bare slug (nord-n100) and a brand-prefixed twin
(oneplus-nord-n100). Split by whether the core specs actually agree:

	pairs	action
identical core specs (true duplicates)	50	merged
core specs disagree (see placeholder note)	717	left untouched, documented

Identical GSMArena scrapes under two row-id slugs and generic cross-source
dupes: no additional clean cases beyond the 50 above once the brand-prefix
grouping was applied — the bulk imports in this category collide through the
brand-prefix axis, not through independent row-ids.

The 50 merges

Keeper is always the shorter, non-brand-prefixed slug (same convention as the Intel
Atom CPU dedup in #296), kept at its existing path. Two sub-patterns:

phonedb.net × sady36-kaggle (allview, blackview, blu, caterpillar, doogee,
lava, qmobile, ulefone, unihertz, zte): the bare side is the richer scrape (cameras,
connectivity) but verified: false; the prefixed twin is the phonedb.net record
(verified: true). Kept the richer bare record, unioned source_urls so the
phonedb.net citation is retained, and set verified: true.
oneplus / xiaomi: the prefixed side carried the fuller fields (connectivity,
display resolution/refresh, wired-charging). Carried those into the bare keeper and
unioned source_urls.

One real spec conflict found hiding inside an apparent duplicate: OnePlus 5T Star
Wars Edition had a main camera of 20 MP on one side and 16 MP on the other. GSMArena
and OnePlus list the 5T main sensor as 16 MP (Sony IMX398) with the 20 MP as the
secondary — corrected to 16 MP and added the GSMArena review as a source. This is
exactly the "don't blind-merge, verify each field" lesson from the earlier CPU/tablet
work.

Each merged record keeps the union of both sides' source_urls; the 50 redundant
brand-prefixed files are deleted; base_model_slug realigned to the kept slug.

Reported, not "fixed": a systemic placeholder-value bug

While scanning I found the same fabricated-defaults pattern #324 caught in
tablet/watch. 487 distinct device names across 624 files share the exact triple
battery_mah=1500, weight_g=150.0, ram_gb=1 — spanning 1994–2020 and including
watches and tablets, overwhelmingly from the gsmarena-mobile-devices Kaggle import.
A 1994 phone, a 2019 phone and a smartwatch cannot all genuinely have identical
1500 mAh / 150 g / 1 GB specs; these are stamped defaults, not real values.

Related placeholder triples with the tell-tale weight_g=150.0, ram_gb=1 signature
and a generic YYYY-01-01 date also appear at (2000,150,1)×182, (1800,150,1)×92,
(1000,150,1)×83, (2500,150,1)×80 names.

This is also why 717 of the 767 prefix pairs look like "conflicts": the bare
(Kaggle) side often carries these 1500/150 defaults plus a generic date while the twin
has the real specs. They are almost certainly the same devices, but merging them
correctly requires repairing the fabricated side against a real source per-record —
which would mean inventing/hunting values for hundreds of devices. Per the "never
invent values" rule, that belongs in a dedicated repair pass, not this dedup PR, so
those pairs are left untouched and flagged here.

Validation

python -m app.validate → passed.
integrity_check.py <data> --strict → no hard anomalies (0 duplicate slugs,
0 slug/file mismatches, 0 verified-without-source). The CPU era/ratio and GPU
outliers printed are the pre-existing advisory-only signals, unrelated to
smartphones.
Post-merge tree: 93,357 smartphone files (−50), zero duplicate slugs.

Dump

Last commit refreshes site/public/v1 from the merged data (python -m app.dump,
no re-scrape): removes the 50 deleted slugs and reconciles score percentiles, which
shift slightly across the scored catalog now that the 50 duplicates no longer inflate
the smartphone population.

Closes #296

The strict integrity gate (integrity_check.py --strict) lists "near-duplicates in other categories not yet scanned" as open on #296. Scanned all 93,407 smartphone records for the three duplicate patterns the tablet/watch pass (PR#324) found, using the same method: normalize slugs/names (ignore punctuation and brand-prefix variance), then cross-check core specs (release_date, soc, ram_gb, battery_mah, weight_g, os) to separate genuine duplicates from similar-sounding but distinct SKUs. Found 767 brand-prefix sibling pairs (a bare slug like "nord-n100" and a brand-prefixed twin "oneplus-nord-n100" for the same brand+model). Of these, exactly 50 had byte-identical core specs and are the same device imported twice; the merges here cover only those 50. The remaining 717 pairs are NOT clean duplicates and are left untouched and documented in the PR (see below). Merge rules (keeps the diff reviewable, never invents a value): - Keeper = the shorter, non-brand-prefixed slug (same convention as the Intel Atom CPU dedup), kept at its existing path. - Two sub-patterns: * phonedb.net x sady36-kaggle pairs (allview, blackview, blu, caterpillar, doogee, lava, qmobile, ulefone, unihertz, zte): the bare side is the richer scrape (cameras, connectivity) but was verified=false; the prefixed side is the phonedb.net record (verified=true). Kept the richer bare record, unioned source_urls (so phonedb.net is retained as the human-review citation), set verified=true. * oneplus / xiaomi pairs: the prefixed side carried the fuller fields (connectivity, display resolution/refresh, wired-charging). Carried those over into the bare keeper, unioned source_urls. - One real spec conflict found inside an apparent duplicate: OnePlus 5T Star Wars Edition had main-camera 20MP on one side and 16MP on the other. GSMArena and OnePlus list the 5T main sensor as 16MP (Sony IMX398); the 20MP is the secondary. Corrected to 16MP and added the GSMArena review as a source. - Every merged record keeps the union of both sides' source_urls; the 50 redundant brand-prefixed files are deleted; base_model_slug realigned to the kept slug. Also surfaced (reported, not "fixed" — repairs would require inventing values): a systemic placeholder-value bug. 487 distinct device names across 624 files share the exact triple battery_mah=1500, weight_g=150.0, ram_gb=1 — spanning 1994-2020 and even watches/tablets, mostly from the gsmarena-mobile-devices Kaggle import. This is the same fabricated-defaults pattern (1500/150) PR#324 caught in tablet/watch. It is also what makes 717 of the 767 prefix pairs look like "conflicts": the bare side often carries these 1500/150 defaults plus a generic YYYY-01-01 date while the twin has real specs. Flagged for a dedicated repair pass rather than mass-editing here. Both gates pass on the merged tree: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Closes #296

Regenerates site/public/v1 from the current data with no re-scrape (python -m app.dump). Removes the 50 brand-prefixed duplicate smartphone slugs that were merged away and reconciles the score percentiles, which shift slightly across the scored catalog now that the 50 duplicate records no longer inflate the smartphone population. `app.validate` passes on the dumped tree.

…uplicates) Follow-up to #329, which merged the 50 identical-core brand-prefix pairs and left 717 pairs that disagreed on a core field (soc / release_date / ram_gb / battery_mah / weight_g / os) documented but unresolved. This resolves the largest, highest- confidence slice of those. Classification of the 717 (scripted, then spot-verified against sources): - The soc "conflict" in 572 pairs is two DIFFERENT slugs that both resolve to a real soc record. For 414 of those the two slugs are the SAME physical chip with a different spelling (e.g. "qualcomm-msm7227-snapdragon-s1" vs "qualcomm-msm7227", "qualcomm-msm8260a-snapdragon-s4-plus" vs "qualcomm-snapdragon-s4-msm8260a") — not a real spec difference. - The other conflicts on those same-chip pairs are the fabricated import placeholders (weight_g=150.0, battery_mah=1500, and the generic YYYY-01-01 launch date) that #329 already flagged as a systemic bulk-import bug. This commit merges the 364 same-chip pairs whose two sides are a GSMArena scrape (bare slug, Kaggle `gsmarena-mobile-devices`, carries a `raw_gsmarena` block) and a phonedb.net twin (brand-prefixed slug). Keeper = the GSMArena record: - Its specs are the real ones. Spot-verified: Acer DX650 weighs 133 g (GSMArena, imei.org, softpedia) — the phonedb twin's 150.0 g is the stamped placeholder; Acer CloudMobile S500 is 122 g (GSMArena) vs phonedb's 150.0. - soc kept as the keeper's slug (confirmed FK-valid for all 364; same physical chip). - name upgraded to the brand-inclusive form ("Acer DX650" over "DX650"). - source_urls unioned so the phonedb.net reference is retained. - verified left unchanged — NOT flipped to true, because the phonedb side's scalar values proved unreliable (fabricated weights), so it is not a trustworthy source. - release_date left as the keeper's value and NOT resolved here: the phonedb twin's specific dates are unreliable too (phonedb says Acer DX650 = 2009-03 but GSMArena/imei say 2009-06), so importing them would be guessing. Dates are reported as still-unresolved rather than changed. The redundant phonedb twin file is deleted. No scalar value was invented; in every one of these 364 pairs the keeper already held the real GSMArena value and only the deleted twin carried the placeholder. Left for follow-up (documented on #296): 37 same-chip pairs sourced against vendor /Wikipedia pages rather than phonedb, 157 pairs with a genuine ram/weight/battery question needing per-record sourcing, and 158 pairs whose soc slugs are two different chips — several of which are genuinely different devices that share a name (e.g. Honor Magic V2 SD8Gen2 vs Kirin 9000S, Huawei P50 Pro SD888 vs Kirin 9000, HTC One M9 Helio X10 vs Snapdragon 810) and will be un-paired, not merged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Closes #296

Continues the brand-prefix dedup from #329/#330. Those resolved 414 pairs and carried forward a residual of 353. This resolves 170 of that residual, source- verified, and explicitly un-pairs 3 that are genuinely different devices. Three sub-slices resolved (Refs #296): 1. 37 vendor/Wikipedia-sourced same-chip pairs. Same device, same physical chip (soc slugs differ only in spelling, e.g. `qualcomm-msm8974-snapdragon-800` vs `snapdragon-800`, `hisilicon-kirin-980` vs `kirin-980`), with battery/weight/ram already matching. Unlike the phonedb twins in #330, the twin here carries a Wikipedia or vendor page as its source, so its specific release_date is reliable and was adopted over the bare side's generic YYYY-01-01 placeholder (spot-checked: Huawei P30 Pro 2019-03-26 = GSMArena; LG G2 September 2013 = Wikipedia). Kept the canonical bare slug, unioned source_urls, set verified=true only where the twin was a vendor/Wikipedia source. 2. 7 same-model pairs whose two sides named different chips — verified against a cited source which side is correct, fixed the soc, then merged: - Oppo Find X8 Pro -> Dimensity 9400 (oppo.com launch; GSMArena) - Motorola Edge 50 Ultra -> Snapdragon 8s Gen 3 (motorola.com; phonearena) - Honor Magic V3 -> Snapdragon 8 Gen 3 (phonearena SM8650-AB; honor.com) - OnePlus Nord CE 2 Lite (Snapdragon 695), Nord CE 2 (Dimensity 900), Sony Xperia Z (APQ8064 S4 Pro), Honor Magic V2 (Snapdragon 8 Gen 2) — the bare side already held the correct chip; merged and cited GSMArena/vendor. 3. 126 pairs where the soc slugs are the same physical chip via a spelling variant, or one side's soc was a vague descriptor (`dual-core 1.0 GHz`, `<brand> mobile platform`, `n-a`) and the twin named the specific FK-valid part. Kept the GSMArena (bare) record, adopted the specific soc when the keeper's was vague, rejected the phonedb stamped placeholders (weight 150.0 / battery 1500), left the (unreliable) phonedb dates alone, unioned source_urls, deleted the twin. Un-paired as genuinely different devices (left as separate records, no merge): - HTC: `one-m9` holds the One M9+ (Helio X10) vs the One M9 (Snapdragon 810). - Motorola: `razr` holds the 2023 Razr (SD 8+ Gen 1) vs the 2019 Razr (SD 710). - Sony: `xperia-z3` holds the Xperia Z3+ (SD 810) vs the Xperia Z3 (SD 801). No value invented; every repaired field cites its evidence URL in source_urls. Left unresolved for a later pass (documented on #296): 21 same-model pairs whose correct chip I could not source this round (includes the Huawei P50 Pro, which genuinely shipped both Kirin 9000 and Snapdragon 888 SKUs), 157 pairs needing a per-record ram/weight/battery source, and 2 odd/other pairs. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296

Scanned the pda category (140 records, reviewed every match by hand) for the three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330) passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112), keeping the better-sourced side field by field and unioning source_urls. Pattern 1 - identical scrape imported twice under different row-id slugs (100% identical specs, differ only by source_row): - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227. - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161. Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the brand-inclusive form, unioned source_urls to retain the phonedb reference, kept the keeper's soc/release_date, left verified as-is, deleted the phonedb twin: - treo-680 <- palm-treo-680-palm-lowrider ("Palm Treo 680") - treo-750v <- palm-treo-750v-htc-cheetah ("Palm Treo 750v") - treo-750 <- palm-treo-750-htc-cheetah ("Palm Treo 750") - treo-500v <- palm-treo-500v-palm-otto ("Palm Treo 500v") Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a different device and was left alone. Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin, and repaired the keeper's ram_gb from the twin's cited memory_internal (the clean side carried the fabricated ram_gb=1 placeholder; the scrape side had the real value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired RAM is sourced: - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB); hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/ voice-messenger -> 0.125 GB (128 MB). - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB. - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB. - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro): kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name -> "Palm Treo Pro", unioned all four source_urls. Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a region-split family (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone. No value was invented: every ram_gb repair took the real figure already present in the deleted twin's cited memory_internal. verified flags unchanged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296

… base slug The brand-prefix scan across #329/#330/#334 kept flagging some pairs as duplicates that are actually two DIFFERENT products which happen to share a base name — the mechanical slug-matching couldn't tell them apart. A focused pass for that specific pattern (name divergence, not the "same device / data conflict" or "needs a source" patterns already handled) found 7, each confirmed against a source as a distinct real device. Un-paired by giving the mis-slugged variant a distinct slug/filename /name so it is no longer a false duplicate of the base model. No merges; both records stay as separate devices. - HTC U11+ vs U11 — distinct fall-2017 model, smaller bezels, SD835/6GB/3930mAh vs the summer U11 (androidcentral). `u11` -> `u11-plus` (name "HTC U11+"). - HTC One M9+ vs One M9 — the M9+ uses a Helio X10 and QHD display, separate from the SD810 One M9. `one-m9` -> `one-m9-plus`. - Huawei P40 Pro+ vs P40 Pro — distinct higher model (pocket-lint, CNET). `p40-pro` -> `p40-pro-plus`. - Sony Xperia Z3+ vs Xperia Z3 — the Z3+ (a.k.a. Xperia Z4) is a distinct 2015 Snapdragon 810 model with its own Wikipedia page, vs the 2014 Z3 (Snapdragon 801). `xperia-z3` -> `xperia-z3-plus`. - Motorola Razr (2023) vs Razr (2019) — different foldable generations (SD 8+ Gen 1 vs SD 710). `razr` -> `razr-2023`. - Fairphone 3+ (2020) vs Fairphone 3 (2019) — successor model. The 3+ record -> `fairphone-3-plus-dual-sim-lte-fp3plus`. - Gigabyte GSmart GS202+ vs GS202 — distinct + variant. The + record -> `gsmart-gs202-plus`. Each renamed record keeps its own specs and sources untouched (nothing merged, nothing invented); only slug, base_model_slug, name and filename change so the two devices no longer collide as a false pair. New slugs are collision-free. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Residual brand-prefix pairs drop from 183 to 176. Refs #296

github-actions Bot assigned Seungpyo1007 Sep 29, 2026

github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026

github-actions Bot added this to the Massive dataset rebuild (1989-2026) milestone Sep 29, 2026

Seungpyo1007 added this to TechAPI-Project Sep 29, 2026

github-project-automation Bot moved this to Todo in TechAPI-Project Sep 29, 2026

Seungpyo1007 mentioned this pull request Sep 29, 2026

Data accuracy: corrections, duplicates and dates (workstream) #296

Open

Seungpyo1007 added 2 commits September 29, 2026 16:46

Seungpyo1007 force-pushed the Seungpyo1007/kiro-trial branch from c9b11e2 to 4fdc6b2 Compare September 29, 2026 08:35

Seungpyo1007 merged commit 2360ec8 into develop Sep 29, 2026
5 checks passed

Seungpyo1007 mentioned this pull request Sep 29, 2026

fix(data): resolve 364 smartphone brand-prefix conflicts #330

Merged

This was referenced Sep 29, 2026

fix(data): merge 29 confirmed laptop near-duplicate records #332

Merged

fix(data): merge 651 confirmed monitor near-duplicate records #333

Merged

This was referenced Sep 29, 2026

fix(data): resolve 170 more smartphone brand-prefix conflicts #334

Merged

fix(data): merge 28 confirmed PDA near-duplicate records #335

Merged

Seungpyo1007 mentioned this pull request Sep 29, 2026

fix(data): resolve 45 generic SoC placeholder references (Google Pixel) #338

Merged

Seungpyo1007 deleted the Seungpyo1007/kiro-trial branch September 29, 2026 14:55

This was referenced Sep 29, 2026

fix(data): resolve 8 GPU near-duplicate and spec-accuracy findings #343

Open

fix(data): un-pair 7 genuinely different smartphone devices #349

Merged

Navigation Menu

fix(data): merge 50 confirmed smartphone near-duplicate records - #329

fix(data): merge 50 confirmed smartphone near-duplicate records#329
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/kiro-trial

Seungpyo1007 commented Sep 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Seungpyo1007 commented Sep 29, 2026

Summary

Per-pattern breakdown

The 50 merges

Reported, not "fixed": a systemic placeholder-value bug

Validation

Dump

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant