Skip to content

fix(data): remove 18 fabricated placeholder values from PDA records - #344

Merged
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/pda-placeholder-scan
Sep 29, 2026
Merged

Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/pda-placeholder-scan

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What

Follow-up to the PDA dedup pass (#335). That pass only caught fabricated import
placeholders inside duplicate pairs (where comparing the two sides revealed the
fake one). This scans all 112 post-dedup pda records for the same
fabricated-default fingerprint — a single fake-precise value repeated across many
unrelated devices — regardless of whether a record had a twin.

Fixed 18 records; documented the rest that can't be repaired without inventing.

Both gates pass:

  • python -m app.validate → passed
  • integrity_check.py <data> --strict → 0 hard anomalies

Clusters found

Field Value Count Verdict
ram_gb 1 83 / 112 fabricated — no PDA 2000-2011 had 1 GB RAM
battery_mah 1500 33 fabricated (classic-Palm cluster) but unfixable — see below
display.ppi 114 26 real — 240x320 QVGA on 3.5" really is ~114 ppi
display 3.5" / 240x320 26 real — QVGA was the ubiquitous PDA screen
weight_g 190 9 real — iPAQ h63xx chassis family + i-mate HTC rebrands

Spot-checked against sources: Dell Axim X5 = 32/64 MB (Wikipedia, PCMag,
DigitalTrends), Palm IIIc = 8 MB (Wikipedia, TidBITS, Computing History) — all
flatly contradict the stamped ram_gb=1.

Fixed (18 records, all cited — ram_gb is required so repaired, not nulled)

  • 13 Dell Axim — Wikipedia Dell Axim specification table. X5/X3/X30 low-end
    ("Basic") = 32 MB (0.03125); X3/X3i Advanced, X30 Advanced, and the entire
    X50/X51 line = 64 MB (0.0625).
  • 5 Fujitsu-Siemens Pocket Loox N520/N560 = 64 MB (0.0625) — Fujitsu N520
    user manual + clubic N560 launch spec.

No value invented — every figure comes from an authoritative source added to
source_urls.

Left as-is, documented (not guessed, not nulled)

  • Classic Palm line (Palm IIIc/IIIxe/VIIx = 8 MB, m100 = 2 MB, m105/m125/m130/
    m500/m505 = 8 MB, m515 = 16 MB, Tungsten/Zire/Treo/Centro …): their real RAM is
    below the schema's ram_gb >= 0.016 (16 MB) floor, so it cannot be
    represented — almost certainly why they were stamped with ram=1 on import.
    Can't set the true value (fails validate), can't null a required field.
  • Other ram_gb=1 records (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio 728,
    later Palm Treo/webOS) need individual per-device sourcing; left documented
    rather than risk inventing.
  • battery_mah=1500: the classic Palms ran on AAA/AA cells (m100/m105/m125 —
    no mAh rating) or small internal Li-ions; there is no clean per-device mAh to
    cite without inventing, and the field is required (can't null).
  • ppi=114, the 3.5"/240x320 combo, and weight_g=190 are genuinely shared
    real values, not fabricated defaults — left untouched.

Ownership / dump

data/pda/ only. Dump refreshed as the last commit (fresh scratch DB,
TECHAPI_DATA_DIR on this worktree): only the 18 repaired detail pages change,
no other collection touched.

Closes #296 — this removes a sourced slice of the pda fabricated-placeholder values;
the required-field-but-unrepresentable/unsourceable remainder stays open and is
documented above.

Post-dedup scan of all 112 pda records for the fabricated-default fingerprint
(one fake-precise value repeated across many unrelated devices), independent of
whether a record had a duplicate twin.

Clusters found:
- ram_gb=1 on 83 of 112 records. Definitively fabricated: no PDA from 2000-2011
  had 1 GB of RAM (Dell Axim X5 = 32/64 MB, Palm IIIc = 8 MB, etc.). This is the
  Kaggle/phonedb bulk-import placeholder.
- battery_mah=1500 on 33 records (mostly the classic Palm line), and
  weight_g=190 on 9 - see "left as-is" below.

Fixed 18 ram_gb=1 values to the real, sourced figure (ram_gb is a required field,
so these are repaired to a cited value, not nulled). No value invented - each
comes from an authoritative source added to source_urls:

- 13 Dell Axim (Wikipedia Dell Axim specification table): X5/X3/X30 low-end
  ("Basic") = 32 MB = 0.03125; every other Axim model (X3/X3i Advanced, X30
  Advanced, and the whole X50/X51 line) = 64 MB = 0.0625.
- 5 Fujitsu-Siemens Pocket Loox N520/N560 = 64 MB = 0.0625 (Fujitsu N520 user
  manual + clubic N560 launch spec).

Left as-is, documented rather than guessed or nulled:
- Classic Palm line (Palm IIIc/IIIxe/VIIx = 8 MB, m100 = 2 MB, m105/m125/m130/
  m500/m505 = 8 MB, m515 = 16 MB, Tungsten/Zire/Treo/Centro, etc.): their real
  RAM is below the schema's ram_gb>=0.016 (16 MB) floor, so it is not
  representable - almost certainly why they were stamped with the ram=1
  placeholder in the first place. Can't set the real value (fails validate),
  can't null a required field. Reported on #296.
- The remaining ram_gb=1 records (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio
  728, later Palm Treo/webOS) need individual per-device sourcing; left documented.
- battery_mah=1500: the classic Palms ran on AAA/AA cells (m100/m105/m125) or
  small internal Li-ions - clearly not a uniform 1500 mAh - but there is no clean
  per-device mAh to cite without inventing, and the field is required (can't null).
  Reported on #296.
- ppi=114 (26 records) and the 3.5"/240x320 combo are NOT fabricated: 240x320
  QVGA on a 3.5" panel really is ~114 ppi, legitimately shared by that whole
  common screen class. weight_g=190 (9) is real chassis-sharing (iPAQ h63xx
  family, i-mate HTC rebrands). Left untouched.

Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies).

Refs #296
Re-dumped the pdas collection from the corrected data tree (fresh scratch DB, TECHAPI_DATA_DIR on this worktree). Only the 18 Dell Axim / Fujitsu-Siemens detail pages whose ram_gb was repaired change; no other collection touched.
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026
@Seungpyo1007
Seungpyo1007 merged commit e9fc443 into develop Sep 29, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the Seungpyo1007/pda-placeholder-scan branch September 29, 2026 15:59
@Seungpyo1007 Seungpyo1007 removed the enhancement New feature or request label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

1 participant