fix(data): resolve 83 PDA placeholder ram_gb/os values - #347
Open
Seungpyo1007 wants to merge 2 commits into
Open
Seungpyo1007 wants to merge 2 commits into
Seungpyo1007 wants to merge 2 commits into
Conversation
The PDA dedup follow-up flagged a systemic placeholder pattern in data/pda/: nearly every clean record carried ram_gb=1 (a placeholder, not 1 GB) and many Palm/Dell/MiTAC records carried os="Feature phone" for devices that actually run Palm OS, webOS, or Windows Mobile. Sourced the real values per device from each record's cited phonedb.net datasheet (RAM Capacity + Platform fields); for the six kaggle-only records without an in-record phonedb id, sourced from Wikipedia (Treo 180/270/600/650) and imei24 (iPAQ Glisten, i-mate Pocket PC), added to source_urls. - os: "Feature phone" -> "Palm OS" / "webOS" / "Windows Mobile" per device (50 records corrected; the Windows-Mobile Treos/Dell/Fujitsu-Siemens that were already correct kept their os). - ram_gb: placeholder 1 -> true capacity, converted MiB/1024 to match the existing dataset convention (16->0.016, 32->0.032, 64->0.0625, 128->0.125, 256->0.25, 512->0.5). variant.memory.ram_gb mirrored where present. Left unresolved (documented, not invented): 12 pre-2003 Palm classics with 2-8 MiB RAM (m100/Zire = 2 MiB; IIIc/IIIxe/VIIx/m105/m125/m130/m500/m505/m515... ) — their true RAM is below the schema's ram_gb floor (0.016 GB = 16 MiB in app/validate.py), so it cannot be represented without overstating it; their os was still corrected to "Palm OS" and ram_gb left at the prior value rather than writing a wrong in-range number. Verified: python -m app.validate passes; integrity_check.py --strict reports no hard anomalies. Refs #296
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves the systemic PDA placeholder pattern flagged by the PDA dedup follow-up:
data/pda/records almost uniformly carriedram_gb=1(a placeholder, not 1 GB) and many carriedos="Feature phone"for devices that actually run Palm OS, webOS, or Windows Mobile.83 of the 112 PDA records are corrected here, each value sourced from that device's own cited datasheet.
What changed
os(50 records):"Feature phone"→ the real platform family per device —"Palm OS"(Palm III/V/VII/m-series/Tungsten/Zire/TX/Z22/Treo up to 755p/Centro),"webOS"(Palm Pre/Pixi and their plus/2 variants),"Windows Mobile"(Dell Axim X5, MiTAC Mio 728). Records already correctly labeled Windows Mobile (the WM Treos, Dell Axim X3–X51, Fujitsu-Siemens RPDA, i-mate) kept theiros.ram_gb: placeholder1→ the true capacity, converted MiB→GB to match the existing dataset convention (16→0.016,32→0.032,64→0.0625,128→0.125,256→0.25,512→0.5).variant.memory.ram_gbmirrored where present.source_urls: for the six kaggle-only records without an in-record phonedb id, added the sourcing URL (Wikipedia for Treo 180/270/600/650; imei24 for iPAQ Glisten and i-mate Pocket PC).Sourcing
RAM/OS were read from each record's already-cited phonedb.net datasheet (the "RAM Capacity (converted)" and "Platform"/"Operating System" fields). Spot-verified several directly (Palm IIIc = 8 MiB / Palm OS 3.5; Palm TX = 32 MiB / Palm OS 5.4; Palm Pre 2 = 512 MiB / webOS 2.0). No values were invented.
Left unresolved (documented)
12 pre-2003 Palm classics have 2–8 MiB of RAM (m100 & Zire = 2 MiB; IIIc/IIIxe/VIIx/m105/m125/m130/m500/m505/i705/Zire 21 = 8 MiB). The schema's
ram_gbfloor is0.016GB (16 MiB) inapp/validate.py, so their true RAM cannot be represented without overstating it. Theiroswas corrected to"Palm OS", butram_gbwas left at the prior value rather than writing a wrong in-range number — the true value is known and cited, but not representable. Raising the validate floor is an engine-side change out of scope for this data PR; flagging it here for a follow-up.Testing
python -m app.validate→ Data validation passed.python integrity_check.py <data> --strict→ no hard anomalies.python -m app.dump; committed only the 83 PDA detail pages whose content changed (timestamp-only churn in other collections was discarded).Refs #296