Skip to content

fix(data): add source citations for 7086 website records - #345

Merged
Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/website-citation-hunt
Sep 29, 2026
Merged

Seungpyo1007 merged 2 commits into
developfrom
Seungpyo1007/website-citation-hunt

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Upgrades 7,086 data/website/ records from uncited (Wikidata-only,
verified: false) to source-verified by adding an independent, checkable
authoritative citation and flipping verified: true. No stored values were
fabricated, guessed, or altered.

Addresses the uncited-website backlog. Closes #296.

Methodology

Every website record previously carried exactly one source_urls entry — its
Wikidata item (wikidata.org/wiki/Q…) — and ~32k were verified: false. A
Wikidata item alone is a database cross-reference, not an independent
confirmation, so those records were effectively uncited.

For each verified: false, Wikidata-only record:

  1. Resolved its Wikidata QID and queried the Wikidata API
    (wbgetentities, props=sitelinks|claims|labels).
  2. If the item has an English Wikipedia article (enwiki sitelink), that
    article is a real, human-checkable authoritative source describing the same
    entity. The exact article URL returned by the API was appended to
    source_urls and verified set to true.
  3. Cross-checked stored launch_date against Wikidata inception (P571).
  4. If the item has no enwiki article, the record was left untouched
    (still verified: false, Wikidata-only) — no citation was invented.

Every added URL is the genuine Wikipedia sitelink Wikidata reports for that
exact entity; verified: true is only ever set together with a real article
URL. This is fully reproducible against the live Wikidata API.

Counts

Outcome Count
Cited-and-verified (added enwiki Wikipedia URL, verified: true) 7,086
Value-corrected 0
Left uncited (no enwiki article; not fabricated) 25,346
Left uncited — launch_date vs Wikidata P571 year mismatch, flagged for manual review 17
Already verified: true before this pass (untouched) 7,635
Total website records 40,084

On the 0 value corrections

17 records had a launch_date whose year disagreed with the Wikidata inception
date (P571). These were not auto-corrected: a website's launch date and the
inception of its subject/parent organisation are distinct facts (e.g. Netflix the
company, 1997, vs. the streaming service, 2007). Overwriting one with the other
would introduce wrong data, which the task forbids. They are left exactly as-is
and flagged for manual review instead.

Verification

  • python -m app.validate → passed.
  • TechEngine python integrity_check.py <worktree>/data --strict → exit 0, no
    hard anomalies
    (empty structural section; only pre-existing advisory
    CPU/GPU benchmark ratio outliers printed, unrelated to this change).
  • Static dump refreshed from TechEngine
    (python -m app.dump, fresh scratch DATABASE_URL, TECHAPI_DATA_DIR
    pointing at this worktree) — regenerated site/public/v1/websites for the
    affected records (last commit). Unrelated pre-existing dump drift in other
    categories was reverted to keep this PR scoped to data/website/ only.

Scope

data/website/ only. 7,086 data files + 7,086 regenerated dump files.

This is one batch; the remaining uncited records (those without an English
Wikipedia article) are intentionally left for future batches per the task's
"batch into multiple PRs" guidance.

Upgrade website records from Wikidata-only (verified:false) to source-verified by adding an independent, checkable citation: the English Wikipedia article linked from each record's Wikidata entity (enwiki sitelink). Only records whose Wikidata entity has an enwiki article were verified; the Wikipedia URL was appended to source_urls and verified set to true. No stored values were fabricated or altered.

Records with no enwiki article were left untouched (still verified:false, Wikidata-only). 17 records whose launch_date disagreed with Wikidata inception (P571) were left untouched for manual review, since a site's launch date and the subject's inception are distinct facts. Refs #296
…bsite records

Regenerated site/public/v1/websites from TechEngine (python -m app.dump against a fresh scratch DATABASE_URL with TECHAPI_DATA_DIR pointing at this worktree). Reflects the added Wikipedia citations and verified:true flags. Refs #296
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 29, 2026
@Seungpyo1007
Seungpyo1007 merged commit 3a4dfbc into develop Sep 29, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the Seungpyo1007/website-citation-hunt branch September 29, 2026 16:12
@Seungpyo1007 Seungpyo1007 removed the enhancement New feature or request label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

1 participant