TechEngine

Validation, ingestion, and serving engine for the TechAPI dataset.

 Code: MIT - Data: lives in TechAPI (CC-BY-SA 4.0)

TechEngine owns everything around the data: schema validation, the FastAPI read API, the static JSON dump generator, the engine's own landing site, automated coverage checks, and the weekly ingestion/refresh crawlers.

The dataset and the public-facing playground site live in TechAPI so each can be versioned, mirrored, and licensed independently. The site shipped in this repo is the engine's own landing: what TechEngine is, what it runs, and links out to docs.

Layout

app/
  validate.py          # schema/range/uniqueness checks
  seed.py              # data/ -> SQLModel database
  dump.py              # API -> static JSON tree
  main.py              # FastAPI entrypoint
  models/              # SQLModel tables
  routers/             # /v1/{brands,socs,smartphones,gpus,cpus}
  schemas/             # Pydantic response models
  services/            # scoring (algorithm_version-tagged)
  coverage/            # upstream-vs-curated diff + Markdown report
  ingest/              # draft new records from upstream pages
tests/                 # unit + integration
site/                  # Astro engine landing (deploys to Pages)
docs/                  # SPEC / DATA_PIPELINE / DEVELOPMENT
TechAPI/               # submodule -> GetTechAPI/TechAPI (clickable @ <sha> link)
.github/workflows/
  validate-data.yml    # workflow_call: PR-time data validation for TechAPI
  weekly-refresh.yml   # cron: live-scrape -> integrity gate -> dump -> PR to TechAPI
  weekly-ingest.yml    # cron: draft new SKUs, open PR against TechAPI
  coverage-report.yml  # cron: gap report, sticky issue (TechEngine + TechAPI)
  refresh-data.yml     # smoke-test: rebuild the dump on engine (app/**) changes
  notify-techapi.yml   # push to main: ping TechAPI to bump its TechEngine submodule
  bump-techapi.yml     # dispatch: advance this repo's TechAPI submodule pointer
  deploy-pages.yml     # build & deploy engine site + dump
  test.yml             # lint + type-check + tests

How The Two Repos Connect

Both repos live in the GetTechAPI org and each includes the other as a git submodule (a clickable @ <sha> pin). Three automations keep them in step:

• validate-data.yml (workflow_call) - TechAPI's PR-time check calls into TechEngine to validate its data.
• weekly-refresh.yml - live-scrapes benchmarks, runs the full-dataset integrity gate (app.validate + integrity_check.py --strict), regenerates the static dump, and opens a dated refresh PR against TechAPI.
• Submodule autosync - every push to TechEngine main fires notify-techapi.yml, which pings TechAPI to bump its TechEngine pointer; conversely bump-techapi.yml advances TechEngine's TechAPI pointer when TechAPI changes. Bumps are loop-guarded, so each real change converges to one.

Every Python entry point reads data from a TechAPI checkout. The location can be overridden via TECHAPI_DATA_DIR; by default TechEngine supports both layouts: TechAPI/TechEngine as a submodule and TechAPI beside TechEngine.

Quickstart

git clone https://github.com/GetTechAPI/TechAPI.git ../TechAPI   # data source
pip install -e ".[dev]"
python -m app.validate          # check data integrity
python -m app.seed              # data/ -> ./techapi.db (SQLite)
uvicorn app.main:app --reload   # serve; curl localhost:8000/v1/cpus/ryzen-9-9950x3d
python -m app.dump              # generate ./dump/v1/... static tree

TECHAPI_DATA_DIR=/path/to/TechAPI/data overrides the data location.

For tablets, watches, and PDAs, battery_mah and weight_g are optional: seed records may omit them or use null when no source value is known, and API detail responses and static dumps preserve unknown values as null. Smartphones continue to require numeric values for both fields.

Docker Compose (Postgres)

docker compose up --build

Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on :8000.

Roadmap

• Split out from TechAPI; sibling-checkout data pipeline
• Coverage gap detector - diff curated dataset vs upstream catalogs and surface missing SKUs as a sticky weekly issue (#1)
• Weekly ingestion crawler - scrape canonical sources and open PRs against TechAPI with new SKUs (requires the TECHAPI_TOKEN secret to push) (#2)
• Weekly refresh pipeline - live benchmark enrichment -> full-dataset integrity gate -> static dump -> dated refresh PR (weekly-refresh.yml)
• Bidirectional submodule autosync between TechEngine and TechAPI
• More sources (Intel ARK, AMD product pages, TechPowerUp DB)

License

Code is licensed under the MIT License. The dataset (in TechAPI) is licensed CC-BY-SA 4.0 separately.