geoai

Two-stage, fully local, open-weights image geolocation for Google Street View panoramas (the GeoGuessr setting). Zero recurring API cost; tuned for a dual RTX 4090 workstation.

• Stage 1 — a SigLIP2-SO400M vision backbone feeding an autoregressive hierarchical S2-cell classifier (levels 3/6/9/12), with a ProtoNet content-aware selector that picks the final cell by image-feature similarity.
• Stage 2 (optional) — a Gemma-4-26B vision-language layer that reads on-image text (OCR → VLM → geocode) to pinpoint within Stage 1's region.

Occlusion attribution — slide an occluder across each view, measure how much it shifts the ProtoNet match, and project the result back onto the equirectangular pano. Brighter red = the model relied on that region more. Rural scenes lean on the roadway, vegetation, and horizon rather than any single landmark.

The training corpus — 3.80M panoramas, coloured by capture year; teal = no date recorded. Regenerate with scripts/render_density_map.py.

Results

Deployed model (stage1_v4/epoch_04 + its own ProtoNet index, ProtoNet-select $K{=}500$, deployed 2026-10-24), against the previous stage1_v3_long/epoch_04 on the same 3,000 random held-out test panoramas, with identical crops and serve settings:

model	median	mean	< 1 km	< 25 km	< 200 km	< 500 km	< 750 km
v4 (deployed)	4.5 km	51.5 km	21.7%	80.8%	96.1%	98.5%	99.3%
v3_long (previous)	43.2 km	158.3 km	3.8%	39.2%	83.0%	95.6%	97.3%

v3_long measured 24.9 km on the test split in July. It scores worse here because about a third of today's test panos come from scrapes added after it was built: it gets 27.3 km on the older panos and 104 km on the newer ones, while v4 gets 4.4 and 4.7 km. The split has no spatial separation (median 805 m from a test pano to the nearest train pano), so the absolute numbers are optimistic. The comparison between the two models is fair, because both get the same advantage. Per-country results are in /data/geolocation/ops/ab_v4_vs_v3_test.txt.

The finding: selection, not capacity

Across three training generations the validation median plateaued near 78 km, and fine-grained accuracy stalled (within-25 km ≈ 22%) despite more data and training. A short battery of diagnostics showed the bottleneck was selection, not the model: an oracle that perfectly classifies to the finest cell hits a 5.4 km median, and the true L9 cell is in the model's top-50 candidates 93.5% of the time (given correct coarse context). The heads find the right cell; the model's density-biased probability can't rank it first.

Replacing that ranker with ProtoNet feature-similarity over the top-K candidate cells — picking the cell whose stored prototypes best match the query image — recovers most of the lost accuracy, with no retraining:

	median	mean	within 25 km
greedy (probability top-1)	~70 km	~110 km	~20%
ProtoNet-select ($K{=}500$)	24.9 km	82.8 km	50.1%

It also simplified the system: every candidate-masking "cascade mode" hurt once the selector was active, so Stage 1 is a single path (raw logits → select). The full study, including the per-country breakdown, is in paper/geoai_paper.md.

The second finding: the loss couldn't express 5–100 km

The remaining error lives between 5 and 100 km, and five separate post-hoc fixes failed to touch it — a learned reranker (25.3 → 41.3 km on a genuinely held-out split), geometric verification (+6 points / 5000), metric reshaping (spread 4.5× wider, retrieval worse), aerial fusion, and offset regression on frozen features. All five tried to recover information downstream of the encoder.

Measuring the loss itself explained why. scripts/sigma_analysis.py evaluates the haversine-smoothed soft target against the real cell layout:

level	cell spacing	σ	ratio	cells sharing target	P(true cell)	contrast @25 km
6	101.8 km	500	4.9×	110	0.020	—
9	12.9 km	100	7.8×	364	0.007	1.03
12	1.9 km	20	10.5×	223	0.011	1.04 @5 km

At L9 the correct cell receives 0.7% of the target mass, and a cell 25 km wrong receives 97% of what the correct one does. The information was never absent from the images — the objective simply never asked for it.

A 5-arm controlled A/B (110k subset, identical warm start, seed 42) confirms it:

arm	median	L9 top-1	L6 top-1
baseline (σ = 100/20)	42.0 km	3.7%	0.230
σ = 10.3/1.5	26.6 km	25.6%	0.156
σ + offset head	26.9 km	25.8%	0.160
σ + L6 σ = 81.4 (ep1)	30.1 km	22.7%	0.516

−37% median and 6.9× fine-cell accuracy from changing two numbers. The baseline moved only 43.3 → 42.0 km across a full extra epoch — stuck, not slow. The offset head adds nothing once σ is fixed, so it stays off; fixing L6's σ too recovers coarse accuracy at no cost. Absolute values here are not comparable to the deployed 24.9 km — a 110k-subset fine-tune with a fresh optimizer perturbs a converged model, which is why the baseline arm scores worse than production. Only the between-arm comparison is meaningful.

Ceiling: perfect L9 classification is 5.96 km. L12 would go finer, but 64.6% of its cells hold a single pano, and scripts/project_l12_coverage.py shows the in-flight scrape moves coverage only 20% → 24.9% — new panos open new cells faster than they fill existing ones. Getting below ~6 km needs regression, not a finer grid.

The third finding: the full σ run was judged on the wrong metric

stage1_v4, the full-corpus geometric-σ run, was stopped after 3 epochs. Its median improved, but its errors beyond 2500 km doubled to 10%. That tail was entirely the L12 head. Its argmax jumps continents whenever the true cell isn't in the ~25%-coverage L12 vocabulary. Training validation scored only that argmax. Decoding the same forward pass at L9 (8,000 val panos, no ProtoNet):

decode (stage1_v4 / epoch_02)	median	< 25 km	> 2500 km	exp. GeoGuessr
L12 argmax (the old val metric)	36.3 km	43.9%	10.25%	4296
L9 argmax, greedy chain	17.3 km	58.1%	0.14%	4826
L9, soft parent conditioning	13.5 km	68.0%	0.14%	4879
L9 with true parents (oracle)	9.5 km	85.3%	0.03%	4961
v3_long / epoch_04, L9 greedy (then-deployed model, raw)	89.1 km	13.9%	0.69%	4646

The soft-conditioned L9 score rises every epoch (4810 → 4852 → 4879), so the run was still improving when it was stopped. It was resumed for epochs 3–4 and then plateaued (full 114k val, soft L9: 12.9 km / 4884 → 12.8 km / 4883), and epoch_04 became the deployed model (see Results). Under ProtoNet-select, soft conditioning makes no measurable difference on the test split. The oracle row measures exposure bias. Heads are trained teacher-forced on true parents but decoded on argmax parents, and L3/L6 top-1 are only 0.64 and 0.47. Conditioning on the probability-weighted parent embedding closes half of that gap with no retraining (heads_forward(soft_cond=True), serve GEOAI_SOFT_COND=1; it hurts v3_long, so it stays off by default). --soft-cond-prob trains against the same bias (scheduled sampling) and is untested. Validation now logs the L9 and soft-L9 decodes and the expected score alongside the L12 number.

Negative results on the same cache, so they aren't retried: a strict P(c3)·P(c6|c3)·P(c9|c6) beam is much worse (207 km), because the weak coarse heads then fix the region. A country-head product and expected-score (MBR) decoding add ≤ 2 points once the decode is at L9, since the tail is already 0.14%.

Architecture (Stage 1)

Component	Choice
Backbone	google/siglip2-so400m-patch14-384, full-finetune
View pooling	Concat across 4 perspective crops (90° FOV at headings 0/90/180/270)
Head	Autoregressive over levels L3/6/9/12 — each conditions on a 256-d embedding of the previous (GeoToken-style)
Loss	Haversine-smoothed soft-CE per level + auxiliary country CE + cross-country border penalty
Selection	ProtoNet L9 retrieval similarity over the top-$K$ candidate cells (1.94M prototypes, 119,026 cells)
Compute	2× RTX 4090, bf16, torch.compile, gradient checkpointing

S2 cell sizes (avg edge): L3 ~1150 km, L6 ~144 km, L9 ~18 km, L12 ~2.3 km. See TECHNICALS.md for a full code walkthrough and paper/geoai_paper.md for the technical report.

Stage 2 (optional reasoning layer)

For text-rich panoramas, refined mode runs: Surya OCR → language ID → NLLB translation → Gemma 4 26B (via an OpenAI-compatible LM Studio endpoint) → Nominatim geocode → country-sanity check. It overrides Stage 1 only when it finds a specific OSM point feature near Stage 1's guess; otherwise it defers. Its chief use is visually-homogeneous countries (e.g. Japan, where pixels alone are ambiguous and the disambiguating signal is on-image text).

gemma mode (geoai/stage2/gemma_refine.py) is a leaner alternative: one multimodal call against any OpenAI-compatible endpoint (point GEOAI_VLLM_URL/GEOAI_VLLM_MODEL at e.g. a local vLLM). It renders 8 overlapping 896px perspective tiles — feeding the raw equirect is useless, since the server downscales it and every sign becomes unreadable — and asks for readable text plus one geocodable place. Stage 1 keeps the country (its country head is ~96%; the VLM is worse), so the refiner may only move the pin within it.

Two guards matter, both learned the hard way: a hit that is an area centroid (place=city, boundary=administrative) is rejected when ProtoNet similarity ≥ 0.95, because overwriting a precise retrieval pin with a city centroid measurably hurts (1.7 km → 18.2 km on one Okinawa pano); and point features are capped at 75 km of drift to reject same-name-wrong-branch matches.

Measured on 120 Japan test panos: it commits on ~1% and leaves the median unchanged, because ~90% of those panos contain no readable text at all. It is a precision tool for text-rich/urban locations, not a fix for the rural ambiguity tail — which is why it should be gated by routing, below.

Confidence routing (cascade=auto)

auto runs Stage 1, then escalates to a VLM refiner only when retrieval looks unsure, using ProtoNet's top cosine similarity as the router signal (≈0.94 in data-starved PHL/IDN vs ≈0.98 in confident-but-wrong JPN). Above GEOAI_ROUTE_SIM_THRESHOLD (default 0.965) Stage 1 has matched a stored pano and is already precise, so escalating can only add latency and risk a regression. The decision is recorded on the response (route_sim, route_escalated, route_reason) so it is auditable.

Reranking and hard negatives (CPU-only tooling)

The ProtoNet index doubles as a cache of ~1.9M encoded panos, so several upgrades train without a GPU or any re-encoding:

# Learned reranker over the top-K L9 candidates, with a distance-aware
# listwise loss. Today's selector is an unlearned argmax over cosine
# similarity; measured headroom on our own data is median 17.9 km → 5.0 km
# (oracle best-in-shortlist), i.e. the right cell is nearly always present.
.venv/bin/python scripts/build_rerank_dataset.py --n-queries 30000
.venv/bin/python scripts/train_reranker.py

# Semivariogram hard negatives → per-country sampling weights for training.
# Also prints the spatial-autocorrelation curve; ours is cleanly monotonic
# (γ: 0.012 at <1 km → 0.999 beyond 8000 km).
.venv/bin/python scripts/build_semivariogram_hardnegs.py
#   → --hardneg-weights-path /data/geolocation/ops/hardneg_weights.json

# GRPO groundwork: sample K completions per pano from a running VLM and score
# them with a smooth geo-similarity reward (abstention earns partial credit).
.venv/bin/python scripts/build_grpo_traces.py --country JPN --n 40 --k 4

Optional web reverse-image-search refinement lives in geoai/stage2/websearch.py; it stays disabled unless a provider key is set.

Loss diagnostics

# What can the smoothing kernel actually distinguish? Prints, per level, the
# effective number of cells sharing the target and the contrast between the
# correct cell and one N km away. Contrast 1.0 = the loss cannot tell them apart.
.venv/bin/python scripts/sigma_analysis.py --level 9

# Will more scraping make L12 usable? (Answer: no — see above.)
.venv/bin/python scripts/project_l12_coverage.py

# Can the features locate a pano WITHIN its L9 cell? --split-by cell holds out
# whole cells; a pano-level split leaks, since our split has no spatial
# separation (median 805 m to the nearest train pano).
.venv/bin/python scripts/test_offset_head.py --split-by cell

scripts/sigma_harness.py trains only the heads on cached frozen features. It answers decoding questions cheaply, but cannot rank σ schedules — σ works by shaping what the backbone encodes, and freezing the backbone removes that mechanism. It scored three schedules at 122.6 / 127.0 / 122.6 km that a real run separates by 37%.

Setup

Python 3.11 is required (Unsloth's Gemma 4 MoE support, for Stage 2).

If the system Python is newer than 3.11, uv can supply the right interpreter without touching system packages (uv python install 3.11).

python3.11 -m venv .venv
.venv/bin/pip install -e .
# torch from PyTorch's CUDA wheel index:
.venv/bin/pip install --index-url https://download.pytorch.org/whl/cu128 torch torchvision
.venv/bin/python -c "import geoai.config; print(geoai.config.DATA_ROOT)"   # /data/geolocation

A run.py wrapper covers the same commands without the venv path (python run.py serve|train|scrape|help).

Usage

# Data pipeline (resumable; reads/writes /data/geolocation)
.venv/bin/geoai-catalog            # CSV + GADM + S2 + climate + pop → SQLite
.venv/bin/geoai-render-crops       # equirect → 4× 384px perspective JPEGs
.venv/bin/geoai-split              # deterministic location-level held-out split
.venv/bin/geoai-cell-stats --out-path .../cells_v4.parquet

# Train Stage 1 (dual-GPU)
.venv/bin/accelerate launch --num_processes 2 .venv/bin/geoai-train-stage1 \
  --cells-parquet .../cells_v4.parquet --country-loss-weight 0.3 \
  --border-lambda 0.5 --compile \
  --sigma-l6 81.4 --sigma-l9 10.3 --sigma-l12 1.5   # geometry-derived; see above

# Build the ProtoNet L9 index (supports --shard/--num-shards for dual-GPU)
.venv/bin/python scripts/build_protonet_index.py \
  --ckpt .../stage1_v4/epoch_04 --out .../epoch_04/protonet_l9.pt \
  --cells-parquet .../cells_v4.parquet --max-per-cell 50

# Serve (FastAPI; ProtoNet-select on by default; auto-loads <ckpt>/protonet_l9.pt).
# Restart to change checkpoints: /api/v1/models/load swaps the model but keeps
# the old ProtoNet index.
.venv/bin/geoai-serve --ckpt .../stage1_v4/epoch_04 \
  --cells-parquet .../cells_v4.parquet --device cuda:0

The userscript (script.js, Tampermonkey) plays live on geoguessr.com against the server, with two modes: ProtoNet (Stage 1) and + Stage 2 (OCR/VLM).

Repository layout

geoai/
  config.py            data paths, raster/GADM lookups, env overrides
  scraper/             Street View scraper (parked, functional)
  processing/          catalog + crop renderer + reverse geocoding
  stage1/              model, loss, dataset, predict (ProtoNet-select), protonet, eval
  stage2/              OCR (extract) + pinpoint (Gemma) + geocode + refine
  serve/               FastAPI inference server
scripts/               ProtoNet index build (+ dual-GPU shard/merge)
paper/                 technical report (Markdown + figures)
TECHNICALS.md          code/architecture walkthrough + gotchas

Constraints

• Open weights only — no proprietary models at train or inference.
• Data lives on /data/geolocation/, never in the repo; read in place via geoai.config (override with GEOAI_DATA_ROOT).
• Resumable by construction — catalog, crop render, and scraper dedupe so kill/restart is always safe.

Non-obvious gotchas (S2 cell IDs stored as TEXT, SigLIP 0.5/0.5 normalization, GHS-POP Mollweide reprojection, location-level splits, …) are documented in TECHNICALS.md.