Skip to content

RES-1858: geocode addresses with Mapbox, and only Mapbox - #644

Open
ngm wants to merge 8 commits into
developfrom
RES-1858_mapbox_geocode
Open

ngm wants to merge 8 commits into
developfrom
RES-1858_mapbox_geocode

Conversation

@ngm

@ngm ngm commented May 3, 2023 •

Copy link
Copy Markdown
Contributor

RES-1858. Turning a group, event or profile location into coordinates and a country code (geocoding) now uses Mapbox, through the geocoder-php library already configured on develop. Before, it called the Google Geocoding API directly.

Changes

  • Geocoding. Before: Google. Now: App\Helpers\Geocoder::geocode() asks Mapbox and reads the coordinates and the ISO country code from its answer. It returns false when Mapbox can't place the address, can't be reached, rejects the token or the quota is used up, and logs the reason. Callers show the existing "location could not be geocoded" message, as before.
  • One provider. Mapbox is the only geocoder. Nothing falls back to GeoPlugin, which only geocodes IP addresses, or to Google, so every environment behaves the same way.
  • Keys. MAPBOX_TOKEN is the geocoding credential. GOOGLE_API_CONSOLE_KEY is still used, but only by the browser for the maps and the address autocomplete.
  • CI. Tests use the stub geocoder when MAPBOX_TOKEN is empty (forked PRs get no secrets), rather than when the Google key is.
  • The unused reverseGeocode() is removed; reverse geocoding goes through set:networkdata:place.
  • docs/local-development.md says which key does what.

Mapbox token

Production CI
Same token as dev and PR previews (identical Fly secret digest) its own token
Type public (pk.), valid, account neilrestart masked, ends wj5g
URL restriction none: server-side geocoding returns 200 with no Referer and with a foreign one n/a
Rate limit 1,000 geocoding requests a minute

Checked from inside the production machine on 2026-10-14: the token geocodes "Hackney, London" through both the v5 and v6 Mapbox geocoding APIs.

Before merging: check the neilrestart account's Statistics page for monthly geocoding usage against the plan's free allowance (100,000 Temporary Geocoding requests). A public token can't read usage or plan, so this needs the account login. Not for release notes until that's checked.

Not included

  • Replacing the Google Places address autocomplete in the browser with a Mapbox one. The location components have been reworked since this branch began, so that is a separate change.

Tests

  • tests/Unit/GeocoderTest.php binds a fake provider, so nothing touches the network. It covers: coordinates and country code are read from the provider's answer; the ISO code is used, not the country name; an empty answer, a missing country, a provider that throws, and the ForceGeocodeFailure sentinel each give the right result.
  • Suite-wide, Tests\Support\GeocoderMock replaces the geocoder, so no test calls Mapbox. Two profile tests that forced a failure by invalidating the Google key now use a town the mock doesn't know.

@sonarqubecloud

sonarqubecloud Bot commented May 3, 2023

Copy link
Copy Markdown

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 2 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.6% 0.6% Duplication

@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2023

Copy link
Copy Markdown

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 2 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.6% 0.6% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

The rest of the Mapbox geocoder plumbing (config/geocoder.php, the
geocoder-php/mapbox-provider + toin0u/geocoder-laravel packages, and
MAPBOX_TOKEN in .env/CI/Fly secrets) already landed on develop as part
of other work and is already used for reverse geocoding in
SetPlaceNetworkData. This just switches the remaining forward-geocode
path, App\Helpers\Geocoder::geocode(), from a raw Google Maps HTTP call
to the same app('geocoder') chain (Mapbox, falling back to GeoPlugin),
matching that existing pattern.

country_code now comes straight from the provider's ISO country code
rather than matching the returned country name against our own country
list, and the dead reverseGeocode() method (superseded by the
reverseQuery() call already in SetPlaceNetworkData) is removed.

Public interface, the Geocoder container binding, and the test mocks
(GeocoderMock, GeocoderFailMock) are unchanged, so no other files need
updating for this to work.

This branch is rebuilt on current develop (~1240 commits ahead of
where it last was) so CI can actually run; the frontend part of the
original PR (replacing vue-google-autocomplete with a Mapbox GL
widget) is intentionally not carried over - see PR comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@edwh
edwh force-pushed the RES-1858_mapbox_geocode branch from 18af27c to fcfbcc0 Compare July 15, 2026 09:02
@edwh

edwh commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Rebuilt this branch on current develop (it was ~1240 commits behind, from before CI was reworked, so no checks were running at all).

While rebasing this, it turned out most of the Mapbox plumbing this PR wanted had already landed on develop separately: config/geocoder.php (Mapbox + GeoPlugin chain), the geocoder-php/mapbox-provider / toin0u/geocoder-laravel packages, and MAPBOX_TOKEN in .env/CircleCI/Fly secrets are all already there and already used for reverse geocoding in App\Console\Commands\SetPlaceNetworkData.

So the only remaining piece of this PR's intent was the forward geocode path, App\Helpers\Geocoder::geocode() (used by group/event/user location editing), which was still calling the Google Maps HTTP API directly. That's now switched to app('geocoder'), the same Mapbox-then-GeoPlugin chain, matching the pattern already used by SetPlaceNetworkData. country_code is now taken directly from the provider's ISO country code rather than matching the country name against our own list. The unused reverseGeocode() method is removed (nothing calls it any more; reverse geocoding goes through SetPlaceNetworkData's direct reverseQuery() call). The public interface, container binding, and existing test mocks (GeocoderMock, GeocoderFailMock) are unchanged, and tests/Unit/GeocoderTest.php on develop already expects exactly this interface, so no other files needed changes.

Infra/build files (Dockerfile, docker-compose.yml, docker_run.sh, package.json/package-lock.json) are taken as-is from develop - the versions in this branch were three years stale and would only have caused conflicts.

Not carried over: the third part of the original PR description - replacing vue-google-autocomplete with a Mapbox GL JS geocoder widget in GroupLocation.vue/VenueAddress.vue. Those two components have been substantially reworked on develop since this branch was created (a shared PlacesAutocomplete.vue wrapper, Leaflet-based map display, and Playwright-testability watchers that are specifically built around the current Google Autocomplete widget's DOM behaviour). Redoing that swap properly is a real frontend rewrite in its own right, not a rebase, and risks breaking the existing Playwright coverage for group/event location editing. Recommend splitting that into its own follow-up PR if it's still wanted, rather than bundling it here.

Needs manual verification:

  • A live Mapbox geocode against a real MAPBOX_TOKEN - CI should exercise this via tests/Unit/GeocoderTest.php (the CircleCI secret should be a real token; local dev's .env only has the placeholder 1234).
  • That the production/Fly MAPBOX_TOKEN secret is actually set (docs say it should be, as a shared secret, but worth double-checking before this merges).

@sonarqubecloud

Copy link
Copy Markdown

edwh and others added 3 commits September 5, 2026 21:47
develop gained #904 (a fetch() seam so GeocoderTest could exercise the
parsing without calling Google) while this branch was replacing the whole
method with the geocoder package. Same method, incompatible approaches.

Taking this branch's: with the provider chain resolved from the
container there is nothing to seam - a test binds its own provider
instead, which is what the code comment here already promised. That also
drops googleKey(), and reverseGeocode(), which was defined and never
called anywhere in the codebase.

GeocoderTest is rewritten onto that mechanism: it builds real
Geocoder\Model\Address objects and binds a provider returning them.

Both container keys have to be replaced, not just one. geocoder-laravel
binds the string 'geocoder' through to ProviderAndDumperAggregator and
registers that as a singleton, so replacing only the string leaves the
real Mapbox chain reachable - and a test that passes then is quietly
talking to the network.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AddressCollection::get(0) raises OutOfBounds on an empty collection, so
an address Mapbox and GeoPlugin both fail to place crashed the request
instead of returning false the way callers expect - and the way the
Google implementation did, since it counted results before indexing.

Caught by rewriting GeocoderTest onto a bound provider: the
nothing-found case throws with the real code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Sep 5, 2026

Copy link
Copy Markdown

@ngm

ngm commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

@edwh I think this is with you - think I recall from catchup we said for this PR you'd check how it compared to what iFixit had done on their fork regarding alternate geocoders, to see if we can minimise drift.

@edwh

edwh commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Compared with iFixit's fork, as asked in the 1 October call.

What this PR does now. It's one small change. Looking up a place's map position (geocoding) goes through the geocoding library already set up on develop, which tries Mapbox first and then GeoPlugin. Before, it called Google directly. The country code comes from the same lookup. The Mapbox settings and MAPBOX_TOKEN are already on develop, so nothing else is needed. CI passes, but it needs develop merged in again before merging.

What iFixit does. Their fork is https://github.com/iFixit/restarters, and the branch they run is hermes.

  • hermes still uses Google for geocoding, the same as our develop. Their last change there split Google into separate keys for the browser and the server (bc2da21, their PR Check for email address already in use on step 2, rather than at step 4 #45).
  • They have an experiment, testing-use-foss-for-maps, that switches maps to OpenStreetMap or Mapbox through a provider setting (a724a7e, 2864dcd, March 2025). It was never merged into hermes, and it's now far behind both their branch and ours.

How they compare. Both move geocoding away from Google, and both can use Mapbox. iFixit's experiment builds its own provider layer. This PR uses the standard library we already have. This PR is the smaller change, so it should be easier for iFixit to pick up if they bring their fork up to date with ours.

Recommendation. Keep this PR: merge develop in, merge it, then tell iFixit that develop no longer needs a Google key for geocoding, so they can drop their experiment rather than maintain it. Following their experiment instead wouldn't reduce drift, because even their own live branch doesn't use it.

@edwh

edwh commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Mapbox token: what it is and what it's allowed

Where it lives. MAPBOX_TOKEN is set in the CircleCI project (confirmed, value masked) and documented as a Fly shared secret. Local .env only has the placeholder, so I could not introspect the real token's scopes or URL restrictions from here; whoever has the Mapbox account should check it at account.mapbox.com → Access tokens. Two things matter for this PR:

  • No URL restriction. The token is only ever used server-side (PHP), and Mapbox enforces URL restrictions via the Referer header, so a restricted token gets 403 from a server. (tokens guide)
  • No special scope. The Geocoding API needs a valid token, not a scope, so a pk. token with the default public scopes is enough. An sk. token also works and must stay server-side, which it does.

Documented limits (the library we use, geocoder-php/mapbox-provider 1.5, calls Geocoding v5 mapbox.places, the Temporary endpoint):

  • Rate: 600 requests/minute by default on v5 (1,000/min on v6), 429 beyond that, adjustable by Mapbox on request. (v5 reference)
  • Volume: 100,000 Temporary requests/month free, then $0.75 per 1,000. Permanent geocoding has no free tier: $5 per 1,000 from the first request, and needs a card on file or a sales contact. (pricing)

Our volume. Geocoding happens server-side when a group, event or profile is saved with a changed location, plus two manual artisan commands: check:grouplocations (one request per approved group per run) and set:networkdata:place (one reverse lookup per group in the network). Neither is scheduled. The browser autocomplete stays on Google. That is hundreds to low thousands of requests a month, nowhere near 100,000. The one thing that could trip the per-minute limit is check:grouplocations on a big group set, since it geocodes back to back; a 429 now logs and reads as "fails to geocode" for the rest of that run.

The terms are the real question, not the numbers. For Temporary geocoding Mapbox says results "may not be retained", requests "must be triggered by user activity", and responses may only be used "in conjunction with a Mapbox map". We store the coordinates in the database, toin0u/geocoder-laravel caches responses, and our maps are CARTO/OSM tiles, not Mapbox. Permanent geocoding (mapbox.places-permanent) allows storage, is supported by the provider via a constructor argument in config/geocoder.php, and at our volume would cost in the region of $5–15/month. Worth a decision before this merges; the code works the same either way.

@edwh edwh changed the title RES-1858 Switch to use mapbox for geocoding RES-1858: geocode addresses with Mapbox, and only Mapbox Oct 8, 2026
…der errors

GeoPlugin geocodes IP addresses, so it could never have placed a street
address; the chain only hid that. Without the chain a bad token or an
unreachable Mapbox would surface as a 500, so the helper catches and logs
it and returns false, as it did when Google was unreachable. CI stubs the
geocoder on a missing MAPBOX_TOKEN rather than a missing Google key, and
the docs and test comments say Mapbox.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@edwh

edwh commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Mapbox token check

I inspected the MAPBOX_TOKEN secret on the restarters-pr-895 preview app (value never printed or stored).

Item Finding
Type Public (pk.), 94 characters
Mapbox validity TokenValid
Account username neilrestart
URL restriction None reported. Geocoding calls with no Referer and with a foreign Referer both returned HTTP 200
Scopes, created, modified Not returned. Mapbox only gives these for a token with tokens:read, and the account token listing returned 404
Same as CI's token No. The last four characters of the preview token do not match the masked wj5g on CircleCI, so CI and the preview use different tokens
Rate limit seen 1,000 geocoding requests per minute

Conclusion: the preview token works for server-side geocoding. It is a plain public token with no URL restriction, which is what server-side calls need.

Not verified:

  • Monthly usage and plan. These cannot be read with a public token. Check the Statistics page of the neilrestart Mapbox account against the 100,000 free Temporary Geocoding requests.
  • The production token. I did not read production secrets, so whether it matches the preview or CI token is unknown.
  • Scopes and creation date. Check these on the Mapbox account's Tokens page.

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants