Interline PlanetUtils

• Features
• Installation
  • Using Docker container
  • Using Homebrew on Mac OS
  • Using Python package
• Command-line Usage
  • osm_planet_update
  • osm_planet_extract
  • osm_extract_download
  • osm_planet_get_timestamp
  • elevation_tile_download
  • elevation_tile_merge
  • valhalla_tilepack_list
  • valhalla_tilepack_download
• Specifying extract extents
  • Bounding box file: CSV format
  • Bounding box/polygon file: GeoJSON format
  • Polygon file: Osmosis .poly format
• Switching toolchains
• Support

Features

Python-based scripts and a Docker container to work with planet-scale geographic data. Using PlanetUtils, you can:

• maintain your own copy of the OpenStreetMap planet (by applying incremental updates)
• cut your copy of the OSM planet into named bounding boxes
• download OSM Extracts from Interline for popular cities and regions
• download Mapzen Terrain Tiles from AWS for the planet or your bounding boxes
• merge and resample Terrain Tiles
• download Valhalla Tilepacks from Interline for the planet (subscription required)

PlanetUtils is packaged for use as a:

• Docker container, for use on any operating system
• Python package, for use on any operating system
• Homebrew formula, for use on Mac OS

PlanetUtils is a "high level" library that makes use of Osmosis, OSM C tools, and Osmium among other great open-source components.

Installation

Using Docker container

Make sure you have Docker installed. Then:

docker pull ghcr.io/interline-io/planetutils:v0.5.1

Any of the example commands below can be executed with docker run. It may be helpful to mount a local directory inside the container for persistence and to access output files.

• Example of using docker run with the data directory mounted as /data:

docker run --rm -v ${PWD}/data:/data -t ghcr.io/interline-io/planetutils:v0.5.1 <command>

Using Homebrew on Mac OS

Make sure you have Homebrew installed. Then:

brew install interline-io/planetutils/planetutils

Using Python package

Requires Python 3.11 or newer. The package is not published to PyPI; install it straight from a git tag:

uv tool install "interline-planetutils @ git+https://github.com/interline-io/planetutils@v0.5.1"

or with pip:

pip install "interline-planetutils @ git+https://github.com/interline-io/planetutils@v0.5.1"

System dependencies

Most commands need no system binaries at all -- OSM and raster handling come from the osmium and rasterio wheels, which ship prebuilt for macOS (Apple Silicon and Intel), Linux (x86_64 and arm64) and Windows.

The one exception is osm_planet_extract, which still shells out. A correct bounding-box extract needs reference completion (osmium's complete_ways / smart strategies), and pyosmium does not expose a spatial filter.

Command	System binaries required
osm_planet_update	none (default --toolchain=osmium); --toolchain=osmosis needs Java + Osmosis
osm_planet_extract	yes -- Osmium Tool, Osmosis, or OSM C tools
osm_planet_get_timestamp	none
osm_extract_download	none
elevation_tile_download	none
elevation_tile_merge	none
valhalla_tilepack_download	none
valhalla_tilepack_list	none

To install Osmium Tool for osm_planet_extract:

brew install osmium-tool                      # macOS
sudo apt install osmium-tool                  # Debian/Ubuntu
conda install conda-forge::osmium-tool        # Windows

Or skip it entirely and use the Docker container, which bundles the full toolchain.

Downloading a planet from S3 with osm_planet_update --s3 needs boto3:

uv tool install --with boto3 "interline-planetutils @ git+https://github.com/interline-io/planetutils@v0.5.1"

Developing

This project uses uv, with a committed lockfile for reproducible environments:

git clone https://github.com/interline-io/planetutils.git
cd planetutils
uv sync
uv run pytest

uv is not required. The packaging is standard PEP 621, so pip works too:

pip install -e . --group dev
pytest

Tests that need an external binary skip automatically when it is absent. To make them fail instead -- as CI does inside the container, where the toolchain is guaranteed:

uv run pytest --require-binaries

Command-line Usage

PlanetUtils supplies the following command-line utilities:

osm_planet_update

Update a local OSM planet. For example:

osm_planet_update planet-recent.osm.pbf planet-with-updates.osm.pbf

If planet-recent.osm.pbf does not exist locally, the most recent planet file will be downloaded, before applying hourly updates to it. (Note: This download is nearly 40Gb.) By default, files are downloaded from planet.openstreetmap.org. Amazon Web Services also provides OSM planets through its Public Datasets program. To instead download the planet file from AWS:

1. Make sure you have your AWS credentials configured locally.
2. Append the --s3 flag.

Note that an entire OSM planet may be upwards of 40Gb in size! In other words, you should have ~80Gb free disk space before running this command.

For complete help on command-line arguments:

osm_planet_update -h

osm_planet_extract

Cut up an OSM planet file into one or more extracts, defined by bounding boxes or polygons. Each extract is assigned a name. (This is like a mini version of Mapzen Metro Extracts!)

To create a single extract:

osm_planet_extract --outpath=data/osm_extracts --bbox=-122.737,37.449,-122.011,37.955 --name=san-francisco planet-latest.osm.pbf

To specify more than one bounding box of tiles to download, list the extents in a CSV file or GeoJSON file. For example:

osm_planet_extract --outpath=data/osm_extracts --csv=data/bboxes.csv planet-latest.osm.pbf

For complete help on command-line arguments:

osm_planet_extract -h

osm_extract_download

Download regularly updated OSM extracts for popular cities and regions from OSM Extracts by Interline. Browse available extracts using the web interface or the GeoJSON file. Anyone can browse the available extracts or propose changes to the extract bounding boxes on GitHub. A subscription is required to download extracts, to cover hosting costs and keep the service sustainable. (See the OSM Extracts website for more information on how profits are donated to OpenStreetMap and other "open" efforts.)

To download the latest copy of an extract (if abcd is your Interline API token and abidjan_ivory-coast is the ID for your chosen extract region):

osm_extract_download --api-token=abcd abidjan_ivory-coast

You can also download extracts in GeoJSON format by using --data-format=geojson. Warning: these can be very large files, but may be useful for filtering and displaying on a web map.

For complete help on command-line arguments:

osm_extract_download -h

(Note: OSM Extracts is a hosted and managed version of the PlanetUtils library. Every day, the pipeline runs the osm_planet_update and osm_planet_extract commands.)

osm_planet_get_timestamp

A simple utility to print the timestamp of an OpenStreetMap PBF file.

osm_planet_get_timestamp planet-latest.osm.pbf

elevation_tile_download

Download elevation tiles from the Terrain Tiles in the AWS Public Datasets program. Download for the entire planet, only tiles within a single bounding box, or within multiple bounding boxes.

Elevation tiles are available in a variety of formats. This command supports the download of:

• GeoTIFF (default): extension .tif in Web Mercator projection, 512x512 tiles
• Skadi: extension .hgt in unprojected latlng, 1°x1° tiles

To download the entire planet in Skadi tiles (which will require about 1.6Tb of space!):

elevation_tile_download --format=skadi --outpath=data/elevation

To download GeoTIFF tiles to cover a single bounding box at a specified zoom level:

elevation_tile_download --outpath=data/elevation --bbox=-122.737,37.449,-122.011,37.955 --zoom=10

To specify more than one bounding box of tiles to download, list the bounding boxes in a CSV file or GeoJSON file. For example:

elevation_tile_download --outpath=data/elevation --csv=data/bboxes.csv

By default tiles are downloaded from the AWS us-east-1 region. To instead download from the eu-central-1 region:

elevation_tile_download --outpath=data/elevation --region=eu-central-1

Skadi tiles are served gzipped and are inflated on write by default. A full planet is about 1.6 TB inflated, against roughly 350-500 GB kept compressed. Valhalla reads .hgt.gz natively, so --keep-compressed stores them packed:

elevation_tile_download --format=skadi --keep-compressed --csv=bboxes.csv --outpath=data/elevation

Compression varies with terrain — roughly 3x for mountainous tiles and up to 20x for flat or open-ocean ones. Tiles already on disk in either form are recognised, so turning the flag on or off does not re-download a complete cache. Note that it does not convert one either: a run whose tiles are all present in the other form writes nothing and says so. Note that Valhalla inflates compressed tiles as it reads them, trading CPU for less I/O per tile; that is usually a win on network storage.

Tiles are downloaded concurrently. The work is latency-bound rather than bandwidth-bound, so this is worth roughly an order of magnitude. Measured over the 528 tiles of --bbox=-122.8,37.4,-121.9,38.2 --zoom=13: about 104 seconds with --workers=1 versus about 10 seconds at the default --workers=16. Adjust it to suit your connection (1-64):

elevation_tile_download --workers=32 --bbox=-122.8,37.4,-121.9,38.2 --zoom=13

Transient connection errors and retryable responses (408, 429, 5xx) are retried with backoff. As before, a tile that still cannot be fetched fails the run; tiles already present in --outpath are skipped, so re-running resumes where it stopped.

For complete help on command-line arguments:

elevation_tile_download -h

elevation_tile_merge

After downloading elevation tiles using the elevation_tile_download command, use this command to merge together multiple tiles. You can optionally resample elevation values as part of the merge process.

This command only operates on GeoTIFF format elevation tiles.

Warnings: merging lots of tiles can be resource intensive!

To merge a directory of GeoTIFF files:

elevation_tile_merge single_tile.tif geo_tiff_tiles/

For complete help on command-line arguments:

elevation_tile_merge -h

valhalla_tilepack_list

Use Valhalla Tilepacks from Interline to power your own instances of the Valhalla routing engine. Anyone can list available planet tilepacks. A subscription and an API key are required to download tilepacks.

To list all available planet tilepacks:

valhalla_tilepack_list

For complete help on command-line arguments:

valhalla_tilepack_list -h

valhalla_tilepack_download

Download Valhalla Tilepacks from Interline to power your own instances of the Valhalla routing engine. A subscription and an API key are required to download tilepacks.

Initial set-up:

1. Sign up for Valhalla Tilepacks from Interline.
2. Set your API token as an environment variable (INTERLINE_API_TOKEN) or use it as an argument to the command

To download the latest planet tilepack (if abcd is your Interline API token):

valhalla_tilepack_download --api-token=abcd

or set your API token as an environment variable, and download the latest planet tilepack:

export INTERLINE_API_TOKEN=abcd
valhalla_tilepack_download

For complete help on command-line arguments:

valhalla_tilepack_download -h

Specifying extract extents

When extracting multiple bounding boxes or polygons from an OSM planet, or when downloading multiple bounding boxes of elevation tiles, you can specify your extents in a single file: CSV, GeoJSON, or Osmosis .poly format.

Bounding box file: CSV format

Do not include a header row. The format is as follows:

[name for extract],[left longitude],[bottom latitude],[right longitude],[top latitude]

For example:

san-francisco,-122.737,37.449,-122.011,37.955
dar-es-salaam,38.894,-7.120,39.661,-6.502

To determine a bounding box, try the tool at http://bboxfinder.com/

Bounding box/polygon file: GeoJSON format

Alternatively, you can specify the bounding boxes or polygons as features in a GeoJSON file, using the --geojson argument.

osm_planet_extract --geojson=examples/test.geojson examples/san-francisco-downtown.osm.pbf

To draw extents in GeoJSON, try the tool at http://geojson.io/

Polygon file: Osmosis .poly format

Extents can also be given as an Osmosis .poly file, the format used by sources such as JamesChevalier/cities:

osm_planet_extract --toolchain=osmium --poly=berlin.poly --outpath=data/osm_extracts planet-latest.osm.pbf
elevation_tile_download --poly=berlin.poly --zoom=12 --outpath=data/elevation

Note the --toolchain=osmium: only Osmium extracts along polygon boundaries. The osmosis and osmctools toolchains take a bounding box, so they widen a polygon to its extent — which for a multi-section file, such as a mainland plus an offshore island, can be far larger than intended. PlanetUtils warns when that happens.

The file's first line names the extract. A section name prefixed with ! subtracts that ring, producing a hole. Multiple sections become a single multi-polygon extract under that one name.

Switching toolchains

PlanetUtils wraps up a number of libraries, including Osmosis, Osmium, and OSM C Tools. Some PlanetUtils commands allow you to switch which library is used to perform the operation:

PlanetUtils command	argument flag	default	options
osm_planet_update	--toolchain	osmium	osmium, osmosis
osm_planet_extract	--toolchain	osmosis	osmosis, osmium, osmctools

Changed in 0.5.0: osm_planet_update now defaults to --toolchain=osmium rather than osmosis. Osmium needs no system binaries on any platform, because pyosmium-up-to-date ships inside the osmium wheel, whereas Osmosis requires Java. The container entrypoint already defaulted to osmium, so the CLI and the container now agree. --toolchain=osmosis still works.

If you are using osm_planet_extract with --toolchain=osmium, you can also use the --strategy= option to select simple, complete_ways (default) or smart.

If you are using osm_planet_update with --toolchain=osmium, you can also use the --size= option to limit the amount of updates downloaded from the OSM replication server. Osmium requires this data to be held in memory. The default is 1024 megabytes.

Support

To report a bug, please open an issue.

Interline Technologies also provides professional support and consulting services around this and other related tools. Contact us at info@interline.io for more information.