# Data

This directory holds intermediate files used to generate the `.pmtiles` map tile files served by the `www` app. Output tiles are written to `www/static/tiles/`. We don't really need suuuuuuper up-to-date maps for this, since we trim a lot of data; the tiles are mainly used as a way to orient the user as they create their own map.

All tile generation is done via the CLI:

```sh
deno task data <command> [options]
deno task data --help
```

## Dependencies

The following tools must be installed and on `PATH`:

- [`tilemaker`](https://github.com/systemed/tilemaker) — PBF → PMTiles conversion (`build`, `build:world`)
- [`osmium`](https://osmcode.org/osmium-tool/) — PBF extraction (`extract` command)

The following resources are necessary to generate pmtiles:

- [Download a planet `.osm.pbf`](https://wiki.openstreetmap.org/wiki/Planet.osm) for the world-level pmtiles (put it in `./osm/planet-latest.osm.pbf`). For me, using a torrentfile has been the most stable way.
- Download regional .osm.pbf files for regions if you don't want to self-extract (can download with `download:osm`)
- Coastlines WGS84 projection from https://osmdata.openstreetmap.de/data/coastlines.html (put it in `./cli/shared/tilemaker/coastline`) (todo for me: create a `download:coastlines` cli cmd)

## CLI Commands

| Command                                    | Description                                                         |
| ------------------------------------------ | ------------------------------------------------------------------- |
| `list [--search <term>]`                   | Browse available region slugs                                       |
| `download:osm <region> [--force]`          | Download `.osm.pbf` from Geofabrik                                  |
| `download:poly [region] [--all] [--force]` | Download `.poly` boundary file(s)                                   |
| `extract <region> --from <source>`         | Carve a sub-region from a larger PBF using osmium                   |
| `build <region>`                           | Convert `.osm.pbf` → `.pmtiles` using tilemaker                     |
| `trim:world`                               | Strip planet PBF to only data needed for world tiles (saves memory) |
| `build:world`                              | Build world-scale basemap tiles                                     |
| `clean --region <slug> \| --all`           | Remove intermediate files                                           |

## World Tiles

The world basemap is built from OSM data using a tilemaker pipeline, sourced from a full planet PBF. Place `planet-latest.osm.pbf` in `data/osm/` and run:

```sh
deno task data trim:world                         # recommended first step — see below
deno task data build:world [--maxzoom <5|7|9>]   # default maxzoom: 7
```

Output is written to `www/static/tiles/world_z<maxzoom>.pmtiles`. The config and Lua script used are separate from the regional ones:

- `data/cli/shared/tilemaker/config.world.json` — layer definitions for the world overview
- `data/cli/shared/tilemaker/process.world.lua` — feature processing logic

Note: this takes a super long time to run.

### Trimming the planet file (recommended)

The full planet PBF (~75 GB) contains a huge amount of data that is irrelevant at world zoom levels — buildings, addresses, shop/amenity POIs, minor roads, etc. Running `trim:world` uses `osmium tags-filter` to produce a much smaller `planet-trimmed.osm.pbf` containing only the tags that `process.world.lua` actually reads:

- Place nodes (countries, states, cities, towns, villages)
- Natural peaks and volcanoes
- Administrative boundaries
- Major roads (motorway → secondary) and ferry routes
- Main-line railways
- Rivers, waterways, water polygons
- Landcover and landuse polygons
- Parks and nature reserves

```sh
deno task data trim:world            # creates data/osm/planet-trimmed.osm.pbf
deno task data trim:world --overwrite  # re-run and replace an existing trimmed file
```

When `planet-trimmed.osm.pbf` is present, `build:world` will automatically use it instead of the full planet file. This significantly reduces tilemaker's peak memory usage and overall build time.

## Regional Tiles

Regional tiles follow a three-stage pipeline:

```
Geofabrik (download) → osmium (extract; optional) → tilemaker (build) → .pmtiles
```

### 1. Download

Regional `.osm.pbf` data and `.poly` boundary files are sourced from [Geofabrik](https://download.geofabrik.de/). Region slugs match Geofabrik's hierarchy (e.g. `asia/japan/kansai`, `europe/monaco`).

`.poly` files for all 499 Geofabrik regions are already committed to `data/poly/` and don't need to be refreshed under normal circumstances. OSM `.pbf` files are gitignored due to size and must be downloaded as needed.

```sh
deno task data download:osm europe/monaco
deno task data download:poly europe/monaco   # already present; only needed to refresh
```

### 2. Extract (optional)

If you already have a large regional PBF, you can carve out a sub-region using `osmium` rather than downloading from Geofabrik again:

```sh
deno task data extract asia/taiwan --from asia
```

This requires both the source PBF and the target `.poly` file to be present.

### 3. Build

Convert `.osm.pbf` to `.pmtiles` using `tilemaker`:

```sh
deno task data build europe/monaco
```

Output is written to `www/static/tiles/<region-name>.pmtiles`. The tilemaker config and Lua processing script are:

- `data/cli/shared/tilemaker/config.json` — layer definitions (OpenMapTiles schema, zoom 0–14)
- `data/cli/shared/tilemaker/process.lua` — feature processing logic

## Common Workflows

**Small country — direct download:**

```sh
deno task data download:osm europe/monaco
deno task data build europe/monaco
```

**Country carved from a larger extract:**

```sh
deno task data download:osm asia # large, but reusable across multiple countries
deno task data extract asia/taiwan --from asia
deno task data build asia/taiwan
```

## Directory Structure

```
data/
  osm/          Downloaded/extracted .osm.pbf files (gitignored)
  poly/         Boundary .poly files for all 499 Geofabrik regions (committed)
  cli/          CLI source code
```

## External Resources

- **[Geofabrik](https://download.geofabrik.de)** — regional OSM extracts; provides both `.osm.pbf` and `.poly` files
- **[OpenStreetMap](https://planet.openstreetmap.org)** — planet PBF source for world basemap tiles
