[GIS] Mapnik renders only “blank” tiles

coordinate systemmapnikopenstreetmap

I'm new to the fascinating GIS world, and I've been researching the whole topic for the past few days. My goal is to serve planet-wide maps with OSM data on my own servers.

My stack is:

In order to import and style OSM data, I'm using

Since the full planet file is huge, I'm currently using a extract of Sao Paulo, which is good/small enough for testing.

Before I reach to the problem I'm having, I'll guide through the steps I took so far:

What worked

PostgreSQL and PostGIS are properly installed. I'm using 9.6 for the former, 2.3 for the latter. I've also added the postgis_topology and hstore extensions, which will be used later.

I have osm2pgsql 0.92.0 installed. In order to import the Sao Paulo extract, I use

osm2pgsql -G U <user> -d <db> -C 1000 -W --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua <pbf>

The hstore, style and tag-transform-script arguments are required for the proper utilization of OSM's style, as described here.

As described on the openstreetmap-carto installation, I've also added custom indexes and downloaded the required shapefiles and fonts (except for the emoji ones).

I verified the extract data is correctly loaded on the database by using QGIS. I was able to view and query with all points, polygons, lines and roads. It's all there.

Problems

The next step is rendering the tiles, and that's where I'm having trouble. I have mapnik and python-mapnik 3.x, as well as carto 0.18.2.

From the OSM's project.mml, I've generated my own project.xml using carto. I've made the following changes to the result:

  1. Add the Postgres username (Parameter user on Mapnik's XML). Remaining connection info is grabbed from ~/.pgpass.
  2. Replace relative directory data/<shape>.shp to /full/path/data/<shape>.shp
  3. Replace relative directory symbols/<symbol>.svg to /full/path/symbols/<symbol>.svg

I sort of unit-tested these changes: incorrect directories for shapefiles and symbols will raise an error when Mapnik tries to load the style. Same applies for incorrect DB information.

And here's the problem: every tile rendered by Mapnik is "blank" (blue-ish background, which is the defined background for OSM). Here's what I've tried to fix this problem:

Things I've tried

Maybe the generated style is invalid somehow. Maybe some polygon/shape (the ocean's?) is covering another one.

I've installed Kosmtik, and it worked fine! I was able to browse through the extract, inspect data, zoom in and out on all levels.

Maybe I am generating tiles at the wrong location

I've set up TileStache with Mapnik as the provider. Then, I've set up redirect rules from http://[abc].tile.openstreetmap.org to http://myserver/<layer> using switcheroo, as described here. This means that accessing openstreetmaps.org would have all tiles rendered on my server.

Interestingly, zoom 1 works (the one with 4 tiles). I suppose it only uses the planet/world shapefile, which was downloaded with the style. All else, regardless of the zoom or location, renders to "blank". So even when browsing at the extract's location, it won't render correctly.

Note that, when using Kosmtik, I was able to see the extract's name from as low as zoom 4.

Maybe I am generating tiles at the wrong location

Projections are hard. Maybe there is something being "mistranslated" somewhere.

So I analyzed the requests Kosmtik was making (since these were working). It uses the Modestmaps library, which translates to Slippy format (<zoom>/<x>/<y>), same format used by Leaflet, TileStache and openstreetmap.org.

For instance, requesting 17/48561/74357.png on Kosmtik works, but the same tile returns as "blank" on TileStache. My theory was that somehow TileStache was translating it to another coordinate, outside the extract area. (Even though TileStache was using "spherical mercator" as the projection).

So I've decided to use Mapnik's python bindings directly. So far I've handled three different types of projections, here's what I know about them. Please correct me if I'm wrong:

  • EPSG4326/WGS84 – Usual lon/lat projection, same as used on GPS. Unit is degrees.
  • EPSG3857/{Web,Google,Pseudo} mercator/900913 – Used on most tiled web maps. Unit is in meters (not "real" meters, as it distorts heavily near the poles).
  • "Slippy format" – Not a projection, but yet another format to translate.

So, the Slippy 17/48561/74357 would be near -46.632(lon), -23.531(lat) on EPSG4326, and near -5191050.49, -2696361.67 on EPSG3875/Google mercator.

Using python, I've generated these coordinates' tiles directly at Mapnik. The result was always blank tiles. I believe I'm using the correct projections (more on this later).

First, one thing to note. It's clear the problem is not in Mapnik, nor the style. For one, Kosmtik uses Mapnik as backend, and it works fine. Second, if I purposely generate a tile outside the extract area on Kostmik, I get a blank tile, as expected.

This leads me to believe I'm using the wrong projection at Mapnik's python script and TileStache. Before showing the code, one remark about OSM (again, please correct me if I'm wrong):

  • OSM "native" data is stored in EPSG4326
  • osm2pgsql imports OSM data as EPSG3857, and that's the projection on Postgis.
  • Mapnik's style (XML) has two projections, input and output, explained here:
    • Input projection is defined at the Map object, and is the "coordinate system in which the map is rendered"
    • Output projection is defined at the Layer object, and must be the same as my data.

My resulting Mapnik's XML has:

<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" background-color="#b5d0d0">
#snip#
<Layer name="world" #snip#  srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs   +over">

So, the way I see it, both the input and output projections are in "merc"(3857), and that's correct! My data is stored with 3857, and the resulting map is expected to be at 3857.

Finally, here are the different Mapnik calls I've tried:

Box2d with 3857 coordinates:

map = mapnik.Map(800, 800)
bbox = Box2d(-5191050.49, -2696361.67, -5191059.49, -2696370.70)
mapnik.load_map(map, '/path/to/style.xml')
map.zoom_to_box(bbox)
mapnik.render_to_file(map, 'output.png')  # Blank

Box2d with 4326 coordinates transformed to 3857

merc = mapnik.Projection('+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over')
longlat = mapnik.Projection('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs')

bbox = mapnik.Box2d(-23.53, -46.63, -24.03, -47.03)
transform = mapnik.ProjTransform(longlat, merc)
merc_bbox = transform.forward(bbox)
# load, zoom, render -> Blank

I've also tried with different styles. Result is always blank, but that's unexpected because I believe these coordinates would be within my extract. When trying to render coordinates outside the extract, it correctly renders as blank.

Anyway, I'm lost and I was wondering if anyone could shed some light, I must be missing something. But so far, I've been battling with this issue for well over 24 hours and I have no idea what could be wrong.

Best Answer

I can't give you a response as I don't see what's wrong in your config. I've made a similar stack (Postgis + Mapnik + Mapproxy + Apache).

I see that your zoom=1 works. I think I could be :

  • a broken CartoCSS when you call higher zoom
  • a source file (ex. a .SHP) that must appears in higher zoom but is not accessible by renderd (or it has no read rights)
  • a problem in TileStach. I don't know it, but with MapProxy, we had memory issues with tiles that were too big (2048x2048, because we wanted to make tiles for 4k screens). Look a its logs perhaps.
  • another thing, I'm relatively new in GIS too !

I suggest you to try with a simpler stack, like me in case of blank or pink tiles :

  • Use a tested project that have already worked (I use https://github.com/mapbox/osm-bright)
  • Ignore Tilestach and connect your client directly to mapnik (renderd). The cache complicates the debugging process, don't forget to empty it each time you updates your CartoCSS project.

Sorry for that "blurry" response!

Related Question