[GIS] Self hosting vector tiles

hostingmapbox-gl-jsvector-tilesweb service

I'm building a web application, which involves (currently) a REST backend and a frontend SPA. The backend is hosting massive geo-enabled data, and the frontend is displaying it on a mapbox map. I'm hitting performance issues, because of the stack I set up. Here's the stack, when it comes to displaying a map.

    MapboxGl.js player
        ^          ^
        | (HTTP)    \
 Node.js backend     Mapbox APIs vectors
        ^
        |
     MongoDB

The HTTP endpoints serve both GeoJSON and REST data. The GeoJSON part is dynamically generated from a set of MongoDB collections (say, gpspoints, images, lines etc.). The Mapbox API is my Mapbox Studio style.

I'm looking for the best alternative to this design. And reading about maps, vectors, tiles, rendering and stuff, I came up to the idea that I should trade my GeoJSON server for a Vector tiles server (only for the map part). I'd like to end up with something like.

         MapboxGl.js player
       ^         ^           ^
      /          | (HTTP)     \
Vector tiles  Node.js backend   Mapbox APIs vectors
   Server            ^
     ^               |
     |               |
Other backend ?   MongoDB

I'm struggling at the point of making design choices.
The questions I have in mind are :

About generating :

I came to the idea that I must generate my vector tiles from a GeoJSON. Is that correct ? Would it be possible to generate vector tiles from scratch, I mean, say, from a node.js routine script, parsing datas from various sources (.csv, other databases, etc.) and make operations on it (using turf.js), then generating vector stuff, like pushing, maybe updating, deleting, etc ?
I found geojson-vt which seems to be able to translate GeoJSON to vector tiles, but only in json format. I found vt-pbf, am I on the right direction ?

About serving

Once I would have been able to generate whatever-format-vector-tile, I am going to be willing to serve it. I found Tilelive.js which seems very good, and has plenty of modules, including backend ones. Is that a good option ?
Speaking generaly, would it be possible not to generate any static file stuff, and maybe dynamically serve any-vector-format from a database, through something like tilelive ?

Additional info :

My datas are "mutable", but I have routine scripts that run at night and parse new datas. Thus, live-map-displayed-data mutability is not a requirement : I can update / re-parse a whole dataset at night to serve it on the morning
Both generating and hosting on Mapbox studio is not an option, since my data is critical and must be hosted in specific coutries. A self-hosted option is mandatory, any cloud-based stuff is not ok :/

This question is redundated in softwareengineering.stackexchange

Best Answer

I had good results following this tutorial on using tippecanoe to generate compressed protobuffer vector tiles.

I just use Amazon S3 for hosting, but for your application a dedicated server running Nginx with a memory cache in front would work well.

You can certainly generate vector tiles on the fly, but if your data is not dynamic enough it's probably not worth the processing overhead.

Related Solutions

Self-Hosting Mapbox Tiles – Guide to Self-Hosting Mapbox Vector Tiles

As pointed out by @Greg, instead of TileStream (my first attempt) you should use Tilelive to host your own vector tiles.

Tilelive isn't a server itself but a backend framework that deals with tiles in different formats from different sources. But it's based on Node.js so you can turn it into a server in a pretty straight-forward way. To read tiles from a .mbtiles source as exported by Mapbox Studio, you need the node-mbtiles tilelive module.

Side note: Current Mapbox Studio has a bug under Windows and OS X that prevents an exported .mbtiles file to show up at your chosen destination. Workaround: Just grab the latest export-xxxxxxxx.mbtiles file in ~/.mapbox-studio/cache.

I found two server implementations (ten20 tile server by alexbirkett and TileServer by hanchao) who both use Express.js as a web app server.

Here is my minimalistic approach which is loosely based on these implementations:

Install Node.js
Grab the node packages with npm install @mapbox/tilelive @mapbox/mbtiles express

Implement the server in the file server.js:

 var express = require('express');
 var http = require('http');
 var app = express();
 var tilelive = require('tilelive');
 require('mbtiles').registerProtocols(tilelive);

 //Depending on the OS the path might need to be 'mbtiles:///' on OS X and linux
 tilelive.load('mbtiles://path/to/osm_roads.mbtiles', function(err, source) {

     if (err) {
         throw err;
     }
     app.set('port', 7777);

     app.use(function(req, res, next) {
         res.header("Access-Control-Allow-Origin", "*");
         res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
         next();
     });

     app.get(/^\/v2\/tiles\/(\d+)\/(\d+)\/(\d+).pbf$/, function(req, res){

         var z = req.params[0];
         var x = req.params[1];
         var y = req.params[2];

         console.log('get tile %d, %d, %d', z, x, y);

         source.getTile(z, x, y, function(err, tile, headers) {
             if (err) {
                 res.status(404)
                 res.send(err.message);
                 console.log(err.message);
             } else {
               res.set(headers);
               res.send(tile);
             }
         });
     });

     http.createServer(app).listen(app.get('port'), function() {
         console.log('Express server listening on port ' + app.get('port'));
     });
 });

Note: The Access-Control-Allow-... headers enable cross-origin resource sharing (CORS) so webpages served from a different server may access the tiles.

Run it with node server.js

Set up the webpage using Mapbox GL JS in minimal.html:

 <!DOCTYPE html >
 <html>
   <head>
     <meta charset='UTF-8'/>
     <title>Mapbox GL JS rendering my own tiles</title>
     <link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.css' rel='stylesheet' />
     <script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.js'></script>
     <style>
       body { margin:0; padding:0 }
       #map { position:absolute; top:0; bottom:50px; width:100%; }
     </style>
   </head>
   <body>
     <div id='map'>
     </div>
     <script>
       var map = new mapboxgl.Map({
         container: 'map',
         center: [46.8, 8.5],
         zoom: 7,
         style: 'minimal.json'
       });
     </script>
   </body>
 </html>

Indicate the location of the tile source and style the layers with the following minimal.json:

 {
   "version": 6,
   "constants": {
     "@background": "#808080",
     "@road": "#000000"
   },
   "sources": {
     "osm_roads": {
       "type": "vector",
       "tiles": [
         "http://localhost:7777/v2/tiles/{z}/{x}/{y}.pbf"
       ],
       "minzoom": 0,
       "maxzoom": 12
     }
   },
   "layers": [{
     "id": "background",
     "type": "background",
     "paint": {
       "background-color": "@background"
     }
   }, {
     "id": "roads",
     "type": "line",
     "source": "osm_roads",
     "source-layer": "roads",
     "paint": {
       "line-color": "@road"
     }
   }]
 }

Serve the webpage and rejoice.

[GIS] Wrapping vector tiles generated by geojson-vt

I have very similar problem, except I try to use angular-leaflet-directive. So far i couldn't find much about it. However, please have a look at this example and the block itself. This may give you more ideas on how to link geojson-vt with leaflet.

Best Answer

Related Solutions

Self-Hosting Mapbox Tiles – Guide to Self-Hosting Mapbox Vector Tiles

[GIS] Wrapping vector tiles generated by geojson-vt

Related Question