Google Map Tile Optimization – Performance of Tile Creation Processes

bing-mapsgdalgoogle mapsoptimizationtiles

I know that question is rather vague but please bear w/me. I'm trying to get an idea of what sort of product performance – specifically timing – people have seen for various methodologies they have used to create google/bing map tiles. There are a slew of methods for how to do this (e.g. gdal2tiles, FME, maptiler, etc). An initial attempt at simply taking a large PNG and creating tiles using imagemagick, on a pretty decent linux server, yielded some pretty long processing times and so I wanted to see what other people are using in production. New tiles would need to be generated at least daily and so turnaround time on this is pretty critical.

The only real requirement is that it can run on a linux server. Obviously, free is better but I don't want to restrict myself to that. The input can be raw gridded/raster data or a large image. The output needs to be image tiles capable of being used as-is in google or bing maps.

Just for the sake of comparison, I'll say that the timings should be for google map's zoom level 7.

I appreciate everyone's help and again I want to apologize for how vague this question probably seems.

UPDATE: As far as the inputs, I currently have multiple (raw) data sources in various formats: netCDF, GRIB, GRIB2. In addition to the raw data itself, I also have the ability to generate really large images of that data which could then be sliced/tiled.

Ideally, I would just be chopping the image up but I am willing to try whatever will get me the fastest results.

Best Answer

I was having issues with gdal2tiles taking quite a while to process a fairly large (380MB, 39K x 10K pixels) tiff into Google tiles for zoom ranges 0-12. On Ubuntu 12.04 64bit without multiprocessing it took just about all day (8 hours) to process the tiff into 1.99 million tiles @ 3.3GB. Like @Stephan Talpalaru mentions above, making gdal2tiles run in parallel is the key. Make a backup of your original gdal2tiles.py, then install the patch from within the directory that houses gdal2tiles.py (mine was /usr/local/bin):

$ sudo patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Now run gdal2tiles like you normally do. I got an incredible increase in performance, with all 4 of my cores (Intel Core i7 3.4GHz) pegged:

$ time gdal2tiles.py -p raster -z 0-12 -w none ds1105-2235df023_23_b.tif gdal-tiles12
Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.

real    39m8.242s
user    104m6.808s
sys 9m16.036s

So from ~8 hours to 39 MINUTES. Game changer.

Related Question