[GIS] Large Open Street Map OSM file to Shapefile. Is there any hope for 32-bit FME

fmeopenstreetmaposmosispostgisxml

I have a large OSM XML file (around 8 GB) that I was hoping to process in FME. I have a workbench that I previously used to process smaller files (500 MB) into feature type-specific Shapefiles (e.g. waterway_line, waterway_polygon etc.).

In this workbench I simply added the OSM reader, left everything as default, and FME helpfully grouped the features into useful categories – sending each to the appropriate Shapefile.

Unfortunately my 32-bit OS cannot manage the larger file. Because the entire XML file has to be read at the start of the transformation the memory just builds and builds until the process limit is reached before FME has even read all the incoming features.

One of the things FME does by default is build polygons from lines (raw OSM data doesn't recognise polygons), so if I break the area up into smaller files I could end up with polygons that are artificially split (e.g. landuse areas) and end up being labelled twice and having spurious borders through the middle.

One option is to load my OSM file into PostGIS, and FME can read data from PostGIS (I'm assuming this will help overcome the memory limit as the database can be read row-by-row) but then FMW won't know to group features into those useful categories and which Shapefiles they should end up in.

Essentially, it looks like I simply can't use FME to process a big OSM file to Shapefiles. What I really want to know is:

  1. Is there any way I can convince FME to do what I want?
  2. Are there other tools that can create similar convenient Shapefile groupings of feature types?
  3. EDIT How do Cloudmade and GeoFabrik create their huge Shapefiles? Are they using a different tool I don't know about?

Any input much appreciated.

(note: I don't have access to a 64-bit FME machine, and the pre-processed Shapefiles from Cloudmade / GeoFabrik are insufficient for my requirements)

Best Answer

Updated: OSM Reader for FME 2013 (Beta)

=========================== BUILD 13082  20120417 ===========================
===========================================================================
OSM reader: Updated to support reading very large datasets, for example
~764 million features on a European OSM dataset (PR#37345)

ftp://ftp.safe.com/fme/beta/whatsnew.txt

FME 2013 Beta

http://www.safe.com/support/support-resources/fme-downloads/beta/

(Only use in a development/testing environment)

Related Question