I'm looking for an online OSM style editor that lets me style OSM ways and nodes by tags. I know CloudMade's editor is no longer available, and Mapbox's does this but has zoom level rendering limits (eg, won't render buildings below zoom level 13).
My goal is to get a map of the USA (or world if I can) that shows where every OSM building has been mapped (ways with building=). And a second map that shows where every OSM building + street address is (ways with addr:housenumber=).
Here's an example map I did using a state OSM extract in Kentucky, processed with Osmosis, and QGIS to render:
https://twitter.com/CivicDataAlly/status/644963898463023108
But unfortunately the North America extract is too large for me to work with, and QGIS can't even open the large Osmosis processed file on my machine. I also set up osm2pgsql and PostgreSQL/PostGIS databases, but again the file sizes and processing times are too huge.
Is there an online tool where I can style buildings at a low zoom level?
SOLVED
I finally got the image I needed of all the buildings and buildings with street addresses for all of North America, by using the info in the answer below.
I basically setup postgres and used pgAdmin3 to manage it and then osm2pgsql in the command line with the correct options and styles to load the raw .pbf export into the database, then connect to it with QGIS to visualize it.
Once the data was in postgres it was easy and fast to deal with, even displaying it in QGIS on my modest computer.
Buildings
Here are the stats and what I did for the building footprint layer:
osm2pgsql -U mschnuerle -d mschnuerle --create --slim --flat-nodes nabuildings.bin --cache 18000 --number-processes 3 --style buildings.style north-america-latest.osm.pbf;
Processing: Node(794783k 1292.3k/s) Way(52669k 32.51k/s) Relation(471970 89.52/s) parse time: 7507s (2 hours)
node cache: stored: 794783721(100.00%), storage efficiency: 66.31% (dense blocks: 695842, sparse nodes: 243029485), hit rate: 100.00%
Osm2pgsql took 88452s overall (24 hours)
Backup DB size: 4GB
with only this in the buildings.style file (everything else deleted):
way building text polygon
way way_area real linear
Addresses
And here is what I did for only the buildings that had street addresses attached to them:
osm2pgsql -U mschnuerle -d mschnuerle --create --slim --drop --flat-nodes naaddress.bin --cache 6000 --number-processes 4 --verbose --exclude-invalid-polygon --prefix address --style address.style north-america-latest.osm.pbf;
Processing: Node(794783k 1271.7k/s) Way(52669k 31.67k/s) Relation(471970 37.13/s) parse time: 14998s (4h)
Osm2pgsql took 17455s overall (5h)
with this in the address.style file (everything else deleted):
way addr:housenumber text polygon
way way_area real linear
way building text delete
And my machine specs are this, for those asking:
Mac Specs
OS X El Capitan 10.11 Beta
iMac 27" Mid 2011
3.1 GHz Intel Core i5 (4 cores)
25 GB 1333 MHz DDR3
1TB SATA HD
This is the only map online that shows building footprints at this level, as far as I know. If you find any other ones, please let me know here.
I also plan to create all the Mapbox tiles needed using Tippecanoe and post the map online over the next few weeks.
Best Answer
The short answer is, there's no outstanding way.
Options that won't work well
Hadoop
Hadoop and similar tools aren't the solution, as it's entirely possible to do this type of analysis on a reasonably powerful server. You may not have a reasonably powerful server, in which case Hadoop wouldn't be a good option since it needs a cluster.
If you happen to have a Hadoop cluster and are an expert in using it, it's reasonable, but otherwise it's more development time for no gain.
Vector tiles
Vector tiles don't remove any processing steps, they just allow some of the work to be shared by multiple styles. As you've seen with Mapbox Streets' styling, buildings aren't often in low-zoom vector tiles, so you'd have to generate them yourself.
You could stitch together low-zoom vector tiles, but you'd have to use your own rendering toolchain for that, and it would be complex.
Reasonable options
OSM has about 160 million building ways, 35 million addresss nodes, and 21 million ways with addresses. Most of the last are also buildings.
osm2pgsql
osm2pgsql can handle this on reasonable hardware, if you take care to exclude other data. To do this you want a custom .style file which includes only address and building tags. Starting with empty.style, the suggested starting point, we can get
node,way addr:unit text linear node,way addr:housename text linear node,way addr:housenumber text linear node,way addr:street text linear way building text polygon
Everything below here is copied from empty.style, with the "building" and z_order lines removed. The former is above as an actual column, the latter isn't relevant for this use way_area is included, as it is useful
way abandoned:aeroway text phstore way abandoned:amenity text phstore way abandoned:building text phstore way abandoned:landuse text phstore way abandoned:power text phstore way area:highway text phstore node,way aeroway text phstore node,way amenity text phstore way building:part text phstore node,way harbour text phstore node,way historic text phstore node,way landuse text phstore node,way leisure text phstore node,way man_made text phstore node,way military text phstore node,way natural text phstore node,way office text phstore node,way place text phstore node,way power text phstore node,way public_transport text phstore node,way shop text phstore node,way sport text phstore node,way tourism text phstore node,way water text phstore node,way waterway text phstore node,way wetland text phstore way way_area real linear # This is calculated during import
Save it as
buildings.style
You can then import the planet, using something based on the suggested osm2pgsql command line for the planet
osm2pgsql -c -d buildings --style /path/to/buildings.style --slim --drop -C <cache size> --flat-nodes <flat nodes> /path/to/planet-latest.osm.pbf
where
<cache size>
is 24000 on machines with 32GiB or more RAM or about 75% of memory in MiB on machines with less<flat nodes>
is a location where a 24GiB file can be saved.There's a couple of different options used
--drop
gets rid of tables used only during the import and during updates, as I'm assuming you'll update by reimporting--style /path/to/buildings.style
specifies to use the style we wrote aboveThis will take a day or two on a reasonably powered server.
Once the import is done, there's a couple indexes you can add which will help performance
CREATE INDEX planet_osm_polygon_area_18250_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 18250; CREATE INDEX planet_osm_polygon_area_1140_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 1140; CREATE INDEX planet_osm_polygon_area_71_idx ON planet_osm_polygon USING gist (way) WHERE way_area > 71;
When defining the layers in Kosmtik, Tilemill, or another map design studio, include the condition
WHERE way_area > 0.05*!pixel_width!::real*!pixel_height!::real
on any polygon layers.Some things to watch for when visualizing
https://switch2osm.org/loading-osm-data/ has more information on setting up PostgreSQL and installing osm2pgsql
libosmium
libosmium is a library for working with OSM data, and probably the best option for operating directly on the planet file.
QGIS or ArcGIS
If using this much data in QGIS or ArcGIS, you probably want to script it, but you could do a more sophisticated analysis.
osm2pgsql multi-backend
This is very similar to the above, but features a more flexable backend, which can result in tables better suited for exporting to another format.