[GIS] How to load geographic data into Hadoop (Hive) and then to be able to call spatial functions on them

hadoophiveshapefile

I need to load some geographic data into Hadoop (Hive), and then, to be able to call some spatial functions on them using Hive.

My input data format is ESRI shapefiles (but could be another format, I could transform if necessary).

I know that ESRI has released a very basic framework for working with spatial data within Hadoop via Hive, GIS Tools for Hadoop by ESRI, but it seems that I need to have my data in ESRI JSON and I don't have ESRI tools to convert my data into that format.

Is there any alternative way to load geodata into Hadoop or is there a way to convert my ESRI shapefiles into ESRI JSON?

Best Answer

ST_Geometry for Hive in the GIS Tools for Hadoop, can load data from several different formats: Well-Known Text (WKT), Well-Known Binary (WKB), GeoJSON, Esri JSON, and coordinates as a list of numbers. Please see the documentation for the constructor functions.

As for shapefile, the java library to read shapefiles by Mansour Raad may be of use.

Related Solutions

[GIS] How to process GeoJSON data and create hive table for spatial analysis

Maybe you could detail a bit more the attempts you made, in order to understand what happened.

In any case, from my experience there is not a smooth pipeline to import things from PostGIS (or any OGC standard) to Hadoop GIS, which is a shame... ESRI has ticked the box of Free and Open Source Software by providing this tool (which by the way, it is one of the few that I know that adds spatial support to BigData), but has failed on the interoperability. AFAIK, the support they give for loading spatial types covers their proprietary JSON format, which is not really convenient unless you are using ArcGIS... What I have done up to now, was exporting the data from PostGIS as either WKT or GeoJSON, and then load it into hive using the csv deserializer.

CREATE EXTERNAL TABLE grid_10 (id bigint, json STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 's3://some-url/data/grids/geojson';

This means the geometry is actually stored as text, but you can easily instantiated on the go for your queries, using the ST_GeomFromGeoJSON or ST_GeomFromText functions. For instance:

create table grid_cnt as 
SELECT grid_10.id, count(grid_10.id) as ptcnt FROM grid_10 JOIN tweets     WHERE ST_Contains(ST_GeomFromGeoJSON(grid_10.json),ST_Point(tweets.longitude,   tweets.latitude))=true GROUP BY grid_10.id;

What I actually failed to do, and I suspect its a bug, was instantiating a spatial table with a geometric field, using this conversion. You can read more about it on my post.

[GIS] Lossless conversion from shapefiles to XML/JSON/Text

All of the vector formats supported by GDAL/OGR are listed here. With each driver, check out the creation options to control the output. These are passed to ogr2ogr using -dco and -lco flags.

Good text-based output drivers include:

CSV - be sure to use -lco GEOMETRY=AS_WKT to get the well-known text geometry
GeoJSON
GML
KML
LIBKML.

Best Answer

Related Solutions

[GIS] How to process GeoJSON data and create hive table for spatial analysis

[GIS] Lossless conversion from shapefiles to XML/JSON/Text

Related Question