Store a GeoJSON file in a parquet format

fastparquetgeojsongeoparquetjupyter notebookpython

I have a large GeoJSON file which I store and read as follows:

geomap.to_file(path_to_output + 'geomap_cleaned.geojson', driver="GeoJSON")

geomap = import_geojson(path_to_data, path_to_cadgis, 'geomap_cleaned.geojson')

My issue is that geomap is a large file (1G) and the kernel in my Jupyter Notebook crashes most of time when I try to read it.

I made an attempt to save the GeoJSON file in a fast parquet format (my aim is to make things faster and more optimised)

geomap.to_parquet(path_to_data + 'geomap_cleaned.gzip', compression='GZIP', engine='pyarrow')

but i get an error

ArrowInvalid: Cannot parse URI: './Source data/geomap.gzip'

How can I solve this problem? And how can I make sure the geometries, when stored in parquet format, are not corrupted?

I also installed successfully geoparquet by using pip install geoparquet but when i save the file as :

geomap.to_geoparquet('geomap_cleaned.geoparquet')

I get an error

TypeError: Object of type CRS is not JSON serializable

What is wrong here?

Best Answer

You can use Geopandas to covert the geojson to geoparquet. Sometimes you need to explicitly set the CRS, for this example I am assuming its 4326.

import geopandas as gpd
geojson_fn='test'
geoparquet_fn='output_geoparquet'
gdf=gpd.read_file(f'{geojson_fn}.geojson')
gdf.set_crs(epsg=4326)
gdf_projected.to_parquet(f'{geoparquet_fn}.parquet')

Related Solutions

[GIS] OpenLayers GeoJSON Data Store

One thing that you could do is long polling. Leave your HTTP connection open until something has changed, and then close it. This is easy with OpenLayers, using e.g. (untested):

layer.events.register("loadend", this, function() {
    layer.refresh({params: {poll: true}});
});

On the server, you would look at the request string, and if it has a poll param set to true, you would not close the request until something has changed.

[GIS] Format a specific GeoJSON file into the correct format

You could write a simple script in (for example) Python that will process the data for you.

import json
from itertools import chain

Open the file and read the data into a Python dictionary:

isil = json.load(open('isil.en.json'))

The points object is just a list of feature collections, so you can use the python itertools library to help chain the features in those collections together:

features = list(chain.from_iterable(fc['feature'] for fc in isil['points']))

And finally write a new feature collection with all 2818 features out to a file.

feature_collection = {
    "type": "FeatureCollection":,
    "features": features
}

with open("isil_points.geojson", "w") as f:
    json.dump(feature_collection, f)

And that should be able to be loaded up into a system of your choice. Looking at the data You'll probably have to do some manual cleaning as well (some "total" locations and a few points that don't have a location), but that should be a start.

Related Question

[GIS] How to store GeoJSON format in Firebase so it can be used for Leaflet GeoJSON