[GIS] GIS for large (>> 2GB) files

arcgis-desktoparcmap

I'm a novice GISer who needs to do some basic computations on very large (say ~20GB) files. Nothing complicated, merge, intersect, add XY columns kind of thing. It's a necessary step for a project, but it won't be something I do frequently. From what I understand, ArcMap refuses to work with files over 2GB and I think QGIS would crash with these files.

As I can see it, my options are:

  1. Knuckle down and learn GIS python scripting just for this one computation
  2. Break my computation down into 2GB chunks and run it 10 times, hoping none of the intermediates exceed 2GB.
  3. Find software or file formats that can handle large files.

Number 3 would be the easiest if such software exists but I'm not sure it does. If not, do I have any options other than 1 and 2? Both are unpalatable (being new at scripting for 1, tedium and possibility that it doesn't work for 2). Are there other options?

Best Answer

I agree with @ChrisW in that ArcGIS can work with massive datasets in file geodatabases (1TB), yet shapefiles have size limitations (2GB). A few recommendations:

  1. Make sure your vector data are in a file geodatabase.
  2. Ensure that 64-bit background geoprocessing is enabled.
  3. Simplify your data in any way you can prior to processing. If your objectives allow, 1) simplify the geometry, 2) delete duplicate features, and/or 3) dissolve.
  4. Make sure your spatial data are all in the same projection, otherwise ArcGIS will be forced to project-on-the-fly, which can be computationally expensive.
  5. If your workflow allows, switch to a raster-based analysis. Raster-based processing can be much more efficient than vector in many cases.
  6. Make sure you have a computer with suitable processing power. These days, 32GB RAM and 8 cores are common. Often, large organizations such as Universities have high powered systems available for use in, for example, the Computer Science or Engineering department--these folks are usually willing to help you out and let you use the system.
Related Question