[GIS] Best way to clip a large raster ECW file

big dataclipecwgdalraster

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely.

Some details of the ECW below

Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0)

file size: 50gb
Size is 450000, 565081
Pixel size: 0.15 0.15
COLORSPACE=RGB
COMPRESSION_RATE_TARGET=9
VERSION=2
Number of bands: 4

The area I want to clip is roughly a 1/5 of the original file.

Here are the methods I have tried out with no success:

  1. Used Arcgis to save the ecw to tiff / other formats… (I quickly gave up)

  2. Used Qgis and its clipper tool… the file creation stayed stuck at about 40%.

  3. Used gdal_translate out of OSGeo4W with other options than Qgis. (Tried that thinking that MAYBE freeing some memory not using Qgis would the trick)

  4. Used gdal_retile thinking that I would cut the image into pieces and grab the one I wanted. The command
    "gdal_retile -ps 10000 10000 -of ecw -tileIndex tile.shp -targetDir input.ecw
    This crashed even quicker"

Does anyone have an idea?

For info I run windows 7 64bits on an i5-3470 3.2Ghz with 16gb of ram.

Best Answer

Other ideas you could try:

  1. gdal_translate with the -srcwin switch
  2. gdalwarp with -cutline and -crop_to_cutline and -wm switches. The last one specifies memory for caching and may get you over the issues you had using clipper in QGIS (as this is essentially the same function)
  3. QGIS raster calculator setting the extent to the area you want (simpler than clipper).
  4. SAGA->Clip grid with polygon - who knows, it might be more memory efficient.
  5. Code a solution using Python and Numpy/SciPy to read just a subset of the raster into memory and save it.

I suspect that doing a very simple crop to extent (either with the raster calculator or -srcwin switch in gdal_translate) will be less memory hungry than cropping with a polygon because you have no geometry checks and conversions. Option 5 should use the least memory as you are only reading in what you need. Have a look at this tutorial if you need a 'how to' (adapt the bit on reading and writing by block).

Related Question