[GIS] Raster IO error (R)

errorrrasterrgdal

I have a raster I'm trying to extract, the projections of my shapefile and raster are the same:

ext <- raster::extract(ras.tiftest2, single, na.rm=TRUE, fun=sum)

I get this error:

Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band = 
object@data@band) : 
Failure during raster IO

I found this post (Why do I get "Error: Failure during raster IO" when extracting values from my raster stack?) which suggested throwing it into QGIS and saving out of QGIS which worked like a charm, except I have about 200 rasters I'd like to do so it isn't feasible to save each one in QGIS.

After saving through QGIS:

ext <- raster::extract(qgisBGD, single, na.rm=TRUE, fun=sum)

> ext
     [,1]
[1,] 149786.1

IS there a way to mimic in R whatever QGIS is doing with the file to fix it?

When I save with QGIS I A. get a tiff file that is over twice as big as the original (76MB vs. the saved QGIS file is 160MB), and B. get an .aux.xml file of the same name as the tif. The content of the xml file contains HISTMIN, HISTMAX, HISTCOUNTS, BUCKET COUNT, STATS MAX, STATS MEAN, STATS MINIMUM, STATS STDDEV. However this doesn't seem to be essential as I tested the dropbox version below without the other .xml file in a new directory.

The raster files that work vs not are in the below dropbox link. Clearly different even based on the preview in dropbox. They are the same exact file, I just saved one through QGIS and didn't the IOERROR one. Not sure how/why QGIS is doubling the file size, I'm more of an R guy. I tested these files off downloads from the dropbox file, they function with the exact errors (or non-errors) above. Loaded the IO error one into QGIS, suddenly twice as big and works like a charm.

Files: https://www.dropbox.com/sh/jr1mybbwntq2zm7/AAAQPyfnK3ud6zVDx0sZ77BDa?dl=0

@Spacedman request

> print(REGRAS)
class       : RasterLayer 
dimensions  : 7271, 5607, 40768497  (nrow, ncol, ncell)
resolution  : 0.0008333, 0.0008333  (x, y)
extent      : 88.00824, 92.68055, 20.57565, 26.63457  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 
names       : WPREGRASTER2 
values      : 0, 934.4766  (min, max)

> print(QGISRASTER)
class       : RasterLayer 
dimensions  : 7271, 5607, 40768497  (nrow, ncol, ncell)
resolution  : 0.0008333, 0.0008333  (x, y)
extent      : 88.00824, 92.68055, 20.57565, 26.63457  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 
names       : QGISTTT 

> summary(REGRAS)
Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band = 
object@data@band) : 
Failure during raster IO

> summary(QGISRASTER)
        QGISTTT
Min.       0.000000
1st Qu.    2.797436
Median     5.271086
3rd Qu.    9.715539
Max.    1247.123779
NA's       0.000000

Best Answer

Don't trust the original raster file.

When I tried converting it on the command line, I got:

$ gdal_translate -co "COMPRESS=NONE" ./IOERROR.tif IOERRORgdal.tif
Input file size is 5607, 7271
0...10...20...30...40...50...60...70...80...90...ERROR 1: TIFFFillTile:Read error at row 4992, col 6656; got 1817 bytes, expected 2564
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: ./IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56
ERROR 1: GetBlockRef failed at X block offset 40, Y block offset 56

So it's not R's fault - R uses gdal, and gdal here didn't like it. So why did QGIS like it? I tried loading it into QGIS, and QGIS didn't like it that much either, throwing some warnings:

Warning: RasterIO error: /home/rowlings/Downloads/Errors/IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56

Something in that file is wrong, and I wouldn't trust the TIFF saved from QGIS either.

Solution

With that massive caveat lodged firmly in your cranium, it is possible to get the GDAL utilities to produce a working version of your file, using gdalwarp on the command line:

$ gdalwarp  IOERROR.tif IOERROR-fix.tif
Creating output file that is 5607P x 7271L.
Processing input file IOERROR.tif.
Using internal nodata values (e.g. -999) for image IOERROR.tif.
Copying nodata values from source IOERROR.tif to destination IOERROR-fix.tif.
0...10...20...30...40...50...60...70...80..ERROR 1: TIFFFillTile:Read error at row 4992, col 6656; got 1817 bytes, expected 2564
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56
ERROR 1: GetBlockRef failed at X block offset 40, Y block offset 56

Which I can read and summary into R:

> IF = raster("./IOERROR-fix.tif")
> summary(IF)
         IOERROR.fix
Min.    3.488778e-02
1st Qu. 3.041900e+00
Median  5.504765e+00
3rd Qu. 9.975844e+00
Max.    1.914248e+03
NA's    0.000000e+00

and it plots too.

The reason the QGIS file is so large is because it isn't using any compression - as does the above command, but you can change this with:

 gdalwarp  -co "COMPRESS=LZW" IOERROR.tif IOERROR-fix-lzw.tif

which returns a smaller file that still works in R.

GDAL utils package

If you struggle with command line stuff, there's an R package that wraps them and might help: https://cran.r-project.org/web/packages/gdalUtils/index.html

Related Question