QGIS Raster – Merging Many GeoTIFF Files Gives Huge Space Issues During Processing

qgisrraster

While I was merging eight GeoTIFF files with each of files has 800MB-1GB of size, on the log of Merge tools in QGIS give me those error, which requires more than 3000GBs of free space. How it can be that? How to merge it effectively?

I already try in RStudio but the process is too long and eat my disk C for the temporary files.

EDITED(1)
My current imageries are like this, with each tile has size of 0.25×0.25 degrees. Are the blank area still counted as new raster and calculated in the size too?

EDITED(2)
I already create using virtual raster, but no values in each band and the raster didnt show up anything. But on the histogram there is a spectral curve for each band

Best Answer

Your inputs are compressed and you are creating a non compressed output? Try adding a compression:

Related Solutions

[GIS] Plot in rstudio a NetCDF File – latbox by lonbox

1) It will be vastly easier if this works:

library(raster)
olr <- raster("t2m_50N70N.nc", varname = "var167")
tm2 <- brick("t2m_50N70N.nc", varname = "var179")

plot(olr[[1]])) ## first of nlayers(olr) bands

If this kind of mapping of the data is what you are after, I'd re-focus and work on learning raster. You'll also need ncdf4 package installed.

2) ncdf is no longer supported, you can now use ncdf4 as a safe default (raster does this for you).

3) To use NetCDF in the raw like this, you need to learn the start and count arguments to the "get var" function, in ncdf4 this is called ncvar_get. But, this is really painful since you have to juggle your time-subset across all the variables and work out from other variables what the right subset indexes for the three dimensions must be.

In a limited context, raster does everything for you - it's not general enough for all of NetCDF, but it's worth trying because if your files conform to its assumptions you get a lot for free.

[GIS] How to efficiently extract time series from a very large multi-band GeoTIFF in R

I've tried out a bit and found that:

RasterBricks, as mentioned by RobertH's answer, do work and are more user-friendly and easy to use;
Rgdal methods like readGDAL also work, but with more parameters it's a little bit less user-friendly;

So which option should one use?

According to my tests (on my 420GB GeoTiff with dimensions of 18660x21592 and 374 bands) Rgdal is faster. Maybe due to less overhead of a higher level library such as the Raster-package.

Here are my results, using system.time and replicate:

With brick:

> modis_ndvi_ts_brick <- brick("../data/pa_br_mod13q1_ndvi_250_2000_2016.tif")
> system.time(replicate(100, modis_ndvi_ts_brick[7000,7000]))
   user  system elapsed 
 94.024   5.468  99.562

While with rgdal:

> system.time(replicate(100, readGDAL("../data/pa_br_mod13q1_ndvi_250_2000_2016.tif", offset=c(7000,7000), region.dim=c(1,1))))
#some printed output from readGDAL here
   user  system elapsed 
 88.752   5.400  94.213

So as you can see Rgdal is slightly faster. For those whose this does not make a difference, I recommend using RasterBrick, it's simpler. But for those whose, like me, are struggling to create a high performance code in which every millisecond matters: Use Rgdal

Note: I'm sorry for the not exactly reproducible data, but one could try it out with other data and post here if their results differ somehow

Best Answer

Related Solutions

[GIS] Plot in rstudio a NetCDF File – latbox by lonbox

[GIS] How to efficiently extract time series from a very large multi-band GeoTIFF in R

Related Question