data-framerraster
I have rasters over which I am performing mathematical operations. For example, where I need to calculate flow distance or distance between two grid points.
What format is generally considered faster for such operations?
Should I work directly on the raster format, instead?
I also converted the data to matrix and dataframe
as.matrix and as.data.frame
The raster has entirely numeric values.
Not sure which is faster. I am new to raster package in R, and I have mostly only worked in dataframes.
You can use the is.na function to select for values which are NA. Eg :
> t<-c(1:10,NA,1:5)
> t
[1] 1 2 3 4 5 6 7 8 9 10 NA 1 2 3 4 5
> mean(t)
[1] NA
> mean(t, na.rm=T)
[1] 4.666667
> t[is.na(t) ]<-mean(t, na.rm=T)
> t
[1] 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000
[8] 8.000000 9.000000 10.000000 4.714286 1.000000 2.000000 3.000000
[15] 4.000000 5.000000
So in your case:
dat$x[dat$TRIP_ID=="e.g.,100022478"&is.na(dat$x)] <-mean(dat$x[dat$TRIP_ID=="e.g.,100022478"], na.rm=T)
You could use a loop to iterate over your TRIP_ID values:
for(trip in unique(dat$TRIP_ID)){
dat$x[dat$TRIP_ID==trip&is.na(dat$x)] <-mean(dat$x[dat$TRIP_ID==trip], na.rm=T)
}
Your attempt is designed to fail. If you look at the image, you see the data arranged as a circle, with black triangles in the corners of the square, where the satellite view goes right into orbit. In your test data, you see only NODATA -32768 for those parts of the image.
The extent is between +/-75 and +/- 78, but these values are only reached in the middle of the egdes. So you can not reproject those black triangles to Earth surface coordinates.
UPDATE
The Metadata of the HDF file reveals some mysteries:
Altitude=42164
Ancillary_Files=MSG+0000.3km.lat
So the satellite height is the same as mentioned in http://geotiff.maptools.org/proj_list/geos.html, and I assume they took the same ellipsoid (not exactly WGS84).
With the help of http://www.cgms-info.org/documents/pdf_cgms_03.pdf and http://publications.jrc.ec.europa.eu/repository/bitstream/JRC52438/combal_noel_msg_final.pdf, I found that the size of 3712px is not the real extent covered by the data. The size provides a scanning angle of the satellite of about +/-8.915 degree, but the angle that was used is smaller.
Proj.4 calculates the extent by multiplying the satellite's scanning angle by the height above ground (see http://proj4.org/projections/geos.html). So with a bit of try, an extent of +/- 5568000m (3712*3000m/2 or 8.915*pi()/180*35785831m) fits to the 3712px used in the 3-km-resolution hdf.
So the correct translation commands are:
gdal_translate -a_srs "+proj=geos +h=35785831 +a=6378169 +b=6356583.8 +no_defs" -a_ullr -5568000 5568000 5568000 -5568000 HDF5:"SEV_AERUS-AEROSOL-D3_2006-01-01_V1-03.h5"://ANGS_06_16 temp.tif
gdalwarp -t_srs EPSG:4326 -wo SOURCE_EXTRA=100 temp.tif output.tif
And the result looks good:
As an alternative, you can take the lat and lon subdatasets from http://www.icare.univ-lille1.fr/archive/?dir=GEO/STATIC/ in file MSG+0000.3km.hdf
Best Answer
"Faster" is relative. Faster to upload? Faster to access and run calculations? Because so many variables exist between systems and data, the best way to get a guaranteed answer for your processes and data is to run it using each of the three formats.
A similar Stack Exchange Answer can be found here: Data frame or matrix?