MATLAB: NetCDF Read Slow Down – Sometimes

netcdf

I am reading data from very large NetCDF files (>25GB/file). The data are stored in separate 2D variables within the NetCDF file and depending on the syntax I use for netcdf.getVar I get dramatically different performance.

If I retrieve 'X' data values from the NetCDF file (different parts of the code require different subsets of the data):

(1) Always Fast: netcdf.getVar(ncid,varid,[0 0],[X 1]) (2) Initially Slow: netcdf.getVar(ncid,varid,[0 0],[1 X]) …however subsequent reads from the same variable seem much faster.

Any thoughts as to what might be going on…is there any way to optimize things such that (1) and (2) are equally fast???

Best Answer

If your variable's 2nd dimension crosses a lot of chunk boundaries, that might explain why the initial read might be slow.

Related Solutions

MATLAB: Error netcdf.getVar if reading several netcdf files in a row

You need not to use such complex functions ....read about ncread, ncinfo. YOu can just use ncread to read the variable you want.

MATLAB: Read specific 2D matrix from 3D matrix variable from netCDF….

To read a matrix of mXn with third dimension being at r \in [1,p] do this:

data = netcdf.getVar(ncid,varid,[1,1,r],[m n 1]);

ALternatively you can use

data = ncread(netcdfFilename,varName,[1,1,r],[m n 1]);

I suggest the second method, i.e. using ncread, unless if you have some special reason to use the low-level netcdf API included in MATLAB. note that ncread does not require ncopen or ncid, or varid. It works just by their name.

Best Answer

Related Solutions

MATLAB: Error netcdf.getVar if reading several netcdf files in a row

MATLAB: Read specific 2D matrix from 3D matrix variable from netCDF….

Related Question