MATLAB: Can chunk caching be used with the high-level NetCDF functions

cachechunkfilehigh-levellow-levelMATLABncreadnetcdfnetcdf.setchunkcacheperformancesetchunkcache

I am using the high-level NetCDF interface to read a NetCDF file. My code might have to access the same data multiple times. In order to increase performance, I want to use chunk caching. The low-level interface includes the function "netcdf.setChunkCache" that is used to configure the default chunk cache size. If I use this low-level function to configure the chunk cache, can I expect a performance increase when I call functions in the high-level interface like "ncread"?

Best Answer

When the default chunk cache is configured with the low-level interface, those settings will be respected by the high-level functions such as "ncread". However, if you intend on using multiple calls to "ncread" to access the same data, the performance improvement from caching may not be fully realized. This is because the "ncread" function always closes the file before it returns. This action of closing the file will clear the chunk cache, and subsequent calls to "ncread" will start with an empty cache.

To fully realize the performance improvements offered by the chunk cache, you may wish to exclusively use the low-level interface. This interface offers much more control over when the file is opened and closed, and therefore enables greater persistence of the chunk cache. For more information regarding the low-level NetCDF interface, please consult the functions described under the section "NetCDF Library Package" at the documentation link below.

https://www.mathworks.com/help/matlab/referencelist.html?type=function&category=network-common-data-form

Related Solutions

MATLAB: What encoding is used by the file I/O functions such as ‘fileread’

Starting in MATLAB R2020a, the "fileread" function along with many other high-level and low-level IO functions use automatic character set detection to determine the encoding of the file. The encoding is determined automatically. Note that this encoding detection adds some overhead to the "fileread" operation.

In cases where additional overhead is undesirable, or where the encoding is already known, an alternative workflow is to use "fopen" to specify the encoding of the file and use other low-level IO functions to read the data with this encoding. For example, the following will open a file for reading with UTF-8 encoding:

>> fid = fopen('test.txt', 'r', 'n', 'UTF-8'); % Set encoding to be UTF-8

More information on automatic file encoding can be found in the File I/O documentation:

https://www.mathworks.com/help/releases/R2020a/matlab/import_export/import-text-data-files-with-low-level-io.html#br4sz4c

MATLAB: CoolProp Low-Level interface

Why not ask this question over at CoolProp (issues or mailing list)? Much better chance of a response. That said, there are now examples online in the development docs: http://www.coolprop.org/dev/coolprop/wrappers/MATLAB/index.html#example-code

Best Answer

Related Solutions

MATLAB: What encoding is used by the file I/O functions such as ‘fileread’

MATLAB: CoolProp Low-Level interface

Related Question