MATLAB: Is the MAT file much larger than the variable I was trying to save when using the -v7.3 flag

MATLAB

I have a very large ensemble generated using the "fitensemble" command. Whenever I try to save the ensemble to a MAT file using the "save" command with the -v7.3 flag, the file size of the MAT file is significantly larger than the size of the ensemble within MATLAB. Why does this happen?

Best Answer

MATLAB attempts to compress the data whenever data is saved using the "save" command. There is a 2GB file size limitation on the default "save" command, but this limitation can be overcome using the "-v7.3" flag. However, the -v7.3 algorithm uses an HDF5 file format to save the data, which adds extra metadata to the file. The amount of metadata associated with data structures will depend on how complicated the data structure is. For complicated data structures (such as "ensemblefit" data structures), the added metadata can increase the file size by a factor of 4 or higher.

To work around this issue, please save the MAT file without the -v7.3 flag.

If you are dealing with regression trees and the data is larger than 2GB, first try to reduce the size of the data by using the "compact" command before saving the data.

If the "compact" command does not reduce the size below 2GB, please try breaking the data set apart and saving each part to its own MAT file.

Related Solutions

MATLAB: How to save a variable as -v7 by default and switch to -v7.3 only if the variable is too large using SAVE on 64-bit MATLAB 7.12 (R2011a)

You can test for the warning that occurs when you attempt to save a very large variable into a -v7 MAT-file. Here is an example:

X = zeros(30000, 10000);    % create a large matrix that is too large to be saved as -v7
lastwarn('')                        % This will clear any last warnings so there is no  confusion
save('test.mat', '-v7', 'X');    % attempt to save the file in -v7 format
[msg,id]=lastwarn('')     % This will save the last warning
if strcmp(id,'MATLAB:save:sizeTooBigForMATFile')    % compare the id of the warning with the one we are looking for
   save('test.mat','-v7.3','X');    % save the variable in a -v7.3 MAT-file 
end

MATLAB: Saving large files using uisave

use uiputfile instead. Credit to Jan.

[filename, filepath] = uiputfile('*.mat', 'Save the project file:');
FileName = fullfile(filepath, filename);
save(FileName, 'yourVariable', '-v7.3');

Best Answer

Related Solutions

MATLAB: How to save a variable as -v7 by default and switch to -v7.3 only if the variable is too large using SAVE on 64-bit MATLAB 7.12 (R2011a)

MATLAB: Saving large files using uisave

Related Question