MATLAB: Reading variables from a big matlab Table

data importdatabaseMATLAB

Hello There we have about 45 big matlab Tables (average size 12 GByte each). Please note that thee are .mat files. We will be running a script with a for loop that will read only a couple of rows from each of theses tables and do some easy computation on the obtained data. Our problem is that on matlab r2017a takes about 25 minutes average to load each table to the matlab environment, a lot of time considering that we are just interested to extract the data of a couple of rows of those tables. As far as we understand the command matfile is not working with tables (is this correct?) to be able to extract variables from the file without loading. What are our options here? If any! Thank you in advance

Best Answer

Yes, matfile won't help since it can't partially load a table.

Unfortunately, as you are now there is no workaround I'm aware of. You could always contact Mathworks support directly to see if they have a solution.

If the data is to be accessed more than once, then what I'd do is convert all these tables to Tall Tables, then save them with the write function. As far as I know there is no way to convert a regular mat file to one compatible with tall arrays so You'd have to take the performance hit once when you load the tables for the first time as regular table, but subsequent loads would allow you efficiently load only the data your require as you'd be using tall arrays.

Related Solutions

MATLAB: In Case Western Reserve University Bearing Vibration signal Data all files are in the form MAT file. how i can use these MAT files or open in Matlab

See https://www.mathworks.com/matlabcentral/fileexchange/51165-matlab-file-association-fix if you installed MATLAB on Windows but .mat files are opening in some other application/

Other than that, look at https://www.mathworks.com/help/matlab/ref/load.html and https://www.mathworks.com/help/matlab/data-import-and-export.html

MATLAB: How to save a tall array / table to a text or csv file

I think the least inefficient method is probably to combine use of tall/write with writetable. Calling gather repeatedly is going to be inefficient - the approach below takes 2 passes over the data (one of which is in the optimised .mat form, so should be quicker). Here's the sort of thing I mean.

%%Step 1 - create a tall table
varnames = {'ArrDelay', 'DepDelay', 'Origin', 'Dest'};
ds1 = datastore('airlinesmall.csv', 'TreatAsMissing', 'NA', ...
                'SelectedVariableNames', varnames);
tt = tall(ds1);
%%Step 2 - operate on tall table
tt.TotalDelay = tt.ArrDelay + tt.DepDelay;
%%Step 3 - use tall/write to emit .mat files
writeDir = tempname
mkdir(writeDir);
write(writeDir, tt);
%%Step 4 - iteratively convert the tall/write output to CSV
ds = datastore(writeDir);
csvDir = tempname
mkdir(csvDir);
idx = 0;
while hasdata(ds)
    idx = 1 + idx;
    fname = fullfile(csvDir, sprintf('out_%06d.csv', idx));
    writetable(read(ds), fname);
end

A refinement of this would be to partition the datastore and operate on it in parallel, using the techniques described here in the documentation.

Best Answer

Related Solutions

MATLAB: In Case Western Reserve University Bearing Vibration signal Data all files are in the form MAT file. how i can use these MAT files or open in Matlab

MATLAB: How to save a tall array / table to a text or csv file

Related Question