MATLAB: Dealing with Large training datasets saved in a number of .mat files

datasetlstmneural networksignal processing

Hello all,

I have run into a problem where I need to train a LSTM signal classifier with huge amount of data.

Each 1D signal is around 100k samples, every 48 signals are saved in a .mat file. The total number of .mat files is around 2000.

The labels are similarly saved in corresponding .mat files in a different folder.

I would like to know if there's a way to train the network without the necessity of loading the whole thing into memory. (with 64GB ram I can only load ~1300files at once)

Your help will be very much appreciated.

Best Answer

You can use fileDatastore for this purpose.

trainData = fileDatastore('/path/to/data', 'ReadFcn', @load, 'FileExtensions', '.mat');

You can either use "load" or your own custom function defining how to load the data.

You can also refer to this link for more information on training LSTM while loading data using fileDatastore.

Related Solutions

MATLAB: Training nework: Combined network is not partitionable but each datastore is partitionable

Hello Shen Zhao,

This is a known limitation in the FileDatastore + combine workflow.

Even though FileDatastore by itself is partitionable, combining two FileDatastores is currently not defined as partitionable. We are working on ways to remove this limitation in future releases.

One possible workaround for this is to directly import each pair of MAT files in a single FileDatastore. You would need to have some kind of mapping between the filenames used in the Train_Noisy/ and Train_Noise/ datasets for this to work:

addpath('Train_Noisy')
addpath('Train_Noise')
Train_Data = fileDatastore(fullfile('Train_Noisy'), 'ReadFcn', @Load_All, 'FileExtensions', '.mat');
function Data = Load_All(noisy_filename)
    File = load(noisy_filename);
    Noisy = File.Noisy;
    noise_filename = Make_Noise_Filename(noisy_filename);
    File = load(noise_filename);
    Noise = File.Noise;
    Data = [Noisy Noise];
end

You'd also need to define another function here called Make_Noise_Filename that converts filenames from the first dataset to the second dataset. This sidesteps the limitation in FileDatastore + combine for now.

I hope this helps!

Rylan

MATLAB: How to load .mat files from a Hadoop filesystem

MATLAB offers the "datastore" object to access remote data on systems like HDFS:

https://www.mathworks.com/help/matlab/import_export/read-remote-data.html#bvn_hcu-3

You can use the "fileDatastore" function to load MAT-files from HDFS, as shown:

% Create a fileDatastore for the MAT-file on HDFS.
ds = fileDatastore('hdfs://myserver/path_to_file.mat', 'ReadFcn', @load);
% Read the MAT-file into the workspace.
s = read(ds);

Additional information on "fileDatastore" and its arguments is available in the documentation:

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html

Best Answer

Related Solutions

MATLAB: Training nework: Combined network is not partitionable but each datastore is partitionable

MATLAB: How to load .mat files from a Hadoop filesystem

Related Question