MATLAB: How to train neural network with more data than can fit the memory

big dataclassificationneural networkrnn

Trying to train a classifier with a recurrent layer using a lot of data. As a result, all data can not fit into the memory. It provides me with the following error:
Error using zeros
Requested 1x2114046976 (15.8GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long
time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in nnMex.perfsGrad (line 3)
TEMP = zeros(1,ceil(hints.tempSizeBG/8)*8);
Error in nnCalcLib/perfsGrad (line 294)
                lib.calcMode.perfsGrad(calcNet,lib.calcData,lib.calcHints);
Error in trainscg>initializeTraining (line 153)
[worker.perf,worker.vperf,worker.tperf,worker.gWB,worker.gradient] = calcLib.perfsGrad(calcNet);
Error in nnet.train.trainNetwork>trainNetworkInMainThread (line 28)
worker = localFcns.initializeTraining(archNet,calcLib,calcNet,tr);
Error in nnet.train.trainNetwork (line 16)
    [archNet,tr] = trainNetworkInMainThread(archNet,rawData,calcLib,calcNet,tr,feedback,localFcns);
Error in trainscg>train_network (line 147)
[archNet,tr] = nnet.train.trainNetwork(archNet,rawData,calcLib,calcNet,tr,localfunctions);
Error in trainscg (line 59)
            [out1,out2] = train_network(varargin{2:end});
Error in network/train (line 369)
    [net,tr] = feval(trainFcn,'apply',net,data,calcLib,calcNet,tr);
It should be noted that currently my training input is 11×52266 and the network has ~3k weight elements due to the recurrent layer. I would like, however, to provide 15 times as much data for training.
How can I cope? Are there any techniques to map the local variable it's trying to initialize on my SSD instead of memory?
There is the "reduction" option for training, but it does not seem to do any difference on this matter. The same error occurs regardless.

Best Answer

You do not need to train on all of that data.
You have 11 inputs dimensions. They are more than adequately sampled if you randomly choose between 30*11 = 330 to 100*11 = 1,100 examples to train a net.
So, divide the data into smaller subsets, train multiple nets and test on all of the data.
Hope this helps
Greg
Related Question