MATLAB: Parallelize calculations on a big cell array without making N input copies

big cell arraycalculationMATLABmemory efficiencyparallelparforshared memory

Hello,
The existing answers on matlab answers, also due to their age, have not made it clear to me if it is possible to parallelize CPU-intensive calculations done with either cellfun or a for loop over a large (15.000.000×1) cell array with each cell containing a 24×24 matrix, and then writing the calculation results for each into a 15.000.000×1 vector.
Using just parfor is no use since my PC runs out of memory (I have enough memory for 1 copy of the cell array but not 6 copies for the 6 workers). Is there a way to perhaps copy it only once, with each of the 6 workers receiving only a 6th of the total array as copy?
An older comment said there was a userwritten solution with shared memory, but that this would not work anymore with newer Matlab versions (I use 2016a).
Thank you for your help!

Best Answer

If each calculation is independent (as in, you don't need the entire 15000000x1 cell to make 1 calculation), then you could rewrite the parfor loop to work on a sliced variable. Workers will only receive a "slice" of the large 15000000x1 cell. This prevents passing the large cell as a broadcast variable, which will use too much memory.
The parfor loop should look something like this:
LargeCell = repmat({zeros(24)}, 100, 1); %Represent your 15000000x1 cell
Results = zeros(size(LargeCell)); %Store results here
parfor k = 1:length(LargeCell)
   Results(k) = complex_function( LargeCell{k} ); %Do your CPU-intensive calculation here
end
Related Question