MATLAB: Parfor: exceed memory

parfor

Hi all,
It seems that my problem is an old one and many people have faced it, but I still have not fixed problem yet so hope to get help here.
My problem is that: I ran a program (complex, including many functions) on the lab cluster and used 25gb of memory. If I use "for" loop I did not see any problem, but when I used "parfor" I soon received a message says that the required memory is exceeded than 25gb I arranged.
I made a search on this problem and know some thing that it is likely due to data cloning problem but I did not see a good suggestion how to fix it.
Can anyone suggest me a good solution?
Thank you in advance.

Best Answer

parfor statically examines all of the variables used in the body of the parfor. Any of the defined variables that are not indexed by the loop variable will generally be completely copied into the process that does the work, even if only a portion of the variable is accessed. If the number of workers times the amount of data read in the loop is greater than available memory, you have an inherent problem. If not all of the data of a variable is needed, sometimes using distributed arrays or co-distributed or sliced helps; other times you should extract subsets of the data before the parfor.
Sometimes the question is one of efficiency rather than amount of storage. In that case see http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper
http://www.mathworks.com/matlabcentral/answers/121831-how-to-make-data-persist-on-workers-between-calls-to-parfor#comment_202694
http://www.mathworks.com/help/distcomp/sliced-variables.html
I know that a recent MATLAB release introduced a replacement for Worker Object Wrapper, but unfortunately I do not remember the name of the feature at the moment.
Related Question