MATLAB: Find optmal number of parallel workers (depending on machine memory)

Question

Dear all, I am running some Image analysis in a parfor loop. is there any way to know before running, what the optimal number of workers is (run time vs. out of memory error)? or even better, adjust the number of workers dynamically?

For my task, memory is the bottleneck not the CPUs. If i have too many workers, I run into the out of memory error.

Based on what I see in the Task manager / performance tab, for 3 workers my memory oscillates heavily between 40 and 90% at the beginning of the job. minutes later, I see the used memory getting more steady at about 50-60% memory. SO I wonder if I could not add another worker or two at that stage

Thanks for any advice!

Answer 1

This is a tricky problem to handle automatically - I think your best bet is simply to time and observe the behaviour on your system and then try to choose the number of workers that gives the best performance.