MATLAB: Advantages of parpool vs. job/tasks vs. multiple batches

Question

I have an "embarrassingly" parallel Matlab problem I am looking to "parallelize" and I was just thinking about the various strategies. I've tried all three of the approaches mentioned in the question title and would just be curious to get some more experienced Matlab users thoughts. Essentially I am just running the same function with different data, and collecting the results.

In my experiments I've noticed that creating multiple batches incurs a significant startup time vs. creating a job with multiple tasks. The only reason I was even considering a multiple batches approach is I was thinking there might be a certain robustness in this approach, in the sense that if one batch fails due to bad data (or a node going offline, etc, etc…), you would still have the results from the other batches. One could then resubmit the batches that failed. Can a job/tasks approach be made equally robust? What happens if a task irrecoverably fails or hangs? Is there some way to recover the results from the other tasks?

As for parpool, is there an advantage to this approach that I'm missing beyond the automatic slicing of variables? Variable slicing is something I could accomplish manually using jobs/tasks or multiple batches.

Regardless of which approach is taken the job will be accomplished via a submitted batch (or batches), as it will likely take quite some time to run and being able to exit out of Matlab on the submitter machine will be nice.

Answer 1

If you want to be able to quit the client machine while the process is running, then either batch or createJob & createTask is the way to go.

As you observe, there is some additional overhead when creating multiple batch jobs compared to a single createJob invocation and then multiple (or vectorised) createTask invocations. (This can be to do with the analysis of the code files required to run the job etc.)

The simplest (from a coding perspective) option is to prototype your code using an interactive parpool with a parfor loop, and then offload using batch specifying the 'Pool' parameter to indicate how many workers to use.

Using multiple independent tasks is more invasive compared to the batch + 'Pool' approach, but it does give you a degree of resilience against individual worker failures.