MATLAB: How to take advantage of useParallel option for a remote job running on workers in a MDCS cluster

batchclusterMATLAB Parallel Servermdcsuseparallel

When using a function with the 'useParallel' option (e.g. fitcecoc), in local, it will start a parallel pool.
However, when doing this with the following script to submit this remotely on a cluster with MDCS with the following workflow:
c=parcluster('profile');
job=createJob(c);
createTask(job,@scriptUseParallel,0);
submit(job)
wait(job)
data=fetchOutputs(job);
this can only get it to run on 1 core in the remote machine. How to get it to utilize all cores?
 

Best Answer

1. In order to use parallel pool with the cluster, please consider using the 'batch' function procedure to submit the jobs into cluster:
https://www.mathworks.com/help/distcomp/batch.html
When using the 'batch' command, it has the option to specify the number of workers to be used in addition to the job itself. Please refer to the example below:
c=parcluster('profile');
j = batch(c,@scriptUseParallel, 'Pool', 2);
wait(j);
diary(j);
Y2=fetchOutputs(j);
2. In the cluster setup, please consider to configure each machine to have the same number of workers as the number of cores.
Therefore each core can be started with a MATLAB process and being utilized by the batch job.
Related Question