MATLAB: Is the communicating job stuck behind a concurrent job in queue in R2017b

clustercommunicatingcomputingconcurrentdistributedjobMATLABmdcsparallelParallel Computing Toolboxparforparpoolpctserverspmdtoolbox

I am running a communicating job on my local machine, but whenever I start it, a "concurrent" job also starts and seems to run forever, causing the other jobs stay queued.

Best Answer

The resolution is to avoid starting a parallel pool when submitting other jobs to the cluster.

The "parpool" object is generally used for interactive "parfor" or "smpd" operations. When the parallel pool is created with "parpool", it creates a concurrent job on the cluster that readies all of the workers to be available for when the user interactively calls "parfor" or "smpd". The concurrent job will continue running, causing other jobs to wait in queue until the parallel pool is shut down.

Related Solutions

MATLAB: How to take advantage of useParallel option for a remote job running on workers in a MDCS cluster

1. In order to use parallel pool with the cluster, please consider using the 'batch' function procedure to submit the jobs into cluster:

https://www.mathworks.com/help/distcomp/batch.html

When using the 'batch' command, it has the option to specify the number of workers to be used in addition to the job itself. Please refer to the example below:

c=parcluster('profile');
j = batch(c,@scriptUseParallel, 'Pool', 2);
wait(j);
diary(j);
Y2=fetchOutputs(j);

2. In the cluster setup, please consider to configure each machine to have the same number of workers as the number of cores.

Therefore each core can be started with a MATLAB process and being utilized by the batch job.

MATLAB: What is the maximum numbers of workers I can use with MATLAB Parallel Computing Toolbox

All releases newer than R2013b do not have a limit on the number of workers (CPU cores) a local machine can utilize through Parallel Computing Toolbox. Prior to R2014a, the maximum number of workers was 12.

Please note that the default number of workers Parallel Computing Toolbox will attempt to use is 12. This number can be set higher in the "local" cluster profile. Additionally, if you edit the profile and leave the number of workers as unset, Parallel Computing Toolbox will attempt to use a number of workers equal to the physical cores of the computer.

If you wish to use multiple computers to perform parallel and distributed job processing, MATLAB Distributed Computing Server will be required to utilize computational resources of other machines.

Best Answer

Related Solutions

MATLAB: How to take advantage of useParallel option for a remote job running on workers in a MDCS cluster

MATLAB: What is the maximum numbers of workers I can use with MATLAB Parallel Computing Toolbox

Related Question