MATLAB: How to ask for a pool size when using parpool with threads

parallel.threadpoolparpool

Dear all,

using R2020b, looks like the flag "threads" when using a parpool excludes providing a pool size… is there a reason for that? How can provide an explicite number of workers?

thanks,

Daniel

Best Answer

There isn't from within parpool. When you create a Threads pool, the size of the pool is identical to maxNumCompThreads, which by default is the number of physical cores on the machine.

One workaround is as follows. Assume you have 4 cores and you want a thread pool of 2.

old_thread_count = maxNumCompThreads(2);
p = parpool("threads");

Keep in mind, adjusting maxNumCompThreads could have an impact on other MATLAB functions (though perhaps not since you're now running a pool), so I'd suggest setting it back once you're done with the pool.

p.delete
maxNumCompThreads(old_thread_count);

Is there a particular reason you want to adjust the size?

Related Solutions

MATLAB: Is the communicating job stuck behind a concurrent job in queue in R2017b

The resolution is to avoid starting a parallel pool when submitting other jobs to the cluster.

The "parpool" object is generally used for interactive "parfor" or "smpd" operations. When the parallel pool is created with "parpool", it creates a concurrent job on the cluster that readies all of the workers to be available for when the user interactively calls "parfor" or "smpd". The concurrent job will continue running, causing other jobs to wait in queue until the parallel pool is shut down.

MATLAB: How to create pool of workers from a list of hostnames

This isn't feasible at this time; however, you might consider wrapping your code with batch. For example, let's assume your code looks something like this

function my_parallel_code
% ...
nworkers = ... ;
% Check if a parallel pool is running.  If so, use it.  If not, initiate a parallel pool with 'nworkers'
p = gcp('nocreate');
if isempty(p)
    % Pool has not been started, spawn pool across multiple nodes (kicks of new job)
    % If the pool has already been started, it would have been done in "submit_job" with call to batch()
    parpool('hpc',nworkers);    
end
parfor idx = 1:N
    % parallel code
end

Now, write a wrapper function

function submit_job
c = parcluster('hpc');
j = c.batch(@my_parallel_code,0,{},'Pool',nworkers);

Submit your MATLAB job (e.g. using Slurm). Call submit_job instead of my_parallel_code directly

#SBATCH -n 1
module load matlab
matlab -batch submit_job

Best Answer

Related Solutions

MATLAB: Is the communicating job stuck behind a concurrent job in queue in R2017b

MATLAB: How to create pool of workers from a list of hostnames

Related Question