I have access to a cluster of nodes. Each node has 64 physical cores. My goal is to run a job in parallel across 2 nodes (128 cores). To do this I create a remote cluster profile that utilises direct slurm integration. I first submit a job script to slurm:
#!/bin/bash
#SBATCH -A account
#SBATCH -p partition
#SBATCH -t 12:00:00
#SBATCH -J job
module purge
module load rhel7/default-peta4 matlab
matlab -nosplash -nodisplay -nodesktop -r "EmitCue(${1},${2},'${3}'); quit" # > matoutfile
which calls a matlab script (EmitCue.m). This matlab script requests the resources (a pool of workers):
clusterKNL = parallel.cluster.Slurm;
clusterKNL.NumWorkers = 128;
clusterKNL.NumThreads = 1;
clusterKNL.JobStorageLocation = JobStorageLocation;
clusterKNL.SubmitArguments = '–time=12:00:00 -A account -p partition -N 2 –ntasks-per-node=64 –cpus-per-task=1 -J job'
parpool(clusterKNL,128)
and then goes on to performs the desired computations.
The call to parpool in EmitCue.m itself submits a job to Slurm, which results in the required resources being allocated (in my case 2 nodes with 64 cores each, giving 128 nodes). Note, however, that there are now 2 jobs running simultaneously – the first job (job 1) that was used to open the matlab script EmitCue.m, and the second job (job 2) that was used inside EmitCue.m to request acess to 2 nodes. Now here is my problem. It is only possible for me to request entire nodes on the cluster; I can't request a subset of cores on a node. Hence, when I run the above code I end up with an entire node (I think this is called the 'client') of 64 cores being used just to request 2 more nodes and delegate the job across the 128 workers on those nodes (job 2). None of the cores on the client contribute to the worker pool. This is hugely wasteful. I have limited computing credits that are measured in terms of hours spent on a node so I want to avoid this.
Thanks in advance
Best Answer