MATLAB: Select a group gpu’s that are discontinuous in gpuDevice

select multiple discontinuous gpu's

On a 4 GPU machine I get the following output (truncated) from gpuDevice().
1. Titan V
2. GTX970
3. Titan V
4. Titan V
I want to use gpu's 1, 3, 4 in a multi-gpu pool. The problem is, the GTX970 is driving the display as a WDDM device and so cannot be set to 'prohibited' in nvidia-smi. Leaving it in the pool hugely reduces the batch size maximum and slows the system to a crawl.
How do I go about creating a pool that includes only gpu's 1,3,4 and not 2?
I've seen this answer (https://www.mathworks.com/matlabcentral/answers/351824-selecting-specific-gpus-for-parpool), but I cannot figure out how to adapt it to my situation.
Thanks!

Best Answer

There are two ways:
The best way is to set environment variable CUDA_VISIBLE_DEVICES to 0,2,3 before you start MATLAB, or as the first thing you do when you start MATLAB:
setenv('CUDA_VISIBLE_DEVICES', '0,2,3');
The second best way is to start a pool of 3 workers before you start training, and set the device manually:
parpool(3);
spmd
    if labindex == 1
        gpuDevice(1);
    else
        gpuDevice(labindex+1);
    end
end
A third way is to use your GTX970, but reduce the amount of work it has to do during training using the 'WorkerLoad' training option:
trainingOptions(..., 'WorkerLoad', [1, 0.1, 1, 1]);
The above will set the second worker to have a tenth of the data of the other workers. Note that this will not necessarily result in faster training than removing your 970 from the pool altogether. You'd have to experiment.
Related Question