I am using 10 workers to submit a job. 1 worker is assigned to the GPU while the rest is assigned to the CPU cores. createTask() calls a function that checks for the current worker. Based on the current worker assigned to the task, that function will branch into the GPU or the CPU code.
The problem is that even when it branched into the CPU code, GPU memory is still pre-allocated. I found that initializing the GPU code with gpuArray.zeros() causes MATLAB to pre-allocate GPU memory even if the program never executes that line of code. This is a major issue because while the one GPU worker is running, the other 9 workers are grabbing GPU memory causing it to run out of memrory.
A quick fix was to initialize gpu arrays with gpuArray(zeros()) but this significantly slows down the GPU worker since data has to be pushed to the GPU each time. I have attached some simple code to reproduce the issue. I used nvidia-smi to look for GPU memory usage.
Does anyone know of a workaround the issue that I just described. Thank you. Geremy
Best Answer