I have deep learning model trained and now want to use it for predict on a batch of my images.
I have a 12GB GPU and 16 physical cores cpu and want to use parfor for predict function. The psedo code looks like below
parfor i = 1:100
image = read('xxx');
mask = predict(net, image);
end
However, I got gpu out of memory issues when use parfor. Just use for works fine. Also, forecefully turn off all gpu by disable them in device manager works as well to run on 12 core CPU with parfor, which is even faster to use for with GPU (I think because copying single image between devices is even more expensive).
Anyone has idea how to use both parfor and gpu to for deep learning inference? I also try to limit the number of the worker in parfor to 2 and even 1. Even 1 worker gives the same gpu out of memeory problem. I thus assume the parfor doesn't realse the memory or resource timely after each prediction. I even added the gpuDevice command after predict trying to forcefully clear out memory, but the error stays.
Best Answer