MATLAB: Using CUDA mex files interoperably with gpuArray

cuda mexgpugpuarrayParallel Computing Toolbox

I have a 3rd party black box CUDA mex file. Every time I use it, it puts my GPU in a state such that gpuArray cannot subsequently use it. This is in R2020a, but I've observed similar problems in previous versions.

>> a=gpuArray(1)
Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_CONTEXT_IS_DESTROYED

The only remedy once this occurs is to restart Matlab. Is there any way gpuArray and my cuda mex can share use of the same GPU without nuclear devastation?

Best Answer

A third party CUDA MEX file could be built with a different version of the CUDA toolkit, and thus a different version of the CUDA runtime. You will not be able to safely pass gpuArray objects into and out of the function.

To avoid this, obtain the source code and compile the function yourself using MEXCUDA.

That said, there's nothing to guarantee it won't work...but in this case it looks like your MEX function is resetting the device itself and thus destroying the CUDA context (for instance, it could be calling the CUDA C runtime function cudaDeviceReset). You won't be able to use it safely with gpuArray variables in your MATLAB workspace. Again, if you had access to the source code, you could modify this behaviour and even have MATLAB successfully respond to the reset event.

Related Solutions

MATLAB: Detect when no GPU is selected

You can query device properties without selecting it using parallel.gpu.GPUDevice.getDevice. So, for instance, you can find the selected device without selecting it like this:

   selectedDevice = [];  % Nothing selected
   for i = 1:gpuDeviceCount
       if parallel.gpu.GPUDevice.getDevice(i).DeviceSelected
           selectedDevice = i;
           break;
       end
   end

For your purposes, you can use this handy internal:

   parallel.internal.gpu.isAnyDeviceSelected

Since you're writing your own CUDA code (using MEX, presumably), you can also just query device properties using the CUDA runtime API (e.g. cudaGetDeviceProperties), or the driver API (e.g. cuDeviceGetProperties).

In answer to your 'elaboration' question, I'd need to know what you're actually doing. If you are resetting the device (using, for instance, cudaDeviceReset) then there's nothing you can do to save your gpuArrays - the device memory has been cleared and every pointer into GPU memory invalidated. Since your GPU is crashing when you subsequently attempt to read those gpuArrays it seems like this is what you're doing. Your only hope would be to copy everything to the CPU and back after the reset. For instance, you could use the mex API to call out to a MATLAB function which walks through the workspace, finds all gpuArrays, copies them to CPU arrays, and then another to copy back afterwards. That's a bit crazy because it could be a huge task to check every workspace variable and its contents.

Perhaps all you really want to do is ensure your card doesn't crash. In that case MATLAB needs to know you reset the device, so call reset(gpuDevice) or gpuDevice([]) in MATLAB instead of doing it your own code.

MATLAB: Checking if persistent variable exists on GPU: check if ‘data no longer exists on the GPU’ for a variable.

You need existsOnGPU, like so:

x = gpuArray(1);
assert(existsOnGPU(x));
reset(gpuDevice);
assert(~existsOnGPU(x));

Best Answer

Related Solutions

MATLAB: Detect when no GPU is selected

MATLAB: Checking if persistent variable exists on GPU: check if ‘data no longer exists on the GPU’ for a variable.

Related Question