Im running Matlab 2017b with update 9 and I wanted to train a convolutional neural net using trainNetwork(…) together with a Nvidia Turing GPU (RTX 2070, Driver version 416.34). However, there is an error message showing up after quite a bit of delay:
_
Training on single GPU.Initializing image normalization.|=======================================================================================================================|| Epoch | Iteration | Time Elapsed | Mini-batch | Validation | Mini-batch | Validation | Base Learning|| | | (seconds) | Loss | Loss | RMSE | RMSE | Rate ||=======================================================================================================================|Error using trainNetwork (line 140)Unexpected error calling cuDNN: *CUDNN_STATUS_EXECUTION_FAILED.*Error in NetTrainTest (line 67)net = trainNetwork(inputConv,outputConv,layers,options);Caused by: Error using nnet.internal.cnngpu.convolveForward2D Unexpected error calling cuDNN: CUDNN_STATUS_EXECUTION_FAILED._
When I train the network using the CPU, everything is fine. I also do not get any errors when I use normal calculus with GPU arrays, everything works fine.
The output of gpuDevice is as follows:
Name: 'GeForce RTX 2070' Index: 1 ComputeCapability: '7.5' SupportsDouble: 1 DriverVersion: 10 ToolkitVersion: 8 MaxThreadsPerBlock: 1024 MaxShmemPerBlock: 49152 MaxThreadBlockSize: [1024 1024 64] MaxGridSize: [2.1475e+09 65535 65535] SIMDWidth: 32 TotalMemory: 8.5899e+09 AvailableMemory: 7.6195e+09 MultiprocessorCount: 36 ClockRateKHz: 1620000 ComputeMode: 'Default' GPUOverlapsTransfers: 1 KernelExecutionTimeout: 1 CanMapHostMemory: 1 DeviceSupported: 1 DeviceSelected: 1
Any ideas on what causes this issue? Thank you for your reply.
Best Answer