MATLAB: Semanticseg – out of memory on device

deep learning tutorialgpuMATLABnvidia

Dear colleagues,
I am experiencing Deep Learning capability of Matlab R2018a. I have no experience with deep learning before.
I followed the tutorial "Semantic Segmentation Using Deep Learning" (https://www.mathworks.com/help/vision/examples/semantic-segmentation-using-deep-learning.html). Every thing turned out OK, until the command:
    pxdsResults = semanticseg(imdsTest,net,'WriteLocation',tempdir,'Verbose',true);
The command seemed to work, because matlab showed:
    * Processing 280 images.
    * Progress: 0.36%
However, after that errors appeared. The error message was:
Error using nnet.internal.cnngpu.poolingMaxBackward2D
Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If
the problem persists, reset the GPU by calling 'gpuDevice(1)'.
The gpuDevice output was: CUDADevice with properties:
                      Name: 'Quadro GP100'
                     Index: 1
         ComputeCapability: '6.0'
            SupportsDouble: 1
             DriverVersion: 9
            ToolkitVersion: 9
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 1.7064e+10
           AvailableMemory: 1.4295e+10
       MultiprocessorCount: 56
              ClockRateKHz: 1442500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1
The linux command nvidia-smi output was:
 | Processes:                                                       GPU Memory |
 |  GPU       PID   Type   Process name                             Usage      |
 |=============================================================================|
 |    0      1016      G   /usr/lib/xorg/Xorg                            68MiB |
 |    0      1238      G   /usr/lib/xorg/Xorg                           231MiB |
 |    0      1326      G   /usr/bin/gnome-shell                         257MiB |
 |    0      3037      G   ...-token=0782EFEAC5474AAE6190DDBA57A724E1  1552MiB |
 |    0      9815    C+G   /opt/MATLAB/R2018a/bin/glnxa64/MATLAB       3712MiB |
 +-----------------------------------------------------------------------------+
It seemed that the memory was more than enough. What was the problem?
Thanks.

Best Answer

I understand that there's a bug in this demo script, and the default MiniBatchSize is too large and should be reduced to 4:
semanticseg(imdsTest,net,'MiniBatchSize',4, ...);
Related Question