MATLAB: Issue with batch normalization layer of saved CNN

batch normalizationcheckpointcnnDeep Learning Toolbox

When loading a previously trained CNN, I got an issue with the batch normalization layers. When looking into the loaded CNN model the trainable mean and variance are empty.

Name: 'batchnorm_1'

TrainedMean: []

TrainedVariance: []

So the checkpoint doesn't seem to save these parameters. Are there any workarounds for this issue? I am using Matlab R2018b.

Best Answer

We were able to reproduce the issue. We will inform you once the issue is fixed.

Since TrainedMean and TrainedVariance are calculated after the training is finished, therefore as a workaround you can use the below mentioned codes to explicitly save and load the Model.

%To save model with name "demoModel", assuming your network is in "net" 
save('demoModel','net')

%To load model to variable net1
net1=load('demoModel.mat','net');
net1.net.Layers(n).TrainedMean %where n is the batch normalization layer

Related Solutions

MATLAB: Want to add regularization (L2) in Resnet50 code

You can specify the L2 regularization factors for the weights and biases in Convolutional and Fully Connected Layers by specifying the BiasL2Factor and WeightL2Factor properties, respectively. trainNetwork then multiplies the L2 regularization factors that you specify by using trainingOptions with these factors.

For more informtion refer to setL2Factor, Solver Options, L2 Regularization, Set Up Parameters in Convolutional and Fully Connected Layers.

MATLAB: Prediction on a pre-trained neural network

Using weights of different network to the current network:

for 2 networks net1 & net2, copy the layers to some variable

layers = net1.Layers;
layers(net1layerindex).Weights(net1index) = net2.Layers(net2layerindex).Weights(net2index);

Let's say you want to set weights of a channel of a convolution2dLayer to zeros(channelDimension)

layers(convLayerIndex).Weights(:,:,channelIndex) = zeros(channelDimension)
%also
layers(convLayerIndex).Weights(:,:,channelIndex,Index) = zeros(respectiveDimension)

In case of batchNormalizationLayer it has TrainedMean, TrainedVariance, Scale & Offset.

Use the above layers to create a DAG or a Series network. You can also refer to the documentation of the assembleNetwork function which helps creating deep learning networks from layers without training.

You can also set the Learning factor for a layer as follows:

layer.WeightLearnRateFactor = 0;

Best Answer

Related Solutions

MATLAB: Want to add regularization (L2) in Resnet50 code

MATLAB: Prediction on a pre-trained neural network

Related Question