I'm creating a convolutional neural network (CNN), where I have a convolutional layer followed by a pooling layer and I want to apply dropout to reduce overfitting. I have this feeling that the dropout layer should be applied after the pooling layer, but I don't really have anything to back that up. Where is the right place to add the dropout layer? Before or after the pooling layer?
Solved – Are pooling layers added before or after dropout layers
conv-neural-networkdeep learningdropout
Related Solutions
Although asked quite a while ago, I bumped into this question and saw it had no answers. Then found this from data science stack exchange:
https://datascience.stackexchange.com/questions/11699/backprop-through-max-pooling-layers
Quoting user104493:
"There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that actually achieves the max. Thus, the gradient from the next layer is passed back to only that neuron which achieved the max. All other neurons get zero gradient."
In convolutional neural network how can I identify overfitting?
Comparing the performance on training (e.g., accuracy) vs. the performance on testing or validation is the only way (this is the definition of overitting).
should dropout be blindly added to the model hoping that it increases testing or validation accuracy?
Dropout often helps, but the optimal dropout rate depends on the data set and model. Dropout may be applied to different layers in the network as well. Example from Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.:
You may also want to do some early stopping:
Best Answer
Edit: As @Toke Faurby correctly pointed out, the default implementation in tensorflow actually uses an element-wise dropout. What I described earlier applies to a specific variant of dropout in CNNs, called spatial dropout:
In a CNN, each neuron produces one feature map. Since
dropoutspatial dropout works per-neuron, dropping a neuron means that the corresponding feature map is dropped - e.g. each position has the same value (usually 0). So each feature map is either fully dropped or not dropped at all.Pooling usually operates separately on each feature map, so it should not make any difference if you apply dropout before or after pooling. At least this is the case for pooling operations like maxpooling or averaging.
Edit: However, if you actually use element-wise dropout (which seems to be set as default for tensorflow), it actually makes a difference if you apply dropout before or after pooling. However, there is not necessarily a wrong way of doing it. Consider the average pooling operation: if you apply dropout before pooling, you effectively scale the resulting neuron activations by
1.0 - dropout_probability, but most neurons will be non-zero (in general). If you apply dropout after average pooling, you generally end up with a fraction of(1.0 - dropout_probability)non-zero "unscaled" neuron activations and a fraction ofdropout_probabilityzero neurons. Both seems viable to me, neither is outright wrong.