I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible.
Currently I have an input layer consisting of 21 Neurons and a Dense Output layer consisting of 4 Neurons. Would it make sense to use a Dropout Layer to prevent Overfitting?
I read that using dropout layers makes sense in general but for this particular problem it seems counterintuitive, since I am creating an Error every time I ignore a certain Neuron so the other neurons would be trained to "overcompensate" this error.
Solved – Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error
dropoutmachine learningneural networksregression
Best Answer
Dropout is part of the array of techniques we developed to be able to train Deep Neural Networks on vast amount of data, without incurring in vanishing or exploding gradients: minibatch training, SGD, skip connections, batch normalization, ReLU units (though the jury is still out on these last ones: maybe they help with "pruning" the network, but not with vanishing gradients). They complement the "old school regularizarion tools" such as weight decay and early stopping.
Your neural network has only two layers (and, unless it has some nonlinear activation units, it's actually equivalent to a single layer NN): first layer has 21 units and second has 4. It's definitely not a Deep Neural Network...Now, without having more details on your specific application and data, I can't say for sure, but with such a shallow network it seems really unlikely that you will need dropout.