neural networksoverfitting
To have a neural network that performs perfectly on training set, but poorly on validation set, what am I supposed to do? To simplify, let's consider it a CIFAR-10 classification task.
For example, "no dropout" and "no regularization" would help, but "more layers" doesn't necessarily. I'm also wondering, do "batch size", choice of optimizer make any difference on overfitting?
In convolutional neural network how can I identify overfitting?
Comparing the performance on training (e.g., accuracy) vs. the performance on testing or validation is the only way (this is the definition of overitting).
should dropout be blindly added to the model hoping that it increases testing or validation accuracy?
Dropout often helps, but the optimal dropout rate depends on the data set and model. Dropout may be applied to different layers in the network as well. Example from Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.:
You may also want to do some early stopping:
Increasing the number of hidden units and/or layers may lead to overfitting because it will make it easier for the neural network to memorize the training set, that is to learn a function that perfectly separates the training set but that does not generalize to unseen data.
Regarding the batch size: combined with the learning rate the batch size determines how fast you learn (converge to a solution) usually bad choices of these parameters lead to slow learning or inability to converge to a solution, not overfitting.
The number of epochs is the number of times you iterate over the whole training set, as a result, if your network has a large capacity (a lot of hidden units and hidden layers) the longer you train for the more likely you are to overfit. To address this issue you can use early stopping which is when you train you neural network for as long as the error on an external validation set keeps decreasing instead of a fixed number of epochs.
In addition, to prevent overfitting overall you should use regularization some techniques include l1 or l2 regularization on the weights and/or dropout. It is better to have a neural network with more capacity than necessary and use regularization to prevent overfitting than trying to perfectly adjust the number of hidden units and layers.
Best Answer
If you have a network with two layers of modifiable weights you can form arbitrary convex decision regions, where the lowest level neurons divide the input space into half-spaces and the second layer of neurons performs an "AND" operation to determine whether you are in the right sides of the half-spaces defining the convex region. In the diagram below you can form regions r1 and r2 this way. If you add an extra later, you can form arbitrary concave or disjoint decision regions by combining the outputs of the sub-networks defining the convex sub-regions. I think I got this proof from Philip Wasserman's book "Neural Computing: Theory and Practice" (1989).
Thus is you want to over-fit, use a neural network with three hidden layers of neurons, use a huge number of hidden layer neurons in each layer, minimise the number of training patterns (if allowed by the challenge), use a cross-entropy error metric and train using a global optimisation algorithm (e.g. simulated annealing).
This approach would allow you to make a neural network that had convex sub-regions that surround each training pattern of each class, and hence would have zero training set error and would have poor validation performance where the class distributions overlap.
Note that over-fitting is about over-optimising the model. An over-parameterised model (more weights/hidden units than necessary) can still perform well if the "data mismatch" is not over-minimised (e.g. by applying regularisation or early stopping or being fortunate enough to land in a "good" local minimum).