Solved – Neural networks – are local minima bad

neural networks

I hear a lot about local minima for neural networks. I understand the theory behind it – but if my neural network finds weights in a local minimum, is that a bad thing?

I understand that finding global minima (in Neural Networks) is usually a bad thing as well, since global minima usually overfits. However I am still a bit confused.

Do convergence to local minima give us bad solutions? Lastly, does convergence to local minima overfit to our training data?

Best Answer

  1. You can consider local minima L bad if a) your model does not overfit on L and b) there's some other minima L' which has significantly lower CV error rate than L.
  2. Global minima in NN is not usually a bad thing. It is bad only if your model overfits, but you can use always proper regularization and stop early.
  3. Overfitting has nothing to do with convergence, a model can overfit long before convergence.
Related Question