I'm currently reading about neural networks, specifically how loss functions saturation can cause problems. During my studies, I was curious if one could remedy the problem during training of neural networks by detecting saturation (such as comparing the gradients to the previous training iteration) and just increase the learning rate in a proportional manner so that learning still can be done effectively. Unfortunately, my (admittately limited) research on google has not come up with any method using this approach.
Is there a reason why this is not being done?
Best Answer
Saturation means it's not updating . Which means it is on some kind of a local minimum rather than in the desired global minima.
So when increasing the learning rate it can jump our from that local minima and move further .
Refer to the image below about local minima and global minima problems .
Here's a good thread - Improving loss with , increasing learning rate