loss-functionsneural networksself-study
I am a beginner reading neural networks. I wonder why there are loss functions, and then there are auxiliary loss functions. (By definition, 'auxiliary' would mean supplementary, help, support).
My question is:
Does constructing an auxiliary loss function mean that they come as an 'afterthought'? After the loss function has been created? So is the purpose of having an auxiliary loss function be to improve performance that cannot be done by the loss function alone?
Unless you restrict the range of your inputs, the sigmoid may be giving you a problem. You won't even be able to learn the function $y=x$ if you have a sigmoid in the middle.
If you have a restricted range, then the input-hidden weights could scale the input values so that they hit the sigmoid at the part of the graph which looks like a line i.e. the part around 0:
Once you have the linear behavior, the hidden-output weights could re-scale the values back. The training process will take care of all this for you - but to test this hypothesis you could just train (and validate) with input values in $[-1,1]$.
Do you know what the network's function looks like? If you plot $(x_1, x_2) \rightarrow y$, does it look anything like a plane? At least in parts?
You could also try training with more data and seeing if the graph gets closer to $x_1+2x_2$.
Saturation means it's not updating . Which means it is on some kind of a local minimum rather than in the desired global minima.
So when increasing the learning rate it can jump our from that local minima and move further .
Refer to the image below about local minima and global minima problems .
Here's a good thread - Improving loss with , increasing learning rate
Best Answer
It was common to use auxiliary loss functions at one time (Google Inception Net and PSPNet) because the auxiliary loss hedged against vanishing gradient at layers far away from the loss function. This has fallen out of favor; my suspicion is that residual connections, rectifier activtation variants, and improved initialization schemes made auxiliary losses unnecessary.