Solved – Output layer of artificial neural networks when learning non-linear functions with limited value range

neural networks

I want to approximate a non-linear function with a limited value range by an artificial neural network (feed forward, back propagation).
Most tools and literature availabe suggest linear functions for the output neurons when doing regressions. However, I know a priori that my goal function is of limited range, therefore is it reasonable to use a function for the output neurons with limited value range, too? To be more explicit: My target function's values are in the range between 0 and 1, but the neural net does predict occasionally values that exceed this range (e.g. -1.3). Can i prevent the net from doing so, and is it reasonable?

Best Answer

I am opposed to cutting values of, since this will lead to an undifferentiable transfer function and your gradient based training algorithm might screw up.

The sigmoid function at the output layer is fine: $\sigma(x) = \frac{1}{1 + e^{-x}}$. It will squash any output to lie within $(0, 1)$. So you can get arbitrarily close to the targets.

However, if you use the squared error you will lose the property of a "matching loss function". When using linear outputs for a squared error, the derivatives of the error reduce to $y - t$ where $y$ is the output and $t$ the corresponding target value. So you have to check your gradients.

I have personally had good results with sigmoids as outputs when I have targets in that range and using sum of squares error anyway.

Related Question