neural networksr
I am using function neuralnet in the package neuralnet to build the neural network, and I see the error:
algorithm did not converge in 1 of 1 repetition(s) within the stepmax
The neural network has 20 inputs and 1 output. The problem is, with the same data and same set of inputs, I ran linear regression or random forest without any problem. So what should I look to for debugging my problem?
In general you would get more stability by increasing the number of hidden nodes and using an appropriate weight decay (aka ridge penalty).
Specifically, I would recommend using the caret package to get a better understanding of your accuracy (and even the uncertainty in your accuracy.) Also in caret is the avNNet that makes an ensemble learner out of multiple neural networks to reduce the effect of the initial seeds. I personally haven't seen huge improvement using avNNet but it could address your original question.
I'd also make sure that your inputs are all properly conditioned. Have you orthogonalized and then re-scaled them? Caret can also do this pre-processing for you via it's pcaNNet function.
Lastly you can consider tossing in some skip layer connections. You need to make sure there are no outliers/leverage points in your data to skew those connections though.
Unless you restrict the range of your inputs, the sigmoid may be giving you a problem. You won't even be able to learn the function $y=x$ if you have a sigmoid in the middle.
If you have a restricted range, then the input-hidden weights could scale the input values so that they hit the sigmoid at the part of the graph which looks like a line i.e. the part around 0:
Once you have the linear behavior, the hidden-output weights could re-scale the values back. The training process will take care of all this for you - but to test this hypothesis you could just train (and validate) with input values in $[-1,1]$.
Do you know what the network's function looks like? If you plot $(x_1, x_2) \rightarrow y$, does it look anything like a plane? At least in parts?
You could also try training with more data and seeing if the graph gets closer to $x_1+2x_2$.
Best Answer
You might try with increasing the
repargument. If it fails, try with increasing thestepmaxargument too.One repetition of the training is called an epoch. Usually, several epochs are needed.