Solved – How to improve the accuracy of a neural network model

Question

I am attempting to create a neural network model using R that estimates the sine function for a given x variable, i.e. sin(x). Dataset has 5000 observations with x iterating by 0.05 each time:

However, when I estimate using the model, my error is quite high at 219.2 and my prediction remains fairly constant at a value of 0.48 or thereabouts, while the actual value deviates sharply between 0 and 1:

The steps I am taking to estimate the model are as follows:

1) I am scaling my sin and x variables using max-min normalization:

sinscaled=(sin-min(sin))/(max(sin)-min(sin))
xscaled=(x-min(x))/(max(x)-min(x))

2) I am then fitting my training and test data:

trainset <- maxmindf[1:3500, ]
testset <- maxmindf[3501:5000, ]

3) The neural network is fitted as below, and the resulting output tested:

#Neural Network
library(neuralnet)
sinscalednet <- neuralnet(sinscaled ~ xscaled, data=trainset, 
hidden=c(2,1), linear.output=FALSE, threshold=0.01)
sinscalednet$result.matrix
plot(sinscalednet)

Note that while I am using 2 hidden layers in this instance, the error is remaining consistent even when the number of hidden layers are being manipulated:

4) Resulting output is tested and accuracy obtained:

#Test the resulting output
temp_test <- subset(testset, select = c("xscaled"))
head(temp_test)
sinscalednet.results <- compute(sinscalednet, temp_test)

#Accuracy
results <- data.frame(actual = testset$sin, prediction = 
sinscalednet.results$net.result)
results
roundedresults<-sapply(results,round,digits=0)
roundedresultsdf=data.frame(roundedresults)
attach(roundedresultsdf)
table(actual,prediction)

Would be grateful for any advice on steps I need to take to improve this model. I am experimenting by trying to modify the hidden layers, but the error is remaining consistently high.

Answer 1

What @Chaconne mentioned in the comments is quite important. You should first shuffle your training set and then split the array into chunks.

But I rewrote your code to the Javascript neural network library Neataptic, and i'm not having any issues. This is the code I wrote:

// Create the network
var network = new Architect.Perceptron(1,2,1);

// Create training set
var set = [];
for(var i = 1; i <= 5000; i++){
  var x = i * 0.05;
  x = x / (5000 * 0.05);

  var y = Math.sin(x);
  y = (y + 1) / 2;

  set.push({ input: [x], output: [y]});
}

var trainingSet = set.splice(0, 3500);
var testSet = set;

// Train the network
var training = network.train(trainingSet, {
  iterations: 500, // it will never reach this
  error: 0.0001,
  rate: 0.01,
  cost: Methods.Cost.MSE
});

var testError = network.test(testSet, Methods.Cost.MSE);

Run the code yourself here (open console)!

I'm getting these kind of results somewhat consistently:

training error: 0.00008
test error:  0.00133

So dropout would fix the large gap, but my point is, something else must be wrong in your model. Consistently getting 0.48 as prediction seems like the network has never learned anything.

What is your training error? What is your test error? How long are you training for (iterations).