I've built a linear model with Matlab using fitlm to predict the next value in a series of doubles. Now I would like to test this model on a different dataset so I get accuracy, p-value etc.
I've used predict, but that just outputs a prediction curve and a confidence interval.
How do I get an accuracy for this new data? What would be the correct way to evaluate this model on the test set? Could I somehow specify the testset when I am building fitlm in the first place?
I know this can be done in packages such as Weka, so I was wondering about the options in Matlab.
Best Answer
You could compute the mean squared error on the test set,
and compare it with the mse that is computed from the training data in the original model.
This function does ordinary least squares. It uses classical statistical approaches to adjust mse for the number of parameters in the model, perform t and F tests, and compute an adjusted R-square. It doesn't do regularization and it takes the model as given. So it doesn't take a test set.
Your ideas are interesting and could help you build up the model that you specify to fitlm. For example, you could watch the relationship between the computed mse and testmse as you add more terms to the model. I haven't seen this done (of course I haven't seen everything), but it seems plausible.