I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one for the true values. Obviously doing a comparison allows one to judge the performance of the model by its predictive power using things like F-Score, Kappa Statistic, Precision & Recall, ROC curves etc.
How does this compare to evaluating numeric prediction like regression? I would assume that you could train the regression model on the training set, use it to predict values, then compare these predicted values to the true values sitting in the test set. Obviously the measures of performance would have to be different since this isn't a classification task.
The usual residuals and $R^2$ statistics are obvious measures but are there more/better ways to evaluate the performance for regression models? It seems like classification has so many options but regression is left to $R^2$ and residuals.
Best Answer
As said, typically, the Mean Squared Error is used. You calculate your regression model based on your training set, and evaluate its performance using a separate test set (a set on inputs x and known predicted outputs y) by calculating the MSE between the outputs of the test set (y) and the outputs given by the model (f(x)) for the same given inputs (x).
Alternatively you can use following metrics: Root Mean Squared Error, Relative Squared Error, Mean Absolute Error, Relative Absolute Error... (ask google for definitions)