MATLAB: Regression Learner App RSME different from validating dataset

crossmodelr^2r-squaredrsmeStatistics and Machine Learning Toolboxvalidation

I'm using the Regression Learner app with the option of SVM Linear algorithm to train a dataset and build a regression model. Then I record the reported R^2 and RSME values from the app. When I export the model and run the model on the same training dataset from the command line, I get a very different R^2 value and RSME. I only get this problem with my large data sets (where the number of features is quite large – ~ 10,000 to 100,000).

Best Answer

The RSME and R-squared values in the Regression Learner App are coming from the Cross-Validated data. This is not the same as plugging in the dataset into the model and calculating the RSME and R-squared values. The Regression Learner App is using K-fold Cross validation and is validating at each step so it is not the same as the final model.

For more information on Cross-Validation, please see the following link:

https://www.mathworks.com/discovery/cross-validation.html

A good way to look deeper into how the App is calculating these values is to generate the code from the Regression Learner App and take a deeper look.

The reason why the RSME and R-squared values might be VERY far off, could be because there are too few data points relative to the very large number of features. This could be causing underfitting to occur.

Related Solutions

MATLAB: Adding MAPE/RSME to subplot

Try something like this

f = figure();
% random values
MAPE = [1.234 2.123 1.034 0.234];
MSE = [2.20 1.20 2.23 1.10];
R2 = [0.9 0.7 0.8 0.9];
% make random graphs
ax = gobjects(1,4);
for i=1:4
    ax(i) = subplot(2,2,i);
    plot((1:10)+2*rand(1,10));
    xLeft = ax(i).XLim(1);
    xDiff = diff(ax(i).XLim);
    yTop = ax(i).YLim(2);
    yDiff = diff(ax(i).YLim);
    text(xLeft+0.05*xDiff, yTop-0.05*yDiff, ['MAPE=' num2str(MAPE(i))], ...
        'HorizontalAlignment', 'left', 'VerticalAlignment', 'top', 'Interpreter', 'latex');
    text(xLeft+0.05*xDiff, yTop-0.15*yDiff, ['MSE=' num2str(MSE(i))], ...
        'HorizontalAlignment', 'left', 'VerticalAlignment', 'top', 'Interpreter', 'latex');
    text(xLeft+0.05*xDiff, yTop-0.25*yDiff, ['$R^2$=' num2str(R2(i))], ...
        'HorizontalAlignment', 'left', 'VerticalAlignment', 'top', 'Interpreter', 'latex');
end

Result:

MATLAB: Is the classifier trained twice when using holdout validation in ClassificationLearner App

The only purpose of the classifier that is trained on half of the dataset is to estimate the out-of-sample accuracy of the model that is trained on the full dataset.

The model trained on the full dataset is the one that is expected to perform better on out-of-sample data, because it is fit to more data. The key point here is that we care about the expected accuracy on out-of-sample data. Both classifiers are trained using the same hyperparameters (i.e., options like “number of learners”). Because they use the same hyperparameters, fitting to more data should only increase out-of-sample accuracy.

Best Answer

Related Solutions

MATLAB: Adding MAPE/RSME to subplot

MATLAB: Is the classifier trained twice when using holdout validation in ClassificationLearner App

Related Question