In answers like this, Greg Heath suggests using the normalized mean square error, NMSE, to compare the performance of different neural networks and pick the best one.
I have been calculating NMSE from all samples in the training set t and prediction y,
[net tr y e ] = train(net,x,t); % Train network
vart1 = var(t',1);% MSE for a naive constant output model
% that always outputs average of target data
MSE00 = mean(vart1);NMSE = mse(t-y)/MSE00; % Normalize
That includes the training samples, and so may favor models that fit the training data well but not new data. In order to choose the most robust model, should I calculate NMSE from the test samples only?
iTest = tr.testInd; % Index to the samples that were set aside for testing
NMSE_test_only = mse(t(:,iTest)-y(:,iTest))/MSE00; % Only use test samples
Best Answer