MATLAB: How to re-train a model optimized by Bayesian optimization on new data

bayesian optimizationfitrensemblehyperparameter optimizationmachine learningre-train modelStatistics and Machine Learning Toolbox

Hi,

I optimized a regression ensemble

model = fitrensemble(X,Y,...);

with active HyperparameterOptimizationOptions, so the resulting model object of type RegressionBaggedEnsemble contains a HyperparameterOptimizationResults object. How can I use this easily to re-train the model on a new data set with the best point found by the hyperparameter optimization algorithm? Something like this would be nice:

newModel = fitrensemble(Xnew,YNew,model.HyperparameterOptimizationResults.bestPoint);

My current approach is tedious, because I have to distinguish between the method the optimization algorithm selected (Bag,LSBoost) and set all parameters manually (and I'm even not sure if this is correct):


best = model.HyperparameterOptimizationResults.bestPoint;
ttmp = templateTree('MinLeafSize',best.MinLeafSize,'MaxNumSplits',...
    best.MaxNumSplits,'NumVariablesToSample',best.NumVariablesToSample);
if best.Method=='Bag' %#ok<BDSCA>
    newModel = fitrensemble(Xnew,YNew,...
        'Method','Bag','Learners',ttmp,'NumLearningCycles',best.NumLearningCycles);
else
    newModel = fitrensemble(Xnew,YNew,...
        'Method','LSBoost','Learners',ttmp,'NumLearningCycles',best.NumLearningCycles,'LearnRate',best.LearnRate);
end

This question is not restricted to fitrensemble, but includes all similar model functions available in Statistics and Machine Learning Toolbox (fitrlinear, fitrgp …).

Best Answer

Your current approach (explicitly passing the values that the optimization found) is the right way to do it.

There is a faster/simpler way which uses undocumented functionality, and which therefore may change in the future. So, use at your own risk. You create a template from your model and then fit it:

model = fitrensemble(X,Y,...);
tmp = classreg.learning.FitTemplate.makeFromModelParams(model.ModelParameters);
newModel = fit(tmp,Xnew,Ynew)

Related Solutions

MATLAB: Finding optimal regression tree using hyperparameter optimization

My guess is that your first run was worse because it was not run for enough iterations. The default MaxObjectiveEvaluations is 30 iterations, but since your first optimization searches a larger space (including a categorical variable) you should probably multiply that a few times. You're also using 'Repartition'=true which calls for more iterations. Try running it for at least 100 iterations. The more the better as time permits. You can pass MaxObjectiveEvaluations inside HyperparameterOptimizationOptions.

The objective being minimized for regression is log(1 + MSE) computed on the validation set. By default that's 5-fold crossvalidation. That's mentioned near the bottom of the OptimizeHyperparameters section on this doc page: http://www.mathworks.com/help/stats/fitrensemble.html#input_argument_d0e360201 Your final calls to kfoldLoss will return MSE, which will differ from the objective function values.

In any case, you should use the model that has the lowest cross-validated MSE no matter how you found it.

MATLAB: How to choose the best kernel for a Gaussian process regression, possibly using “bayesopt” function? How to predict unseen data with the resulting model

There are two ways you could do this. One of them involves "bayesopt" function.

(1) The easiest way is to use the "fitrgp" function. There is an example of how to optimize a fitrgp hyperparameter (Sigma) on the "fitrgp" function Documentation page:

https://www.mathworks.com/help/stats/fitrgp.html?searchHighlight=fitrgp

To optimize ‘KernelFunction’, using default values for all other hyperparameters, you could do the following (using data from the documentation example):

load(fullfile(matlabroot,'examples','stats','gprdata2.mat'))
model1 = fitrgp(x,y,'OptimizeHyperparameters',{'KernelFunction'});

Because fitrgp tends to be sensitive to the Sigma parameter, you will probably get a better result if you simultaneously optimize KernelFunction and Sigma. Also, searching categorical variables inherently requires more function evaluations, so you should probably run it longer:

model2 = fitrgp(x,y,'OptimizeHyperparameters',{'KernelFunction','Sigma'},...
    'HyperparameterOptimizationOptions', struct('MaxObjectiveEvaluations',100));

The "fitrgp" function documentation page lists the hyperparameters that are eligible for optimization, and what values are searched:

https://www.mathworks.com/help/stats/fitrgp.html#input_argument_namevalue_OptimizeHyperparameters.

You can use this model to make predictions on unseen data directly with the "predict" function:

model2 = fitrgp(x,y,'OptimizeHyperparameters',{'KernelFunction','Sigma'},...
    'HyperparameterOptimizationOptions', struct('MaxObjectiveEvaluations',100));
% Predict on unseen data

xNew = 0.2
ypred = predict(model2, xNew)

(2) A second approach would be to use "bayesopt" function directly, defining your own objective function. Here, the objective function is the 5-fold crossvalidation loss of the model.

load(fullfile(matlabroot,'examples','stats','gprdata2.mat')) 
% Define kernel variable
kernel = optimizableVariable('KernelFunction',{'exponential','squaredexponential','matern32','matern52',...
'rationalquadratic','ardexponential','ardsquaredexponential','ardmatern32','ardmatern52','ardrationalquadratic'},...
'Type','categorical') 
% Call bayesopt, capturing x and y in the objective function

bo = bayesopt(@(T)objFcn(T,x,y), kernel)
% Define an objective function
function Loss = objFcn(Vars, x, y)
m = fitrgp(x, y, 'KernelFunction', char(Vars.KernelFunction), 'KFold', 5);Loss = kfoldLoss(m);
end

(2.1) To optimize both KernelFunction and Sigma using "bayesopt" function, you can do the following:

% Define 2 variables
kernel = optimizableVariable('KernelFunction',{'exponential','squaredexponential','matern32','matern52',...
'rationalquadratic','ardexponential','ardsquaredexponential','ardmatern32','ardmatern52','ardrationalquadratic'},...
'Type','categorical')
sigma = optimizableVariable('Sigma',[1e-4,10],'Transform','log')
 
% Call bayesopt, capturing x and y in the objective function
bo = bayesopt(@(T)objFcn(T,x,y), [sigma, kernel], 'MaxObjectiveEvaluations', 100)
 
function Loss = objFcn(Vars, x, y)
m = fitrgp(x, y, 'KernelFunction', char(Vars.KernelFunction), ...
                 'Sigma', Vars.Sigma, 'ConstantSigma', true,...
                 'KFold', 5);
Loss = kfoldLoss(m);
end

To get a model that can make predictions, you need to fit a model without passing the ‘kfold’ argument, but keeping the optimal kernel function obtained from "bayesopt" function.

kernel= bo.XAtMinObjective.KernelFunction
model = fitrgp(x, y, 'KernelFunction', char(kernel));
% Predict on unseen data
xNew = 0.2
ypred = predict(model2, xNew)

Best Answer

Related Solutions

MATLAB: Finding optimal regression tree using hyperparameter optimization

MATLAB: How to choose the best kernel for a Gaussian process regression, possibly using “bayesopt” function? How to predict unseen data with the resulting model

Related Question