MATLAB: What are the implications of the observation weights on the “Bag”, and “AdaBoostM1” methods in the “fitcensemble” function

adaboostm1bagobservationStatistics and Machine Learning Toolboxweights

What are the implications of the observation weights on the "Bag", and "AdaBoostM1" methods in the "fitcensemble" function?

Are the observations with higher weights more likely to be randomly picked by the “bootstrap-sample-algorithm” used to create a tree?

Best Answer

The ‘Bag’ method implements random forest. For every tree, observations are sampled with replacement using the weights that you specify. The tree is then grown using these sampled observations without weights. That is, the weights are only used to sample observations, but they are not passed to the tree.

In the "AdaBoostM1" method, the observation weights are used in the training of the model for minimization of the exponential loss. Please refer to the following link for more information on the usage of observation weights in "AdaBoostM1" algorithm:

https://www.mathworks.com/help/stats/ensemble-algorithms.html#bsw8aue

Unless you specify the ‘Resample’ option as ’on’, all the observations are used to train every tree in the ensemble.

At every iteration, the observation weights are updated as described in the above link and passed to the tree. The tree then uses these weights to find the optimal splits. That is, splitting criteria such as Gini index are computed by summing the observation weights per class.

Related Solutions

MATLAB: How to use the “predictorImportance” function with models generated by applying “crossval” to RegressionTree objects

It is important to note that ‘predictorImportance’ can only be applied to one model at a time. The result of using the “crossval” function on a regression tree will be a set of regression models and thus you will need to index into each of these models to determine the predictor importance.

The following example code which you can execute in the MATLAB command window shows how you can call ‘predictorImportance’ on the results of ‘crossval’

%%Load the sample data.
load carsmall; 
%%Construct a regression tree using the sample data.
tree = fitrtree([Weight, Cylinders],MPG,...
                'categoricalpredictors',2,'MinParentSize',20,...
                'PredictorNames',{'W','C'})  ;
%%cross validation
Ctree = crossval(tree);
% where tree is the original ‘RegressionTree’ object. Note that the default number of folds i.e. regression models generated will be 10
%%Calling predictorImportance on individual regression models
predictorImportance(Ctree.Trained{1})
predictorImportance(Ctree.Trained{10})
% where 1 and 10 represent the indices of the trained models. This would range from 1 to 10 by default

For more information on how to use the ‘fitrtree’ function please refer to the following link:

https://www.mathworks.com/help/stats/fitrtree.html

MATLAB: How will there be any out-of-bag predictions if each replica in the TreeBagger has the data set size equal to the number of observations in the training set

The "InBagFraction" property is used to indicate what fraction of the input data is to be sampled with replacement from the input data for growing each new tree.

If the value of this property is "1", then it means that each bootstrap replica has the same total number of observations, however this does not mean it has the same set of observations. This is because each replica is constructed via sampling with replacement, which means that the observations fed to a bootstrap replica are sampled from the original data.

For example, let N be the number of observations in the data passed to "TreeBagger". If "InBagFraction" is 1, N observations are sampled out of N. Sampling is done with replacement if you use the default value for "SampleWithReplacement". There would be out-of-bag predictions in this case because sampling is done with replacement.

Best Answer

Related Solutions

MATLAB: How to use the “predictorImportance” function with models generated by applying “crossval” to RegressionTree objects

MATLAB: How will there be any out-of-bag predictions if each replica in the TreeBagger has the data set size equal to the number of observations in the training set

Related Question