MATLAB: Difference between individual and cumulative oobMargin of TreeBagger

Statistics and Machine Learning Toolboxtreebagger

Why aren't the following two plots the same?

b = TreeBagger(500,X,Y,'oobpred','on');
mc = oobMargin(b,'mode','cumulative');
mi = oobMargin(b,'mode','individual');
figure; plot(mc.')
figure; plot(bsxfun(@rdivide,cumsum(mi.'),(1:500).'))

Best Answer

When you ask for an OOB margin from one tree, you get zero if this observation was in bag for this tree. The margin is undefined in this case, and TreeBagger returns 0 by default. The cumulative calculation averages over trees for which this observation was out of bag only. Check this out:

>> load fisheriris

>> b = TreeBagger(10,meas,species,'oobpred','on');

>> mi = oobMargin(b,'mode','individual');

>> mi(1,:)

ans = 1 0 0 0 1 0 1 1 0 0

>> b.OOBIndices(1,:)

ans = 1 0 0 0 1 0 1 1 0 0

>> mc = oobMargin(b,'mode','cumulative');

>> mc(1,:)

ans = 1 1 1 1 1 1 1 1 1 1

Related Solutions

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

For every tree, the bagger randomly selects N*bagger.FBoot out of N observations with replacement (default) for training. Observations that were not selected for training are out-of-bag observations. If bagger.FBoot=1 (default), on an average roughly 2/3 of input data is selected for training for every tree and the remaining 1/3 is used as out-of-bag observations. This number can fluctuate from one tree to another, and out-of-bag observations for one tree are not identical to out-of-bag observations for another tree.

You can use the following code as an example to determine the fraction of out-of-bag observations per tree.

load imports-85;
Y = X(:,1);
X = X(:,2:end);
ntrees = 50;
for j = [0.5 0.8 1]
      b = TreeBagger(ntrees,X,Y,'oobvarimp','on','Fboot',j);
      [obs vars] = size(b.X);
      num_oob_per_tree = sum(sum(b.OOBIndices))/ntrees;
      fprintf(['\n\nFor ' num2str(ntrees) ' trees and FBoot = ' num2str(j) ':\n'])
      frac_oob_observations = num_oob_per_tree/obs
end

MATLAB: Remove or replace trees from a TreeBagger ensemble

Combining two objects would be hard. You can work around this by growing one big ensemble and treating parts of it as separate ensembles. Take a look at the 'trees' argument for PREDICT and similar mehods.

This recipe is justified because every tree in a TreeBagger ensemble is grown independently of others. For prediction, trees are combined by simple averaging. This approach would not work for ensembles of other types.

Here is how I effectively grow two TreeBagger ensembles and then combine them:

% Load data
load ionosphere;
% Grow two ensembles, 100 trees each
b = TreeBagger(200,X,Y,'oobpred','on');
% Out-of-bag prediction from 1st ensemble
Yoob1 = oobPredict(b,'trees',1:100);
% Out-of-bag prediction from 2nd ensemble
Yoob2 = oobPredict(b,'trees',101:200);
% Randomly drop 25 trees from the 1st ensemble and combine the two
% ensembles.
idx1 = datasample(1:100,75,'replace',false);
Yoob3 = oobPredict(b,'trees',[idx1 101:200]);

Best Answer

Related Solutions

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

MATLAB: Remove or replace trees from a TreeBagger ensemble

Related Question