MATLAB: How will there be any out-of-bag predictions if each replica in the TreeBagger has the data set size equal to the number of observations in the training set

MATLABout-of-bagpredictionstreebagger

How will there be any out-of-bag predictions if each replica in the TreeBagger has the data set size equal to the number of observations in the training set?

If the "InBagFraction" is "1" by default, then how much data is sampled from the entire set for each tree?

If this fraction refers to how much data is sampled for each tree, and by default it is 1, then how would there be any out of bag predictions?

Best Answer

The "InBagFraction" property is used to indicate what fraction of the input data is to be sampled with replacement from the input data for growing each new tree.

If the value of this property is "1", then it means that each bootstrap replica has the same total number of observations, however this does not mean it has the same set of observations. This is because each replica is constructed via sampling with replacement, which means that the observations fed to a bootstrap replica are sampled from the original data.

For example, let N be the number of observations in the data passed to "TreeBagger". If "InBagFraction" is 1, N observations are sampled out of N. Sampling is done with replacement if you use the default value for "SampleWithReplacement". There would be out-of-bag predictions in this case because sampling is done with replacement.

Related Solutions

MATLAB: How to find the Multi-Class Hyperplane Decision Boundaries using Support Vector Machines (SVM)

Example on Multi-Class Hyperplane Decision Boundaries

Please refer to the following example, illustrating the required steps for 3 different classes. The example can also be found in the attached "multiclassHyperplaneDecisionBoundary" MATLAB Live Script.

Create and visualize random data with 3 classes

M  = 200;
X = [mvnrnd([-1,0],2*eye(2),M); ...
    mvnrnd([2,2],eye(2),M); ...
    mvnrnd([-1,4],eye(2),M)];
y = [ones(M,1); 2*ones(M,1); 3*ones(M,1)];
figure;
plot(X(y==1,1),X(y==1,2),'.','DisplayName','1'); hold on
plot(X(y==2,1),X(y==2,2),'.','DisplayName','2');
plot(X(y==3,1),X(y==3,2),'.','DisplayName','3'); hold off

Create a multiclass error-correcting output codes (ECOC) model for support vector machines (SVM) Create an SVM template, and standardize the predictors

template = templateSVM( 'Standardize', 1);

Except for the StandardizeData, Method, and Type properties, most of the template object's properties are empty. When training the ECOC classifier, the software sets the applicable properties to their default values.

Use the specified predictor data in X and the response data in y to train the ECOC model:

ecoc = fitcecoc(X,y,'Learners',template);

Use separating hyperplane equation to classify sample data A compact SVM object for binary classification can be obtained from the trained ECOC model by indexing the 'BinaryLearners' array:

idx = 1;
svm = ecoc.BinaryLearners{idx};

The hyperplane separating two classes, for an observed sample x = (x1,x2), can be found by setting its function f(x) = 0. The separating hyperplane equation is given by

f =@(x) ((x(1)-svm.Mu(1))/svm.Sigma(1))*svm.Beta(1)/svm.KernelParameters.Scale + ...
    ((x(2)-svm.Mu(2))/svm.Sigma(2))*svm.Beta(2)/svm.KernelParameters.Scale + svm.Bias;

Note that svm.Beta/svm.KernelParameters.Scale can be simplified to svm.Beta since

svm.KernelParameters.Scale
ans =
     1

The relative location of a test sample w.r.t. the separating hyperplane can found by evaluating the separating hyperplane equation and looking at the sign of the obtained value:

testSample = mean(X) + std(X).*randn(1,2);
testSVMClass = sign(f(testSample))
testSVMClass =
     1

The relation between binary SVM class names and the corresponding ECOC input data class can be easily found with the following command:

[svm.ClassNames ecoc.ClassNames(ecoc.CodingMatrix(:,idx)~=0)]
ans =
    -1     1
     1     2

Please note that the the predict function (<https://mathworks.com/help/releases/R2018a/stats/compactclassificationsvm.predict.html documentation link>) can also be used for this purpose:

testSVMClassPred = predict(svm,testSample)
testSVMClassPred =
     1

Plot decision boundary for each binary classification SVM in the ECOC model Following this approach, the decision boundaries for all ECOC model input classes can be calculated following the calculation procedure below:

for i = 1:length(ecoc.BinaryLearners)
    svm = ecoc.BinaryLearners{i};
    % Step 1: Choose hyperplane's x1 values
    x1vals = linspace(min(X(:,1)),max(X(:,1)));
      % Step 2: Solve for ((x2-svm.Mu(2))/svm.Sigma(2)) in terms of x1.
      x2fun =@(x1) -svm.Beta(1)/svm.Beta(2) * ((x1-svm.Mu(1))./svm.Sigma(1)) - svm.Bias/svm.Beta(2);
      % Step 3: Calculate corresponding x2 values
      x2vals = x2fun(x1vals) * svm.Sigma(2) + svm.Mu(2);
      hold on;
      plot(x1vals,x2vals,'--', ...
          'DisplayName',mat2str(ecoc.ClassNames(ecoc.CodingMatrix(:,i)~=0)'));
      hold off;
  end
  legend('show','Location','eastoutside','Orientation','vertical');

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

For every tree, the bagger randomly selects N*bagger.FBoot out of N observations with replacement (default) for training. Observations that were not selected for training are out-of-bag observations. If bagger.FBoot=1 (default), on an average roughly 2/3 of input data is selected for training for every tree and the remaining 1/3 is used as out-of-bag observations. This number can fluctuate from one tree to another, and out-of-bag observations for one tree are not identical to out-of-bag observations for another tree.

You can use the following code as an example to determine the fraction of out-of-bag observations per tree.

load imports-85;
Y = X(:,1);
X = X(:,2:end);
ntrees = 50;
for j = [0.5 0.8 1]
      b = TreeBagger(ntrees,X,Y,'oobvarimp','on','Fboot',j);
      [obs vars] = size(b.X);
      num_oob_per_tree = sum(sum(b.OOBIndices))/ntrees;
      fprintf(['\n\nFor ' num2str(ntrees) ' trees and FBoot = ' num2str(j) ':\n'])
      frac_oob_observations = num_oob_per_tree/obs
end

Best Answer

Related Solutions

MATLAB: How to find the Multi-Class Hyperplane Decision Boundaries using Support Vector Machines (SVM)

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

Related Question