MATLAB: How to get probabilities of each class which is classified with RUSBoost for an imbalanced data set

fitcensemblerusboostscores

I have a dataset with 7 classes and 3 features. The data set is hugely imbalanced. So, I referred https://www.mathworks.com/help/stats/classification-with-imbalanced-data.html to classfy the data. I get a prediction accuracy of 94%. But I need the probability of getting each class for a feature or set of features. How to get probability of each class to a given feature?

[Nt Mt] = size(y); % Number of observations in the training sample

t = templateTree('MaxNumSplits',Nt);

rusTree = fitcensemble(X,y,'Method','RUSBoost', 'NumLearningCycles',1000,'Learners',t,'LearnRate',0.1,'nprint',100);

[~,scores] = predict(rusTree,[1 16 3 5])

I get following scores for above code, 0.7345, 3.5105, 1.1893, 0, 0, 0, 0.0082

But above scores are not probablities, how to get values between 0-1 where sum of proabilities in all classes is equal to 1?

Best Answer

Hi,

The reason behind predict not returning scores as probability estimates is because the ‘RUSBoost’ algorithm used in the model does not treat scores as probabilistic estimates. Instead, the score represents the confidence of a classification into a class, higher, being more confidence as it is explained in the documentation link of fitcensemble .

If you would like to get probabilistic estimate for scores you can set the 'ScoreTransform' to 'logit' in 'fitcensemble'. This name-value pair transforms the score to probabilistic estimates. This is explained here. Then using predict on the model returns scores as probability values for each class.

Related Solutions

MATLAB: How to obtain scores as probabilistic estimates using the predict function on a fitcensemble model

The reason behind the command '[label, score] = predict(Mdl1,pX)' not returning scores as probability estimates is because the 'LogitBoost' algorithm used in the model does not treat scores as probabilistic estimates. Instead, the score represents the confidence of a classification into a class, higher, being more confidence.

This is explained in the documentation link below:

<https://www.mathworks.com/help/stats/compactclassificationensemble.predict.html#bvciha4>

If you would like to get probabilistic estimate for scores, you may consider one of two options:

1) You can set the 'ScoreTransform' name-value pair in the 'fitcensemble' to 'logit'. This name-value pair transforms the score to probabilistic estimates.

https://www.mathworks.com/help/stats/fitcensemble.html#bvcj_s0-1_sep_shared-ScoreTransform

Using predict on the model then returns scores as probability values for each class.

For example:

MdlFinal = fitcensemble(X,Y,'NumLearningCycles',idxNumTrees,...
    'Learners',tFinal,'LearnRate',learnRate(idxLR),'ScoreTransform','logit')
>> [~,scores] = predict(MdlFinal,X)
scores =
      0.0360    0.9640
      0.8054    0.1946
      0.0278    0.9722
  ...

2) Setting the 'Method' name-value pair of 'fitcensemble' to 'Bag'.

As mentioned in the documentation link here , the 'Bag' algorithm returns scores as probability values.

The documentation link for 'Method' name-value pair can be found here:

https://www.mathworks.com/help/releases/R2017a/stats/fitcensemble.html?s_tid=doc_ta#input_argument_d0e290851

MATLAB: What is the meaning of loss() and predict() in the case of random forest

From your question I understand that you are trying to fit a random forest classifier on your training data. I assume that ' trainY' contains categorical label since it is a classification task.

The answer to your question is in two folds. In your code

L = loss(rusTree,testX,testY,'mode','cumulative');

L will be having cumulative loss. This vector indicates the loss of the model when you add trees to your ensemble one by one cumulatively.This is very different from your evaluation of false positive and negatives from prediction label,since predict function uses all the trees at a time.You can plot the loss to see the trend

 figure;
 plot(loss(bag,Xtest,Ytest,'mode','cumulative'));
 xlabel('Number of trees');
 ylabel('Test classification error');

This trend will give you the idea of how your loss is varying as the number of trees grow.

MATLAB invokes different functions depending on the function signature(Function overloading).Function signature is determined by number of inputs, types of inputs,number of outputs.' predict' function has multiple definitions.In your code

[label,score] = predict(rusTree,testX);

there is a chance that predict function corresponds to predicting image category( doc_link ) getting invoked. In that case 'score' would be having negated average binary loss per class. Please be careful in your function signature.

I would recommend you to use following code to do your evaluation.

L = loss(rusTree,testX,testY);
[label,score,node,cnum] = predict(rusTree,testX);

You can refer to this link for more details. I hope this will answer your question.

Regards,

Darshan Bhat

Best Answer

Related Solutions

MATLAB: How to obtain scores as probabilistic estimates using the predict function on a fitcensemble model

MATLAB: What is the meaning of loss() and predict() in the case of random forest

Related Question