Solved – Threshold to build confusion matrix

bayesian networkconfusion matrixmachine learningnaive bayesthreshold

I a have data set with 10 sections of data and each section shows one day observation. I
designed the training and test set as follows: 8 sections for training the data and the last two sections for the test set.
In this case, we have a binary (Male/Female) classification problem, so we applied different
classifiers (e.g., naive bayes, decision tree and bayesian network) to predict the output.
After applying the classifier on the training set and feeding the test set to assess the
performance of each classification algorithm, we obtained such probabilities for each class (male/female).

For example for Naive bayes:

    prob(Male)  prob(Female)   prediction
        0.3         0.70           ?    
        0.45        0.55           ?
        0.67        0.33           ?
        0.52        0.48           ?

For Decision Tree:

    prob(Male)  prob(Female)   prediction
        0.4         0.60           ?    
        0.65        0.35           ?
        0.54        0.26           ?
        0.49        0.51           ?

my first question is how the classifiers (e.f., naive bayes or decision…) construct the confusion matrix? do they use 50%
as the cut-point to build the confusion matrix (in binary classes), or no they use a specific function to calculate the optimal threshold?
My second question is, can we use the AUC function to calculate the optimal threshold and use
it to build the confusion matrix for all classifiers?

Best Answer

The classifiers can use arbitrary thresholds (default k/n - prior probability) so you can change them at will if you want. You can look into the prior argument and change these at will, for exp. setting them to 0.5 for equal probability.

You can use AUC or some other metric (such as Precision, Recall, etc.) to get the "optimal" threshold upon which to cut the probabilities.

Related Question