Solved – How to calculate 95%CI for F1 score and Matthews correlation coefficient

confidence intervalcorrelationmachine learningmathematical-statistics

How can I calculate the 95%CI for F1 score, Matthews correlation coefficient and fraction correct ((TN+TP)/(TP+TN+FP+FN)) when I evaluating a perfomance of binominal diagnostic test?

Best Answer

F1, MCC and other such metrics can be thought of as proportions, since they are in [0,1]. The binomial proportion confidence interval can be estimated with (using normal approximation)

Related Solutions

Solved – Mantel test vs. Pearson’s correlation coefficient

If you have $n$ objects you will have $\frac{n(n-1)}2$ distances. So, unless $n<4$, you always have more distances than objects. This artificially inflates the number of independent samples when using Pearson's correlation.

Solved – Relationship between the phi, Matthews and Pearson correlation coefficients

Yes, they are the same. The Matthews correlation coefficient is just a particular application of the Pearson correlation coefficient to a confusion table.

A contingency table is just a summary of underlying data. You can convert it back from the counts shown in the contingency table to one row per observations.

Consider the example confusion matrix used in the Wikipedia article with 5 true positives, 17 true negatives, 2 false positives and 3 false negatives

> matrix(c(5,3,2,17), nrow=2, byrow=TRUE)
     [,1] [,2]
[1,]    5    3
[2,]    2   17
> 
> # Matthews correlation coefficient directly from the Wikipedia formula
> (5*17-3*2) / sqrt((5+3)*(5+2)*(17+3)*(17+2))
[1] 0.5415534
> 
> 
> # Convert this into a long form binary variable and find the correlation coefficient
> conf.m <- data.frame(
+ X1=rep(c(0,1,0,1), c(5,3,2,17)),
+ X2=rep(c(0,0,1,1), c(5,3,2,17)))
> conf.m # what does that look like?
   X1 X2
1   0  0
2   0  0
3   0  0
4   0  0
5   0  0
6   1  0
7   1  0
8   1  0
9   0  1
10  0  1
11  1  1
12  1  1
13  1  1
14  1  1
15  1  1
16  1  1
17  1  1
18  1  1
19  1  1
20  1  1
21  1  1
22  1  1
23  1  1
24  1  1
25  1  1
26  1  1
27  1  1
> cor(conf.m)
          X1        X2
X1 1.0000000 0.5415534
X2 0.5415534 1.0000000

Best Answer

Related Solutions

Solved – Mantel test vs. Pearson’s correlation coefficient

Solved – Relationship between the phi, Matthews and Pearson correlation coefficients

Related Question