Solved – no ‘ information rate ‘ algorithm

confusion matrixmachine learningmodelpandaspython

I plan to implement ' no information rate ' as part of summary statistics. This statistic is implemented in r (Optimise SVM to avoid false-negative in binary classification) but not in Python (at least I cannot find a reference) .

Is there a canonical reference that I can refer to so as to implement this algorithm ?

I've searched Wikipedia and various Google searches but have not found a reference.

Update :

Reading caret doc https://cran.r-project.org/web/packages/caret/caret.pdf

"
The overall accuracy rate is computed along with a 95 percent confidence interval for this rate
(using binom.test) and a one-sided test to see if the accuracy is better than the "no information
rate," which is taken to be the largest class percentage in the data.
"

Best Answer

Suppose that you have response $y_i$ and covariates $x_i$ for $i = 1 ...n$, and some loss function $\mathcal{L}$. The no information error rate of a model $f$ is the average loss of $f$ over all combinations of $y_i$ and $x_i$:

$${1 \over n^2} \sum_{i=1}^n \sum_{j=1}^n \mathcal{L}\left(y_i, f(x_j)\right)$$

If you have a vector of predictions predicted and a vector of responses response, you can calculate the no info error rate by generating all the combinations of predicted and response and then evaluating some function loss on these resulting vectors.

In R, supposing RMSE loss, (using the tidyr library) this looks like:

predicted <- 1:3
response <- 4:6
loss <- function(x, y) sqrt(mean((x - y)^2))

combos <- tidyr::crossing(predicted, response)
loss(combos$predicted, combos$response)

In Python this looks like

import numpy as np

predicted = np.arange(1, 4)
response = np.arange(4, 7)

combos = np.array(np.meshgrid(predicted, response)).reshape(2, -1)

def loss(x, y):
    return np.sqrt(np.mean((x - y) ** 2))

loss(combos[0], combos[1])
Related Question