I fail to see why you are trying to use per-label accuracy for the calculation of the overall accuracy. For any problem, you can compute the overall accuracy as:
$$Acc =\sum \frac{\#CorrectPredictions}{\#TotalNumberofPredictions}$$
Your method for computing per label accuracy is not entirely correct. When you traverse a particular row in a confusion matrix, then you can only get the True Positive values and the False Negative values. Let's consider the general case of binary classification, if the confusion matrix is something like the following:
PREDICTED
____________
T | | 1 0 |
R |====|======|
U | 1 | 2 1 |
E | 0 | 1 3 |
============
Here we are just labeling $1$ as Positive and $0$ as Negative. Just consider your method of calculating per-label accuracy here which in essence should be equivalent to calculating the overall accuracy. If you traverse along a particular row, then the values for label $1$ should be: $TP:2, TN : ?, FP : ?, FN : 1$. If you traverse across the column then you can get the value of FP ($1$ in this case). But even here you can easily see that the overall accuracy is just the ratio of the sum of diagonals and the sum of the matrix. The True Negative here is just the diagonal value for label $0$.
You are probably mistaken in some idea about the correctness of a prediction. The formula $\frac{(TP + TN)}{ TP+TN+FN+FP}$ holds for binary classification. The $TN$ there is just the $TP$ for label $0$. When you generalize over multiclass classification then the diagonal value corresponds to True positive for the column label and row label (which are the same). All other values in a specific row correspond to False Negative values for the row label and similarly, all other values in the column correspond to False Positive values for the column label. For example for label $2$:
PREDICTED
_____________________
| | 1 2 9 11 |
|====|================|
T | 1 | . FP . . |
R | 2 | FN TP FN FN |
U | 9 | . FP . . |
E | 11 | . FP . . |
=====================
The problem with computing the overall accuracy based on the per-label accuracy computed by your method is to distinguish between the contributions of TP and TN values.
Let's first establish here that the overall accuracy is the ratio of the diagonal values and the overall sum of the matrices, which is to say that the numerator contains the count of True Positives for each label and the denominator is the count of total predictions. The problem with deriving the overall accuracy from per-label accuracy is to differentiate the effects of the TN counts in each per-label accuracy. For instance, if we assign confusion_mat[2,2] as $0$, then the overall accuracy is clearly $0$ but the per-label accuracy is still non-negative for each label due to the contributions of TN values. What is more important to note is that the per-label accuracy of label $9$ still remains $1.00$, although the per-label accuracy of every other label decrease due to a decrement in their TN value. What this shows is that in some cases the per-label accuracy could be entirely due to the contribution of TN values. What this implies is that when you are given only the per-label accuracies, then there is no additive weighted combination from which you can derive the overall accuracy since you lack any information whatsover as to how much of each per-label accuracy is based on the TP value. The per-label accuracies are likely to overestimate the actual overall accuracy due to the additional contribution of TN values.
The reason why TP+TN works in case of binary classification is that the TN value is just another diagonal value corresponding to the TP of the other label but for multi-class case here, you are considering non-diagonal values in your TN for a particular label (thus considering them as positive or correct predictions).
Best Answer
Yeah, that's wrong.
FN = false negative = 10
FP = false positive = 0
TP = true positive = 0
TN = true negative = 9990
accuracy = $\frac{9990 + 0}{10 + 9990 + 0 + 0}$ = 99.9%