Solved – Confusion matrix for events in a time series

confusion matrixtime series

I'm not big on statistics, so please excuse my ignorance.

I have a video recording that I want to evaluate, I have an algorithm that can transform this video into a time series where I have 0 everywhere except for a couple of frames where an event (A) occurs.

Then I have manual annotations of another event (B) that I think is related to the 1st event (A) (i.e. event B appears shortly after event A).

I want to construct a confusion matrix like this:

Event A is condition, event B is "test"
True positive value is when there is an event A and within 50 frames there is also event B
False positive is when A is not present, but B is
False negative is when A is present, but B is not

Now my problem is about True Negatives. If you do vaccine testing, you have a total number of tests, and you can quantify True Negatives without issue. But what about my case? I either have event A or B but True Negative is by definition everything else in the time series?

Does it make sense to even use confusion matrix?

Best Answer

Yes, the default value of 'no event found' would be a True Negative, just like any other "needle in a haystack" classification problem.

For binary classification usually you wouldn't call it a confusion matrix, the Precision and Recall terminology would be more clear.

Related Solutions

Solved – How to find TP,TN, FP and FN values from 8×8 Confusion Matrix

True Positive, True Negative, False Positive, and False Negative are defined for binary classification problems (in which one outcome is defined as "positive" and the other "negative").

So to use this terminology as a measure of success for a larger confusion matrix, you must break the table down into a series of binary classification sub-tables.

For example, we could calculate the TP, TN, FP, and FN for class A. This breaks down as follows

TP: True A's that are correctly classified as A
TN: All other classes correctly classified as not A
FP: Other classes incorrectly classified as A
FN: A's that were incorrectly classified as not A

This could then be repeated for all classes to get a TP, TN, FP, and FN conditional on each actual class.

Alternatively, you could use a measure calculated on the table as a whole such as overall accuracy. More sophisticated measures can also be calculated to evaluate purity of the classification. This Stack Overflow link provides a good start for investigating some of these methods: how to calculate classification error rate

Solved – Find true negatives in a confusion matrix

I'm one of the PyCM developers.

Your precision calculation method is completely correct.
For recall calculation you should consider improperly classified items in each row :

class 0 :

recall =  TP*100/(TP+FN)
recall =  66/(66+1+0)
recall =  98.5074

class 1 :

recall =  TP*100/(TP+FN)
recall =  81/(81+2+3)
recall =  94.1860

class 2 :

recall =  TP*100/(TP+FN)
recall =  56/(56+0+1)
recall =  98.2456

Accuracy formula is correct.
Finding TN for each class :

You should consider class vs other and add up items that classified correctly as other, in other words eliminate row and col related to class and add up remaining.