I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score (where if this similarity score is below a certain threshold, then the user is authenticated). So, I have 72 trials total (36 trials containing a positive case and 36 that contain a negative case). What I want to do is graph the ability of this system to authenticate people by illustrating it with a ROC graph.
Unfortunately, I don't quite understand how to choose a threshold. Is there some mathematical procedure involved for choosing a threshold for the similarity scores? Do I just choose a bunch of different thresholds, and graph the corresponding ROC curves for all these different threshold values? The resulting similarity scores vary from [0.6,1.2] where the positive cases tend to lie around 0.6. All my coding is being done in Matlab.
Best Answer
Generally, the cut-off value is chosen such as to maximize the compromise between sensitivity (Se) and specificity (Sp). You can generate a regular sequence of thresholds and plot the resulting ROC curve, as shown below, based on the DiagnosisMed R package.
Actually, the raw data looks like
and the optimal threshold is found as
To sum up, I would suggest to generate a regular sequence of possible thresholds and compute Se and Sp in each case; then, choose the one that maximize Se and (1-Sp) (or use other criteria if you want to minimize FP or FN rates).