I have a classification process which I test using leave-one-out.
Precisely, how do I compute the standard deviation of the result?
If I compute it normally, I get a very large deviation which makes sense because each test is either 100% or 0%.
What is the accepted convention in this case?
Thanks.
Best Answer
In fact you can't, unless you make some simple significance test to check if it is better than random guessing.
Anyway, you can still use stochastic or N-fold CV to get more continuous error scale; or, supposing that your algorithm is stochastic, just aggregate several LOO results with different random seeds.