Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample testing?
Doing an extra out-of-sample testing would basically equal to getting an additional test set and applying my classifier to that again, wouldn't it?
Please explain if and how my interpretation is wrong.
PS: If relevant to your answer, you can assume that approximately 60% was used as training set, 20% as validation set and the last 20 as test set.
Best Answer
Out-of-time test means that your test set can't go back in time with respect to the training/validation sets. I.e. training should be earlier in time than testing. So it is not necessarily the same as out-of-sample, which can contain any observations, not used in training. I think you would not need an extra test set if your current test set obeys the time order.