Solved – When we should NOT use k-fold cross validation to assess the predictor

cross-validationmachine learningvalidation

Does anybody know in which cases –of learning and predicting–, it is better to use "validation test" or something else, instead of "k-fold cross validation" to assess the performance of the algorithm in predicting the target classes?
Here is my case and problem: Yes, I know that if the data is less, CV works well, but this case –with limited samples–
is a bit different. Because the order of the data as the test set is important
for us, so that we can not apply CV, since CV gives you the average performance
of all folds. More in detail: I have the
following data set with 4 features and 10 samples;

            f1,f2,f3,f4,class
       S1  
       S2
       S3
       S4
       S6
       S7
       S8
       S9
       S10

Every sample referees to an activity that accrued in a specific date (E.g, S1 logged in the first week,
S2 logged in the second week, and so on).
It means an activity that logged in sample (S1) is much different from the last sample (S10),
on the other-hand S1 and S2 or S3 are not much different. So in this case, I used S1 to S8 as the training set,
and S9 and S10 as the test set to validate the classification performance. Since, the samples
and values change over time, I could not use CV to validate the performance. So my questions is, can I rely on this
statement and my way to design the training and test is reliable?

Best Answer

  • single test set vs. cross validation has been discussed before.

  • As for your data, you essentially have a time series. This leads to dependence along your data. Thus, a single random split or a cross validation will not achieve independence between training and test sets.

    Dealing with that will depend on how your model will be used.

    • in case you'd train on your S1-S10 data, and then start predicting S11,S12, and so on: look into sliding windows cross validation for time series. For these models, you can actually use the temporal correlation to help with your prediction.
    • If you do not want or cannot use that temporal correlation, you may be able to obtain test data after finding out the temporal length of correlation and obtain data that is far enough away in time from all your training data so that training and test data are not correlated any more.
Related Question