My method of cross validation is to first split my sample into two sub-samples, with 80% respectively 20% of the observations, and then to correlate the predicted values of my model (created with the bigger sample) with the actually observed values (of the smaller sample).
This correlation can be tested for significance and here is my question about that process: In order for the model to make sense, the correlation should be greater than 0. If the correlation is negative (and significant), my model makes exactly the opposite predictions it should do and is therefore not useful. Thus, is it in the end not already enough to only have a one tailed test if the correlation is greater 0?
Best Answer
You should use a two sided test.
A one sided test assumes that the "other side" is impossible. But you yourself said that it's possible. Undesirable and improbable, to be sure, but not impossible.
In addition, one-sided p-values are known to lead to extremely overconfident inferences (see page 3).