I understand Cross Validation is used to parameter tuning and finding the machine learning model that will generalize well on the test data/
Leave one out cross validation: One data point is left out (validation) and trained on rest of the data points. This is done iteratively and then results are averaged out. But how do we find the best model when the results are average out ?
K-Fold we say K = 5 , 4 fold data is trained on and 1 fold data is validated and the results again are average out. Again how do we know which is the best model here ? Or do we choose a hyper parameter and do K-Fold here to see best hyper parameter in the model ?
Lastly, what is the difference between grid search CV and K-Fold ?
Best Answer
Cross validation is used to either compare different models or to tune the parameter. For the case of the parameter tuning, you can consider the models with different parameters to be unique models that you want to test. For both cases, the $MSE_i$ are averaged out for each model, and then compared to $CV$ parameter in different models. These $CV$ parameters cannot be used in absolute sense, they only make sense when compared to $CV$ parameters from the same data, same kind of cross validation, and different model (think of this like you would think of AIC and BIC). Once you have the $CV$ parameters from different models, you can choose the appropriate model (or parameter) by picking the smallest number (smaller the average $MSE$ the better).
Grid search cross validation is usually employed for parameter tuning, while K-fold is more general approach. Grid-search is a way to select the best of a family of models, parametrized by a grid of parameters.