I watched the lecture about Lasso and at the end of this module (between 00:40 and 01:25) she explains how to choose the regularization parameter Lambda and it sounds like using (K-fold)Cross Validation technique is not a good option for Lasso. But, I don't understand why? What's the problem?
Solved – Why using cross validation is not a good option for Lasso regression
cross-validationlassomachine learningregression
Related Solutions
Basically you select whichever $\alpha$ gives you the lowest error rate (on a validation set). So to be complete cross-validation entails the following steps:
- Split your data in three parts: training, validation and test.
- Train a model with a given $\alpha$ on the train-set and test it on the validation-set and repeat this for the full range of possible $\alpha$ values in your grid.
- Pick the best $\alpha$ value (i.e. the one that gives the lowest error)
- Once you have complete this, retrain a new model using this optimal value of $\alpha$ on (trainset+validationset).
- You can now evaluate your model on the test-set.
I haven't gone over your code, but your graph suggests to also look for even lower values of $\alpha$. Note, however, that learning curves usually look something like this when evaluated on the validation set:
That is, your error should be high in the beginning, drop to a low and then go somewhat up again.
The comment of @frank-harrell relates to the fact that you should probably repeat this experiment a few times to get robust estimates of your $\alpha$ values. For this you can also use k-fold cross-validation like you did so you should be fine.
The short answer is, its up to you, depending on your interest. In the past I have used AIC for Lasso.
However it sounds like you are using this model for prediction, and thus using the mis-classification rate is a good idea. However misclassification can be categorized in many ways. Are you interested in the the absolute % classified correctly? Or maybe you just care about of those classified as 1 (or yes, etc), how many of those were classified correctly? I would do some reading into Positive Predictive values, Negative predictive values, etc.
https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values
In addition when doing your cross validation, there are a plethora of criteria you could use to validate your model. A short list of other common criterion are:
- $R^2$
- $MSE$
- $Mallow’s$ $C_p$
- $AIC$
Look them up and see which is most relevant to you!
Best Answer
So, the point is that when you define an optimal value of $\lambda$ you must ask optimal for what? In the case of the LASSO, there are two possible goals:
Estimate $\lambda_{\text{pred}}$, the value of $\lambda$ that leads to the best prediction error.
Estimate $\lambda_{\text{ms}}$, the value of $\lambda$ that produces the correct model (or at least something that is close to it).
As Dr. Fox correctly notes, in general it is not the case that $\lambda_{\text{pred}} = \lambda_{\text{ms}}$, and typically $\lambda_{\text{pred}} < \lambda_{\text{ms}}$. But choosing $\lambda$ by cross-validation is using prediction error, and hence one would expect it to estimate $\lambda_{\text{pred}}$. Consequently, if you choose $\lambda$ by cross validation, you may select a $\lambda$ which leads to a model which is too big. If your goal is recovery of the true model, it follows that one should be careful applying cross validation.
I personally encounter this issue a lot when writing papers whenever I do a simulation study looking at the lasso for variable selection. Invariably, using cross-validation to select $\lambda$ is a disaster. I have had much better luck applying Lasso$(\lambda)$ to select the model and then fitting by least squares, then applying cross-validation to this entire procedure to select $\lambda$. It's still not ideal, but it is a big improvement.
That's not to say that cross-validation is completely off the table for model selection, it's just that you need to think carefully about what $\lambda$ your method is estimating. For example, lets consider ignoring the lasso and just think of a low-dimensional linear regression. In this case, leave-one-out cross validation is known to be more or less equivalent to some variant of AIC, and AIC is well-known to be inconsistent for model selection. Similarly, BIC is generally associated with leave-$V$-out cross validation where $V$ is some function of the size of the data, and it is well-known that variants of BIC are model selection consistent. Hence, there is some way of doing cross-validation that we would expect to be consistent for model selection, but leave-one-out is not.