A newbie question here. I am currently performing a nonparametric regression using the np package in R. I have 7 features and using a brute force approach I identified the best 3. But, soon I will have many more than 7 features!
My question is what are the current best methods for feature selection for nonparametric regression. And which if any packages implement the methods. Thank you.
Best Answer
Unless identification of the most relevant variables is a key aim of the analysis, it is often better not to do any feature selection at all and use regularisation to prevent over-fitting. Feature selection is a tricky procedure and it is all too easy to over-fit the feature selection criterion as there are many degrees of freedom. LASSO and elastic net are a good compromise, the achieve sparsity via regularisation rather than via direct feature selection, so they are less prone to that particular form of over-fitting.