I have a binary supervised classification problem with about 62 features, by eye about 30 of them could have reasonable discriminating power. Depending on the situation I have between 12,000 and 2,000 samples ( I consider a number of cases but the features are the same for all ).I am using sklearn and the MLP does not have a dedicated feature selection tool like decision trees do. My question is what is the recommended way to preform feature selection here? I have read in the sklearn documentation that LDA should not be performed in a binary classification problem and PCA is under the unsupervised methods on the sklearn website.
Does anyone have any experience with this that could suggest a method?
Edit: Added number of samples
Best Answer
Most probably, you do not need dimensionality reduction.
People do dimensionality reduction if the problem is intractable, but with 62 features, it is not the case.
People also sometimes reduce dimensionality because the number of observations is too small in comparison with the number of features. But if your sample is small, using neural network is a bad idea anyway - use logistic reression or SVM instead, as it is more robust.
Dimensionality reduction and feature selection are also sometimes done to make your model more stable. But you can stabilize it by adding regularization (parameter
alphain theMLPClassifier).Dimensionality reduction and feature selection lead to loss of information which may be useful for classification. So if you don't have a very serious reason for this, do not use PCA or LDA fith MLP.