Solved – LDA for dimensionality reduction usage

classificationdimensionality reductiondiscriminant analysissvm

I have a original dataset with 70 samples, each sample with 96 features. The samples are labeled as 0 or 1. So I use linear discriminant analysis (LDA) to reduce the dimensionality of all the dataset, generating a samples with only one feature.

My results with all 96 features is 83% of accuraccy, with the projeted samples i have a 100% of accuracy. I'm using a svm for classification, with a split of 80% for training and 20% for test.

So my questions is : LDA is known as a supervised method of classification, but often used as a dimensionality reduction technique. The usage of LDA in all samples is doing a pre-training in the data? If yes, did I have to split the data before use LDA for reduction and later use the transformation to project the test data?

Best Answer

LDA used as a dimensionality-reducing technique can be seen as a "supervised PCA", so it will redistribute your data in a new space (of lesser dimension) where classes should be better separated (based on the labels you provided).

The projection matrix is made of the first eigen vectors (of positive eigen values) given by LDA to project your test data into that new feature space, then input that vector into your SVM.

Note that you should use a non-linear kernel in your SVM (e.g. RBF), otherwise you'll have a linear transformation on top of another linear transformation, which will not improve discrimination. SVM and LDA are pretty much equivalent when it comes to linear classification.