Solved – How to incorporate PCA step into SVM classification

classificationdimensionality reductionMATLABpcasvm

I've been using SVM to classify a data set without applying PCA. The classification rate was not bad, but I thought maybe applying PCA increases performance.

I have a training set (without labels) with size of 700×60 (i.e. 700 feature vectors each comprises of 60 different features). I applied PCA to that matrix in Matlab, getting a 60×60 matrix.:

coeff = pca(trainVector);

But I am confused about what to do next. How should I utilize this information?

I applied PCA to only training set and have 60×60 PCA matrix only. What will be my new training set after PCA and what should I do with the test set? Something like transformation matrix? I wasn't doing something like cross validation. As I know, SVM algorithm already applies cross validation. Previously I used a SVM tool to get a prediction model and then use this Model and the SVM tool to classify test set. I wanted to improve the accuracy by using PCA before classification.

Best Answer

The purpose of PCA is to reduce the dimensionality of your data, in part to reduce the number of parameters and therefore the variance of your predictor. I think you are applying PCA to the transpose of the matrix you want to apply it to, because your result should be $700 \times n$ for some $n$ that you choose. I am not familiar with the matlab implementation. That being said what you should be doing with PCA and SVM is,

  1. Normalize Data: Center and scale your data. The transformation, $C$, that centers and scales your data should be stored in some fashion.

  2. "Train" PCA: PCA learns a variance maximizing linear transformation onto a lower dimensional space. Learn this transformation, and then apply it to your training data. This transformation, $B$, should be stored in some fashion.

  3. "Train" SVM: As you were already doing train SVM on your reduced data. The predictor $A$ should be stored in some fashion.

Classifying new data point, $x$, is then as easy as $A \circ B \circ C(x)$.

Related Question