autoencodersdimensionality reductionvariational-bayes
I know that VAE is generative model. However it is also used as a dimensionality reduction method. In this case, what are advantages of VAE??
Also I saw that well-applied vae on mnist, and it was more adjusted to mnist data than SAE(Stacked Auto Encoder). In this case, what makes it more effective on mnist data than SAE?
According to the manifold hypothesis, the data is assumed to lie on a low-dimensional manifold, the implication being that the residual is noise, so if you do your dimensionality reduction correctly, you should improve performance by modeling the signal rather than the noise. It's not just a question of space and complexity.
For image classification task, how can a stacked auto-encoder help an traditional Convolutional Neural Network?
As mentioned in the paper, we can use the pre-trained weights to initialize CNN layers, although that essentially doesn't add anything to the CNN, it normally helps setting a good starting point for training (especially when there's insufficient amount of labeled data).
any pre-trained step before first convolution operation like Dimensionally Reduction or AutoEncoder output can be used as input image instead of real image data in CNN
Becaues of CNN's local connectivity, if the topology of data is lost after dimensionality reduction, then CNNs would no longer be appropriate.
For example, suppose our data are images, if we see each pixel as a dimension, and use PCA to do dimensionality reduction, then the new representation of a image will be a vector and no longer preserves the original 2D topology (and correlation between adjacent pixels). So in this case it can not be used directly with 2D CNNs (there are ways to recover the topology though).
Using the AutoEncoder output should work well with CNNs, as it can be seen as adding an additional layer (with fixed parameters) between the CNN and the input.
how much it affects the performance of Convolution Neural Network in context of image classification tasks
I happened to have done a related project at college, where I tried to label each part of an image as road, sky or else. Although the results are far from satisfactory, it might give some ideas about how those pre-processing techniques affects the performance.
(1) image of a clear road (2) outcome of a simple two-layer CNN
(3) CNN with first layer initialized by pre-trained CAE (4) CNN with ZCA whitening
The CNNs are trained using SGD with fixed learning rates. Tested on the KITTI road category data set, the error rate of method (2) is around 14%, and the error rates of method (3) and (4) are around 12%.
Please correct me where I'm wrong. :)
Best Answer
I think we have to further break this question in order to approach its solution.
First, I think the prime comparison is between AE and VAE, given that both can be applied for dimensionality reduction. The advantage of VAE, in this case, is clearly answered here. The main point is in addition to the abilities of an AE, VAE has more parameters to tune that gives significant control over how we want to model our latent distribution.
Second, what do you mean by "Saw well-applied VAE on mnist", if it means you observed the resulting images and concluding that the images are of better quality in case of VAE w.r.t to SAE. I think you should also check the MSE loss values between the input and output of the network.
Third, SAE is a version of AE itself where the hidden layers of AE are trained by unsupervised pre-training. So on a higher level, it helps you to find an optimal solution to AE. On the other hand, VAE may further enhance the solution depending on its settings.