Autoencoders in Machine Learning – Dimensionality Reduction Tools

autoencodersdimensionality reductionmachine learning

I'm trying to get a basic understanding of AutoEncoders. Basically they are neural networks with a very low representation of the original input in some hidden layer, and then a final layer which has exact same dimension as the input but 'reconstructed' from the latent lower-dimensional representation. What I don't understand is what should we do with autoencoders, apart from denoising or compression. For example, would it be possible to use the latent representation as a dimensionality-reducted version of the original input, and then using it to train some Machine Learning Model?

Many thanks,

James

Best Answer

Yes, dimension reduction is one way to use auto-encoders. Consider a feed-forward fully-connected auto-encoder with and input layer, 1 hidden layer with $k$ units, 1 output layer and all linear activation functions. The latent space of this auto-encoder spans the first $k$ principle components of the original data. This can be useful if you want to represent the input in fewer features, but don't necessarily care about the orthogonality constraint in PCA. (More information: What're the differences between PCA and autoencoder?)

But auto-encoders allow a number of variations on this basic theme, giving you more options for how the latent space should be constructed than does PCA.

  • Using CNN layers instead of FFNs is clearly a different kind of model compared to PCA, so it will encode a different kind of information in the latent space.
  • Using nonlinear activation functions will also yield a different kind of latent encoding than PCA (because PCA is linear).
  • Likewise sparse, contractive or variational auto-encoders have different goals than PCA and will give different results, which can be helpful depending on what problem you're trying to solve.
Related Question