Solved – Variational Auto-encoders vs Restricted Boltzmann Machines

Question

What are the differences of modeling ability between Variational Auto-encoders (VAEs) and Restricted Boltzmann Machines (RBMs)?

What I am interested in is to know about the unsupervised learning power differences.
Where exactly VAEs work better than RBMs and vise versa?
Also what RBMs can capture that VAEs can not?

I am already aware of this post and I know the concept of VAEs and RBMs. Also this one is very brief and is not what I am looking for.

Does anybody know about any (theoretical) limitations on the modeling power for one that makes another superior?

Answer 1

Theoretically RBMs are undirected models with no connections between any observed variables or any latent variables, VAEs are directed models with continuous latent variables.

The Deep Learning book (Chapter 20 Deep Generative Models) provides a very good summary of pros and cons of VAEs and all types of RBMs.

To quote some

The VAE framework is very straightforward to extend to a wide range of model architectures. This is a key advantage over Boltzmann machines, which require extremely careful model design to maintain tractability.

The variational autoencoderis defined for arbitrary computational graphs, which makes it applicable to a wider range of probabilistic model families because there is no need to restrict the choice of models to those with tractable mean field fixed point equations.

One very nice property of the variational autoencoder is that simultaneously training a parametric encoder in combination with the generator network forces the model to learn a predictable coordinate system that the encoder can capture. This makes it an excellent manifold learning algorithm.