Does the use of "variational" always refer to optimization via variational inference?
Examples:
- "Variational auto-encoder"
- "Variational Bayesian methods"
- "Variational renormalization group"
Solved – What does “variational” mean
inferencemachine learningoptimization
Related Solutions
For some of these parameters you can simply reason about what the range should be. For example, since the sparsity parameter is the desired average activation of the hidden units, this will only make sense if it is in $(0, 1)$, assuming you're using a sigmoid activation. However, you can still narrow this down farther as a sparsity $> 0.5$ is somewhat nonsensical. For other parameters your can look for the values people commonly report in the literature, use this as a rough guess, and expand your ranges if you find it necessary. For example I would probably start with $(0, 0.2)$ for the learning rate.
Another useful place to look would be Geoff Hinton's practical guide to training restricted Boltzmann machines. I imagine most of this advice would be applicable for autoencoders as well.
I do not see why you cannot use EM for LDA. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)=p(z|x,\theta,\phi)$ ($z$ is the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.
Of course, if you use less-than-one hyperparameters for the Dirichlet distributions, then you cannot use EM because in the M-step the expected log likelihood would contain Dirichlet over $\theta$ and $\phi$ that may have multiple modes (i.e., some of the parameters of the Dirichlet may be less than 1, leading to infinite probability density at the corners/edges of the simplex, as shown below).
Best Answer
It means using variational inference (at least for the first two).
In short, it's an method to approximate maximum likelihood when the probability density is complicated (and thus MLE is hard).
It uses Evidence Lower Bound (ELBO) as a proxy to ML:
$log(p(x)) \geq \mathbb{E}_q[log(p, Z)] - \mathbb{E}_q[log(q(Z))]$
Where $q$ is simpler distribution on hidden variables (denoted by $Z$) - for example variational autoencoders use normal distribution on encoder's output.
The name 'variational' comes most likely from the fact that it searches for distribution $q$ that optimizes ELBO, and this setup is kind of like in calculus of variations, a field that studies optimization over functions (for example, problems like: given a family of curves in 2D between two points, find one with smallest length).
There's a nice tutorial on variational inference by David Blei that you can check out if you want more concrete description.
EDIT:
Actually what I described is one type of VI: in general you could use different divergence (the one I described corresponds to using KL divergence $KL(q, p)$). For details see this paper, section 5.2 (VI with alternative divergences).