What is the difference between mixture and hierarchical models?
Are they of the same nature with different names or they are totally different things?
If there are any references, I will be happy to know.
Solved – the difference between a mixture model and a hierarchical model
mixture-distributionmodelmultilevel-analysis
Related Solutions
KL divergence would be natural because you have a natural base distribution, the single Gaussian, from which your mixture diverges. On the the other hand, the KL divergence (or its symmetrical 'distance' form) between two Gaussian mixtures, of which your problem is a special case, seems to be intractable in general. Hershey and Olson (2007) looks like a reasonable summary of the available approximations, including variational methods that may possibly offer easier bounds.
However, if you want to have an argument about the ill-effects of assuming something is Gaussian when it's really a mixture then it's best to have a good idea about the consequences you're actually interested in - something more specific than simply 'being wrong' (this is @Michael-Chernick's point). For example, the consequences for a test, or an interval, or somesuch. Two obvious effects of the mixture are overdispersion, which is pretty much guaranteed, and multimodality, which will confuse maximizers.
Similarity
Fundamentally both types of algorithms were developed to answer one general question in machine learning applications:
Given predictors (factors) $x_1, x_2, \ldots, x_p$ - how to incorporate the interactions between this factors in order to increase the performance?
One way is to simply introduce new predictors: $x_{p+1} = x_1x_2, x_{p+2} = x_1x_3, \ldots$ But this proves to be bad idea due to huge number of parameters and very specific type of interactions.
Both Multilevel modelling and Deep Learning algorithms answer this question by introducing much smarter model of interactions. And from this point of view they are very similar.
Difference
Now let me try to give my understanding on what is the great conceptual difference between them. In order to give some explanation, let's see the assumptions that we make in each of the models:
Multilevel modelling:$^1$ layers that reflect the data structure can be represented as a Bayesian Hierarchical Network. This network is fixed and usually comes from domain applications.
Deep Learning:$^2$ the data were generated by the interactions of many factors. The structure of interactions is not known, but can be represented as a layered factorisation: higher-level interactions are obtained by transforming lower-level representations.
The fundamental difference comes from the phrase "the structure of interactions is not known" in Deep Learning. We can assume some priors on the type of interaction, but yet the algorithm defines all the interactions during the learning procedure. On the other hand, we have to define the structure of interactions for Multilevel modelling (we learn only vary the parameters of the model afterwards).
Examples
For example, let's assume we are given three factors $x_1, x_2, x_3$ and we define $\{x_1\}$ and $\{x_2, x_3\}$ as different layers.
In the Multilevel modelling regression, for example, we will get the interactions $x_1 x_2$ and $x_1 x_3$, but we will never get the interaction $x_2 x_3$. Of course, partly the results will be affected by the correlation of the errors, but this is not that important for the example.
In Deep learning, for example in multilayered Restricted Boltzmann machines (RBM) with two hidden layers and linear activation function, we will have all the possible polinomial interactions with the degree less or equal than three.
Common advantages and disadvantages
Multilevel modelling
(-) need to define the structure of interactions
(+) results are usually easier to interpret
(+) can apply statistics methods (evaluate confidence intervals, check hypotheses)
Deep learning
(-) requires huge amount of data to train (and time for training as well)
(-) results are usually impossible to interpret (provided as a black box)
(+) no expert knowledge required
(+) once well-trained, usually outperforms most other general methods (not application specific)
Hope it will help!
Best Answer
Terminology is not as standardized as one may whish, so what I understand under these terms may not be what others understand under these terms.
I understand under a mixture model a model that posits that posits a mixture of "brand name distributions" for the dependent variable. For example a discrete mixture of normals assumes that a person is drawn with a to be estimated probability from one normal distribution and with another probability form another normal distribution, etc., for a given number of groups. Notice that in this model we do not know which person belongs to which group
Under a hierarchical model I understand that we have observations that are nested in groups. For example: students are nested in classrooms, which are nested in schools, which are nested in countries. Here membership of the groups is known and part of the data.