distributionsgaussian mixture distributionmachine learningmathematical-statisticsmixture-distribution
We often study Gaussian Mixture model as a useful model in machine learning and its applications.
What is the physical significance of this "Mixture"?
Is it used because a Gaussian Mixture Model models the probability of a number of random variables each with its own value of mean? If not, then what is the correct interpretation of this word.
A mixture of Gaussians is defined as a linear combination of multiple Gaussian distributions. Thus it has multiple modes. The dimension refers to the data (e.g. the color, length, width, height and material of a shoe) while the number of components refers to the model. Each Gaussian in your mixture is one component. Thus each component will correspond to one mode, in most of the cases.
I suggest you read up on mixture models on wikipedia.
If we want to fit a Gaussian to a single data point using maximum likelihood, we will get a very spiky Gaussian that "collapses" to that point. The variance is zero when there's only one point, which in the multi-variate Gaussian case, leads to a singular covariance matrix, so it's called the singularity problem.
When the variance gets to zero, the likelihood of the Gaussian component (formula 9.15) goes to infinity and the model becomes overfitted. This doesn't occur when we fit only one Gaussian to a number of points since the variance can not be zero. But it can happen when we have a mixture of Gaussians, as illustrated on the same page of PRML.
Update:
The book suggests two methods for addressing the singularity problem, which are
1) resetting the mean and variance when singularity occurs
2) using MAP instead of MLE by adding a prior.
Best Answer
A mixture distribution combines different component distributions with weights that typically sum to one (or can be renormalized). A gaussian-mixture is the special case where the components are Gaussians.
For instance, here is a mixture of 25% $N(-2,1)$ and 75% $N(2,1)$, which you could call "one part $N(-2,1)$ and three parts $N(2,1)$":
Essentially, it's like a recipe. Play around a little with the weights, the means and the variances to see what happens, or look at the two tags on CV.