Gaussian Mixture Models – Different Covariance Types

covariance-matrixgaussian mixture distribution

While trying Gaussian Mixture Models here, I found these 4 types of covariances.

'full' (each component has its own general covariance matrix),
'tied' (all components share the same general covariance matrix),
'diag' (each component has its own diagonal covariance matrix),
'spherical' (each component has its own single variance).

I googled a lot to find more details about each of these types but found very high-level descriptions (such as this) only.

Appreciate if someone can help me understand these, or at least direct me to somewhere I can read about these.

Best Answer

A Gaussian distribution is completely determined by its covariance matrix and its mean (a location in space). The covariance matrix of a Gaussian distribution determines the directions and lengths of the axes of its density contours, all of which are ellipsoids.

These four types of mixture models can be illustrated in full generality using the two-dimensional case. In each of these contour plots of the mixture density, two components are located at $(0,0)$ and $(4,5)$ with weights $3/5$ and $2/5$ respectively. The different weights will cause the sets of contours to look slightly different even when the covariance matrices are the same, but the overall shapes of individual contours will still be similar for identical matrices.

Clicking on the image will display a version at higher resolution.

NB These are plots of the actual mixtures, not of the individual components. Because the components are well separated and of comparable weight, the mixture contours closely resemble the component contours (except at low levels where they may distort and merge, as shown in the center of the "tied" plot for instance).

Full means the components may independently adopt any position and shape.
Tied means they have the same shape, but the shape may be anything.
Diagonal means the contour axes are oriented along the coordinate axes, but otherwise the eccentricities may vary between components.
Tied Diagonal is a "tied" situation where the contour axes are oriented along the coordinate axes. (I have added this because initially it was how I misinterpreted "diagonal.")
Spherical is a "diagonal" situation with circular contours (spherical in higher dimensions, whence the name).

This exhibit a gamut from the most general possible mixture to a very specific kind of mixture. Other (fussier) restrictions are possible, especially in higher dimensions where the numbers of parameters grow rapidly. (A covariance matrix in $n$ dimensions is described by $n(n+1)/2$ independent parameters.)

Related Solutions

Solved – Fitting a gaussian mixture model using stochastic gradient descent

Assuming that mus[d] is $\mu_j$, j.sigma is $\Sigma_j$, and G(x)/M(x) indeed computes the posterior probability of component $j$ given the data $x$, $$p(j \mid x) = \frac{\rho_j \mathcal{N}_x(\mu_j, \Sigma_j)}{\sum_k \rho_k \mathcal{N}_x(\mu_k, \Sigma_k)},$$ the gradient itself seems correct to me. But here are some things that I noticed that might help you to find your problem:

I would expect the access to the mean, the covariance, and the calculation of the posterior to all involve either j or d, whichever variable represents the component for which you want to compute the gradient in your code. If you tell us what j and d stand for, we might be able to tell you more.
If G(x)/M(x) accesses j.Sigma to compute the posterior, your code might not compute what you think it does. It might be better to first compute all the gradients of all the parameters, and then perform the update.
Stochastic gradient descent is usually not the first choice to optimize mixtures of Gaussians. Most often, expectation maximization (EM) is used (see, for example, Bishop, 2007). Even if you don't use EM, you might want to consider BFGS or L-BFGS (implemented in scipy.optimize) before using SGD. And even if you stick to SGD, you should consider using multiple data points ("batches") at a time to estimate the gradient, or at least including a momentum term. Briefly looking at Toscano and McMurray's paper, my guess is that they chose to use SGD because they were interested in modeling speech acquisition in a biologically more plausible way, rather than obtaining the best possible fit, and doing this online (that is, one data point at a time). If you don't need this, my advice would be to use EM.

(I just realized you specifically asked for online learning, so the only viable option for you might be to add the momentum term to speed things up a bit.)
The way you chose to compute the gradient is quite inefficient, which will further slow down learning. You might not have seen reasonable results because it takes forever before the algorithm converges to something interesting. Here is a slightly better way to calculate the gradient:
```
sigmaInv = inv(j.sigma)
dSigma = G(x)/M(x) * 0.5 * (-sigmaInv + numpy.sum(sigmaInv.dot(x - mus[d]) * x))
```
There are still ways to further improve the computation of the gradient. For example, we still get a valid ascent (although not a steepest ascent) direction if we multiple the gradient by a positive definite matrix (such as $\Sigma_j$, which would simplify the gradient a bit). It might also work better if we used a different parametrization of the covariance, such as Cholesky factors, and computed the gradients of those instead.

Solved – Gaussian mixture regression in higher dimensions

Your idea is exactly what I would recommend. To enforce the positive definiteness of the covariance matrices, just use the Cholesky decomposition $\Sigma_i = L_iL_i^T$, and minimise the objective with respect to $L_i$ instead of $\Sigma_i$. I refer to this article for more details. Also, you may try to optimise the log-likelihood instead of the mean square error.

Best Answer

Related Solutions

Solved – Fitting a gaussian mixture model using stochastic gradient descent

Solved – Gaussian mixture regression in higher dimensions

Related Question