Solved – Plotting a gaussian mixture with pdf values >> 1 in MATLAB

data visualizationgaussian mixture distributionMATLAB

From a given dataset X, I learn a 7-component gaussian mixture model using matlab's gmdistribution.fit:

model = gmdistribution.fit(X,7,'Start', gmm_init,'Options',options, 'Regularize', 1e-05)

However, if I want to create a contour-plot using matlab's ezcontour, the result is that a lot of areas are actually blanked out in the plot (matlab's plot data cursor tells me these entries are "NaN"):
enter image description here

If I look at the pdf of my model evaluated at X's data points, I get that a lot of pdf values are >> 1:

 >>PDF = pdf(model, X);
 >>max(PDF)

 ans =

    2.806149789376303e+13

 >> mean(PDF)

 ans =

    6.587701947551628e+12

From other questions I have learned that in general, a continuous PDF may indeed be larger than 1. However, since in my case the average pdf value is some orders of magnitude higher than the smallest one, it seems that ezcontour has some difficulties plotting it.

My dataset X contains some very dense regions. Also, some mixture components only contain very few components (e.g. ~1% of all data points). In a different post, it is suggested that I look at the (co)variance of my components to get an idea about the density for a given component's assigned data points.

My questions are:

Is the model I get from gmdistribution.fit of an inferior quality because the PDF has some really crazy high values in the dense,"middle" regions of my dataset?
What can I do to plot this gaussian mixture model anyway (e.g. with something else than ezcontour?)

Best Answer

Is the model I get from gmdistribution.fit of an inferior quality because the PDF has some really crazy high values in the dense,"middle" regions of my dataset?

This is tricky to say without looking at your data. Are many many points densely clustered in some central points? maybe you should include a scatter plot of the data. Remember, you're fitting a gaussian mixture, so if a given "cluster" of data is highly kurtotic, it could lead to overfitting.

What can I do to plot this gaussian mixture model anyway (e.g. with something else than ezcontour?)

This is a cheap trick, but just take the log(PDF) and plot that using whatever method you like. The logarithm will sort of compress the range of your pdf. I would recommend the function imagesc which will make a nice looking heatmap.

Related Solutions

Solved – Fitting a gaussian mixture model using stochastic gradient descent

Assuming that mus[d] is $\mu_j$, j.sigma is $\Sigma_j$, and G(x)/M(x) indeed computes the posterior probability of component $j$ given the data $x$, $$p(j \mid x) = \frac{\rho_j \mathcal{N}_x(\mu_j, \Sigma_j)}{\sum_k \rho_k \mathcal{N}_x(\mu_k, \Sigma_k)},$$ the gradient itself seems correct to me. But here are some things that I noticed that might help you to find your problem:

I would expect the access to the mean, the covariance, and the calculation of the posterior to all involve either j or d, whichever variable represents the component for which you want to compute the gradient in your code. If you tell us what j and d stand for, we might be able to tell you more.
If G(x)/M(x) accesses j.Sigma to compute the posterior, your code might not compute what you think it does. It might be better to first compute all the gradients of all the parameters, and then perform the update.
Stochastic gradient descent is usually not the first choice to optimize mixtures of Gaussians. Most often, expectation maximization (EM) is used (see, for example, Bishop, 2007). Even if you don't use EM, you might want to consider BFGS or L-BFGS (implemented in scipy.optimize) before using SGD. And even if you stick to SGD, you should consider using multiple data points ("batches") at a time to estimate the gradient, or at least including a momentum term. Briefly looking at Toscano and McMurray's paper, my guess is that they chose to use SGD because they were interested in modeling speech acquisition in a biologically more plausible way, rather than obtaining the best possible fit, and doing this online (that is, one data point at a time). If you don't need this, my advice would be to use EM.

(I just realized you specifically asked for online learning, so the only viable option for you might be to add the momentum term to speed things up a bit.)
The way you chose to compute the gradient is quite inefficient, which will further slow down learning. You might not have seen reasonable results because it takes forever before the algorithm converges to something interesting. Here is a slightly better way to calculate the gradient:
```
sigmaInv = inv(j.sigma)
dSigma = G(x)/M(x) * 0.5 * (-sigmaInv + numpy.sum(sigmaInv.dot(x - mus[d]) * x))
```
There are still ways to further improve the computation of the gradient. For example, we still get a valid ascent (although not a steepest ascent) direction if we multiple the gradient by a positive definite matrix (such as $\Sigma_j$, which would simplify the gradient a bit). It might also work better if we used a different parametrization of the covariance, such as Cholesky factors, and computed the gradients of those instead.

Solved – Manually fitting a mixture distribution in matlab

You are right that it's not meaningful to evaluate the pdf and think of it as predicted values comparable to the data. You could try something like this:

xx = linspace(1,100);                               % grid of values
ksdensity(x,'support','positive')                   % empirical density plot
line(xx,mypdf(xx,p(1),p(2),p(3),p(4)),'color','r')  % fitted density overlay

Here I used ksdensity function from the Statistics Toolbox, but you could use hist if you normalize to unit area. The mypdf function is the pdf that you define. The first input xx is a grid of values spanning the data. The p(1),...,p(4) values are the estimated parameters that you found.

Best Answer

Related Solutions

Solved – Fitting a gaussian mixture model using stochastic gradient descent

Solved – Manually fitting a mixture distribution in matlab

Related Question