finite-mixture-modelgaussian mixture distributionlikelihoodoverfitting
I am trying to understand how to select the number of components in a Gaussian Mixture Model (GMM). Most presentations mention the use of criteria such as AIC and BIC.
But if we simply follow model selection approaches for supervised learning, we could for example perform a cross-validation and estimate the likelihood for each held-out set. Then we choose the model with the highest averaged likelihood. Is this a valid approach for selecting the number of components of a GMM?
First regarding the practical issue of vanishing variance most implementations of GMM will have a limit on how small the variance can get, you can set the minimum to some small value based on your specific problem.
Secondly EM is guaranteed to improve on the objective function (Likelihood in the case of ML) or remain the same. After each iteration the parameters are updated. If the parameters have converged you have a local optimum, which is your final solution. The short answer is there is nothing preventing you form finding such a solution - If you have the necessary precautions to prevent variance going to zero.
Whit clustering it is very likely that the error measure decreases (even on test set) as you increase the number of clusters (here the number of components in GMM). There are two competing factors that change the error metric on the test set.
For example consider the following picture. Modelling this data as mixture of two Gaussian distributions seems sufficient. But you will have some dispersion as not all the points are at the mean of each Gaussian.
If you continue and model it with a mixture of a higher number of Gaussian distributions then the total likelihood will increase on the training set. At the same time it is very likely that the likelihood on the test set increases too.
When you use AIC you add a penalty term $2p$ which is not enough in your case to change the direction of decreasing the dispersion on test set.
For determining the number of clusters generally people look at the elbow on the test error. When you add the AIC of BIC penalty then you increase the sharpness of the elbow.
https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#cite_note-6
You can also have look at the 'elements of statistical learning' pages 518-520, available at: http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf
Best Answer
I would say using hold-out data set likelihood is a good approach. For Mixture of Gaussians, the more Guassians we have, the better likelihood we can get. Just like the order in polynomial regression problem.
AIC and BIC will penalize on number of parameters (Gaussians) used automatically, but using a separate testing set will also be a good choice. I will use an extreme example to explain, suppose we select number of Gaussian is as same as number of data points in your training set. Your training score will be really good (Infinite likelihood), but the testing score will not be so. Which is as same as other machine learning model selection process.