expectation-maximizationgaussian mixture distributionmachine learning
It's typical to initialize EM for Gaussian Mixture Models using the result of kmeans clustering. However, kmeans only gives you the means (centers) of the starting GMM, but EM initialization often requires a complete GMM description (that is, including the covariance matrices and weights).
Therefore what is a 'good' way to come up with initial covariance matrices and weights for kmeans-based GMMs? Simply assign random values (assuming sum(weights)=1) ?
In the second paper you linked to:
Naive Bayes represents a distribution as a mixture of components, where within each component all variables are assumed independent of each other. Given enough components, it can approximate an arbitrary distribution arbitrarily closely. ... When learned from data, naive Bayes models never contain more components than there are examples in the data ...
and then later, the quote you mentioned,
Naive Bayes models can be viewed as Bayesian networks in which each $X_i$ has $C$ as the sole parent and $C$ has no parents. A naive Bayes model with Gaussian $P(X_i |C )$ is equivalent to a mixture of Gaussians with diagonal covariance matrices (Dempster et al., 1977).
Then they go on to show that, in the discrete case, $P(X=x)=\sum_{c=1}^k\left(P(c)\prod_{i=1}^{|X|}P(x_i|c)\right)$.
So consider $X\sim N(\mu,\Sigma^\text{diag})$. It's easy to see that: $$ P(X=x)=\sum_{c=1}^kP(c)\prod_{i=1}^{|X|}P(x_i|c)=\sum_{c=1}^kP(c)\cdot P(x_1,\dots,x_{|X|}|c) $$ since the joint distribution of $X$ is just the product of the distributions of its independent components. This, of course, is a mixture of $k$ Gaussians, each with weight $P(c)$. So unless they're talking about something else, I think I was right in the comments. What makes me less than 100% confident is that a) they talk about "enough components" which makes me think it should be more than $k$, and b) I have no idea where in Dempster (1977) this is or why they cite that paper of all things. I tried sifting through Dempster but it's on a completely different subject (E-M algorithm for ML with missing data) and I don't think they even had the term "Naive Bayes" back then. APA really needs to start requiring page numbers somehow.
I think the reason they point this out is that they develop Naive Bayes from a graphical / network perspective, so this provides a different intuition.
Pattern Recognition and Machine Learning by Bishop, gives detailed steps for implementing EM for Gaussian mixture models.
Using the steps given here, it should be easy to implement.
Best Answer
k-means also tells you which data points belong to which cluster. A good starting estimate for the covariances should be the within-cluster covariances, and a good estimate for the weights should be the fractions of data points allocated to each cluster.