Maximum Likelihood for Bernoulli Mixture – Known Mixture Proportions in EM Algorithm

bernoulli-distributionestimationexpectation-maximizationmaximum likelihoodmixture-distribution

Given the hierarchical model
$$
\begin{align}
k & \sim \text{Categorical}(\pi_1, \dots, \pi_K) \\
X \mid k & \sim \text{Bernoulli}(\theta_{k})
\end{align}
$$

and an i.i.d. sample $X_1, \dots, X_N$, it is standard to use the EM algorithm to produce a joint estimate of $(\boldsymbol{\theta}, \boldsymbol{\pi})$.

But what if the mixture proportions $\pi_1, \dots, \pi_K$ are known? In this case, is there a closed form maximizer for the likelihood of the unknown Bernoulli parameters $\theta_1, \dots, \theta_K$,
$$
\mathcal{L}(\boldsymbol{\theta}) = \prod_{n = 1}^N \sum_{k=1}^K \pi_k \theta_k ^{X_n} (1 – \theta_k) ^{1 – X_n}
$$

Best Answer

No, the parameters are not identifiable. For example, if $\boldsymbol{\pi} = (.2, .3, .5)$ and the samples are evenly split between 0's and 1's, $\boldsymbol{\theta}$ could be (0, 0, 1), (1, 1, 0), (.5, .5, .5), (.25, .25, .75), etc.

Related Question