I am wondering how I can solve this problem more elegantly:
Let's say I have two set of i.i.d normal random variables with known variance $\sigma^2=1$:
$$
X_1, X_2, …, X_m \sim N(\mu_1, 1)
$$
$$
Y_1, Y_2, …, Y_n \sim N(\mu_2, 1)
$$
And it's given that $\mu_1 \leq \mu_2$. So the MLE of such an observation should be:
$$
L = \prod_{i=1}^m f(X_i | \mu_1) \prod_{j=1}^n f(Y_j|\mu_2)
$$
Where $f(\cdot|\mu)$ is the pdf of normal random variable with $\mu$ as mean and 1 as variance.
Under the constraint of $\mu_1\leq\mu_2$, I can think of this as an optimization problem, and that is to optimize for the L ( or log(L)), subject to the constraint. But this is pretty hard to reach at some kind of closed form solutions.
I am wondering if there is some more clever way to process this problem?
Thanks
Best Answer
The joint likelihood of the parameters is actually $$\mathcal L(\mu_1, \mu_2 \mid \boldsymbol x, \boldsymbol y) \propto \exp\left(-\frac{m}{2}\left((\bar x - \mu_1)^2 + \hat \sigma_x^2\right) - \frac{n}{2}\left((\bar y - \mu_2)^2 + \hat \sigma_y^2 \right) \right) \mathbb 1 (\mu_1 \le \mu_2)$$ where $\bar x$, $\bar y$ are the respective sample means, and $\hat \sigma_x^2$, $\hat \sigma_y^2$ are the (biased) sample variances. After removing the remaining multiplicative factors and taking the logarithm, we get the log-likelihood on the sufficient statistics $$\ell (\mu_1, \mu_2 \mid \bar x, \bar y) \propto -\frac{m(\bar x - \mu_1)^2 + n(\bar y - \mu_2)^2}{2} + \log \mathbb 1 (\mu_1 \le \mu_2).$$ Since the first term is always negative unless $(\mu_1, \mu_2) = (\bar x, \bar y)$, this choice, when possible (i.e. $\bar x \le \bar y$), yields the maximum likelihood. If $\bar x > \bar y$, then this choice is not possible because $\log 0 = -\infty$. By considering the transformation $$(\mu_1', \mu_2') = (\sqrt{m} \mu_1, \sqrt{n} \mu_2),$$ we seek to minimize the squared distance $$(\mu_1' - \sqrt{m}\bar x)^2 + (\mu_2' - \sqrt{n} \bar y)^2$$ subject to the constraint $\mu_1'/\sqrt{m} \le \mu_2'/\sqrt{n}$. As the level sets of this function of $(\mu_1', \mu_2')$ comprise circles centered at $(\sqrt{m} \bar x, \sqrt{n} \bar y)$, we observe that the smallest radius occurs for a circle tangent to the boundary of the transformed parameter space $$\mu_1'/\sqrt{m} = \mu_2'/\sqrt{n}.$$ Back-transforming shows that this corresponds to the boundary of the original parameter space, $\mu_1 = \mu_2$. Now we can solve for the constrained MLE via simple differentiation. The critical points satisfy $$0 = \frac{d \ell(\mu_1, \mu_1 \mid \bar x, \bar y)}{d \mu_1} = \mu_1 (m+n) - (m \bar x + n \bar y),$$ hence the MLE is $$(\hat \mu_1, \hat \mu_2) = \begin{cases} (\bar x, \bar y), & \bar x \le \bar y \\ \left(\frac{m \bar x + n \bar y}{m+n}, \frac{m \bar x + n \bar y}{m+n}\right), & \bar x > \bar y. \end{cases}$$