Solved – Maximum likelihood estimator for distribution with bound constraints

bernoulli-distributionmaximum likelihood

In class, we talked about finding Maximum Likelihood Estimators but there is something I don't think we talked about. How is the MLE different for distributions with constrained bounds?

For example, for an iid random sample $X_1,\ldots,X_n$, if we compare the MLE for $\theta$ from a Bernoulli distribution versus the MLE for $\theta$ from another Bernoulli distribution but with 0 < $\theta$ < $\frac 12$, how is the MLE affected?

I know the MLE for a Bernoulli($\theta$) with 0 < $\theta$ < 1 is $\frac{\Sigma x_i}{n}$.

I thought it was the same for the 0 < $\theta$ < $\frac 12$ case but thinking back to a Uniform distribution, the MLE will vary depending on the bounds. However, I don't think order statistics is helpful here. Does anyone have any ideas?

Best Answer

In general, but not always, what will happen is that the constrained MLE will be the closest possible value to the unconstrained MLE. To re-use your example, if $\sum x_i / n = 0.7$ but $0 < \theta \leq 0.5$, then the unconstrained MLE for $\theta$ is $0.7$ but the constrained MLE for $\theta$ is 0.5. This will always be the case if the log likelihood is concave in the parameter of interest.

Note that if you have a strict inequality constraint, for example, $0 < \theta < 0.5$, the constrained MLE doesn't exist, as there is no largest number less than $0.5$.

In some cases, though, the likelihood function can be multimodal. (The Student-t distribution is an example of this.) The unconstrained MLE corresponds to whichever mode is the highest (or, in the case of ties, is any value from the set corresponding to the highest modes.) In these cases, it may be that the constraint excludes the highest mode, so that the constrained MLE becomes EITHER the value corresponding to the highest mode remaining in the interior of the set of feasible values OR a value on the boundary of the set of feasible values - which latter may not appear to be a mode when looking at the unconstrained problem.

The following graph shows an example of this. The unconstrained MLE is at $6.4$, but, if we add a constraint at the red line ($5.5$), the new MLE is at $5.5$, despite the presence of a mode at $0.7$.

enter image description here

Related Question