[Math] Minimum mean squared error of uniform distribution

maximum likelihoodmean square errorstatistics

Let $X_1,X_2,\ldots,X_n$ be i.i.d $\operatorname{Uniform}(-\theta,0)$.
Now consider all of the estimates of the form $S_\rho=\rho \hat{\theta}_\text{MLE}$.

I have to find which of these estimates has the minimum mean squared error.

I found that the MLE of $\theta$ is $\hat{\theta}_\text{MLE}=-x_{(1)}$.

I know that the mean squared error is $\operatorname{MSE}(\hat{\theta})=\operatorname E_{\hat{\theta}}[(\hat{\theta}-\theta)^2]$ and that the minimum (best) mean squared error is $\operatorname{MMSE} = \operatorname E[X\mid Y]$. However I wouldn't know how to apply these to $S_\rho=\rho \hat{\theta}_\text{MLE}$. Is there a better (more intuitive) way to see which estimate is the best without using these?

Is the best estimator simply the MLE?

This is from a past exam of 5 years ago and seems very foreign to me, so it is possible that this was never covered in class. Any hint is appreciated.

Best Answer

You showed already that $\rho \hat{\theta}_{MLE} = - \rho X_{(1)}$, so the MSE is $$E[(-\rho X_{(1)} - \theta)^2].$$ It suffices to compute $E [X_{(1)}^2]$ and $E[X_{(1)}]$.

You can compute the density of $X_{(1)}$ by $$f(t) = \frac{d}{dt} P(X_{(1)} \le t) = -\frac{d}{dt} P(X_{(1)} > t) = - \frac{d}{dt} (-t/\theta)^n = (-1)^{n+1} n t^{n-1}/\theta^n,$$ so we have \begin{align} E[X_{(1)}] = (-1)^{n+1} \frac{n}{\theta^n} \int_{-\theta}^0 t^n \mathop{dt} = -\frac{n}{n+1} \theta. \\ E[X_{(1)}^2] = (-1)^{n+1} \frac{n}{\theta^n} \int_{-\theta}^0 t^{n+1} \mathop{dt} = \frac{n}{n+2} \theta^2. \end{align}

Thus the MSE is $$E[(-\rho X_{(1)}-\theta)^2] = \theta^2 \left(\frac{n}{n+2} \rho^2 - 2 \frac{n}{n+1} \rho + 1\right).$$

Now find $\rho$ that minimizes the quadratic on the right-hand side. I believe it should be $\rho=\frac{n+2}{n+1}$.

Interpreting the answer / sanity check: The MLE $-X_{(1)}$ underestimates $\theta$ since $X_{(1)} \ge - \theta$ almost surely. Thus it makes sense that introducing a little inflation by multiplying by a factor $\rho > 1$ might improve the MSE. Of course finding the specific value of the optimal $\rho$ requires computation.

Related Solutions

[Math] Mean Squared Error Maximum uniform distribution

Let $T$ be an estimator of $\theta$ and denote $E(T) = \mu.$ Then $$\text{MSE} = E[(T-\tau)^2] = E[((T - \mu) + (\mu -\tau))^2]\\ = E[(T-\mu)^2] + 2(\mu - \tau)E(T - \mu) + (\mu - \tau)^2\\ = Var(T) + 0 + b_T^2(\tau)$$

Here is a simulation in R statistical software for $W = \max(X_1, \dots, X_n)$ as an estimator of $\theta,$ where $X_i \stackrel{indep}{\sim} Unif(0,\theta),\ n=5,$ and $\theta=10.$ With a million iterations, quantities with original units should be accurate to three places, and quantities with squared units perhaps about two places.

m = 10^6;  n = 5;  th = 10  
x = runif(m*n, 0, th)
DTA = matrix(x, nrow=m)
w = apply(DTA, 1, max)
mean(w);  var(w);  mean((w-th)^2)
## 8.3332              # aprx E(W) = 8.333
## 1.986969            # aprx Var(W) = 1.984
## 4.765187            # aprx MSE = 4.762

mu = n*th/(n+1); mu
## 8.333333            # exact E(W)
var.w = n*th^2/((n+2)*(n+1)^2); var.w
## 1.984127            # exact Var(W)
bias.sq = (mu - th)^2; bias.sq
## 2.777778            # exact squared bias
var.w + bias.sq
## 4.761905            # exact MSE

Except for your incorrect sign (mentioned in the comments), your computations are compatible with simulation results.

Note: An unbiased estimator of $\theta$ is $2\bar X$, but it has a much larger variance ($4\theta^2/12n$), and hence a much larger MSE, than the maximum.

doub.avg = 2*rowMeans(DTA)
mean(doub.avg)
## 10.00068    # unbiased 
var(doub.avg)
## 6.687577    # relatively large var = MSE

The histograms below compare the properties of these two estimators.

[Math] Mean squared error for vectors

Note that if $\hat \theta(X)$ is an estimator (depending on random data $X$) for the parameter $\theta\in \mathbb{R}^n,$ the MSE is a scalar quantity defined as

$$\begin{align}MSE(\hat\theta,\theta)&\equiv E[\|\hat\theta(X)-\theta\|^2]\\ &=E[(\hat\theta(X)-\theta)'(\hat\theta(X)-\theta)].\\\end{align}$$

With some matrix algebra, one can easily prove the identity

$$\begin{align}MSE(\hat\theta,\theta)&=\|Bias(\hat\theta,\theta)\|^2+tr(Var(\hat\theta(X))),\\ Bias(\hat\theta,\theta)&\equiv E[\hat\theta(X)]-\theta. \end{align}$$

So rather than look at a vector of individual MSEs, we typically look at the above metric as the generalization of MSE.

However, the MSE is only one metric to judge an estimator by. One may also be interested in looking at the variance-covariance matrix $Var(\hat\theta(X))$, in which case your question still stands, namely how do we decide which of $V_1\equiv Var(\hat\theta_1(X))$, $V_2\equiv Var(\hat\theta_2(X))$ is "greater" given two estimators $\hat\theta_1(X),\hat\theta_2(X)$?

A common partial order used in this respect that is defined on the set of symmetric positive semidefinite matrices is the Loewner order: $$V_1\geq V_2\iff V_1-V_2 \text{ is positive semidefinite (p.s.d)}.$$

Being a partial order, this relation cannot be used to compare any two variance-covariance matrices summoned from the ether, but it is still meaningful. For instance, because p.s.d matrices have nonnegative diagonal entries, one immediate implication of $V_1\geq V_2$ is that the variance of each component of $\hat\theta_1(X)$ is at least as great as the variance of the corresponding component of $\hat\theta_2(X).$

Best Answer

Related Solutions

[Math] Mean Squared Error Maximum uniform distribution

[Math] Mean squared error for vectors

Related Question