Asymptotic distribution of a Maximum Likelihood Estimator using the Central Limit Theorem

central limit theoremmaximum likelihoodparameter estimationstatistical-inferencestatistics

Let $(X_1,\dots,X_n)$ be a random sample from a population $X$ having probability density function
$$f(x;\vartheta)=\vartheta\,x^{\vartheta -1}\,I_{(0,1)}(x)$$
$$\vartheta>0\qquad\qquad I_{(0,1)}(x)=
\begin{cases}
1\qquad\text{if }x\in(0,1)\\
0\qquad\text{otherwise}
\end{cases}$$
Find:

$\hat{\vartheta}_n$, the MLE (Maximum Likelihood Estimator) of the parameter $\vartheta$
the approximate distribution of $\hat{\vartheta}_n$ for $n$ big

While I'm pretty confident on how to solve the first point, I'd like some advice on the second one.

To find the MLE $\hat{\vartheta}_n$, the likelihood function is calculated:
$$\mathscr{L}(\underline{x};\vartheta) = \vartheta^n\,\left( x_1 \times\dots\times x_n \right)^{\vartheta -1} = \vartheta^n\,\left( \prod_{i=1}^n x_i \right)^{\vartheta -1} \qquad\qquad(0<x_i<1,\, \forall\,i=1,2,\dots,n)$$
The first derivative with respect to $\vartheta$ is
$$\frac{\partial\mathscr{L}(\underline{x};\vartheta)}{\partial\vartheta} = \vartheta^{n-1}\,\left( \prod_{i=1}^n x_i \right)^{\vartheta -1}\left[ n+\vartheta\,\log{\left(\prod_{i=1}^n x_i\right)}\right]$$
and $\hat{\vartheta}_n$ is obtained solving for
$$\frac{\partial\mathscr{L}(\underline{x};\vartheta)}{\partial\vartheta} = 0\quad \implies \quad \hat{\vartheta}_n = -\frac{n}{\log{\left(\prod_{i=1}^n x_i\right)}}$$
To be precise, it should now be checked that
$$\left. \frac{\partial^2\mathscr{L}(\underline{x};\vartheta)}{\partial\vartheta^2}\right|_{\vartheta=\hat{\vartheta}_n} < 0$$
In order to do so, the second derivative with respect to $\vartheta$ is calculated:
$$\frac{\partial^2\mathscr{L}(\underline{x};\vartheta)}{\partial\vartheta^2} = \underbrace{\vartheta^{n-2}\,\left( \prod_{i=1}^n x_i \right)^{\vartheta -1}}_{\Gamma}\underbrace{\left[ n(n-1)+2n\vartheta\log{\left(\prod_{i=1}^n x_i\right)}+\vartheta^2\log^2{\left(\prod_{i=1}^n x_i\right)}\right]}_{\Delta}$$
$\Gamma$ is always positive, hence we only need to evaluate
$$\left.\Delta\right|_{\vartheta=\hat{\vartheta}_n} = -n <0$$

To obtain the approximate distribution of $\hat{\vartheta}_n$ for $n$ big (a.k.a. the asymptotic distribution of $\hat{\vartheta}_n$ for $n\to\infty$), I thought of applying Cramér's Theorem for the asymptotic normality of the MLE.
This assures that
$$\sqrt{n}\left(\hat{\vartheta}_n-\vartheta\right)\xrightarrow{d}\mathscr{N}(0,1/I(\vartheta))$$
where $I(\vartheta)$ is Fisher information, calculated after simple but tedious algebra:
$$I(\vartheta) = \mathbb{E}\left[\left(\frac{\partial}{\partial\vartheta}\log{f(x;\vartheta)} \right)^2\right] = \frac{1}{\vartheta^2}$$
MY QUESTION

While the procedure presented above to obtain the asymptotic distribution is completely general and should be correct (please tell me if it's not), I was wondering if there is any way to straightforwardly apply the Central Limit Theorem (CLT) to the variable
$$\hat{\vartheta}_n = -\frac{n}{\log{\left(\prod_{i=1}^n x_i\right)}}$$
in order to obtain the same result, since $\hat{\vartheta}_n$ can be almost re-written as the sum of (the $\log$ of) iid variables:
$$\hat{\vartheta}_n = -\frac{n}{\sum_{i=1}^n \log{x_i}}$$

Any idea/suggestion would be greatly appreciated!

Best Answer

As already stated, thanks to the Cramer's Theorem it is found that $$\sqrt{n}\left(\hat{\vartheta}_n-\vartheta\right)\xrightarrow{d}\mathscr{N}(0,\vartheta^2)$$ The same result can be found applying in a more straightforward manner the CLT and the Delta method.

The MLE is rewritten as $$\hat{\vartheta}_n = \left(\frac{\sum_{i=1}^n -\log{x_i}}{n}\right)^{-1} = \left(\frac{\sum_{i=1}^n y_i}{n}\right)^{-1} = (\bar{Y}_n)^{-1}$$ with $y_i := -\log{x_i}$ and $\bar{Y}_n := \frac{\sum_{i=1}^n y_i}{n}$.

The CLT assures that $\bar{Y}_n$ is asymptotically normal - since it is the sum of iid variables with finite variance (this will soon be proved) - and so that $$\sqrt{n}(\bar{Y}_n - \mu_y)\xrightarrow{d}\mathscr{N}\left(0,\sigma_y^2\right)$$ where $$\mu_y = \mathbb{E}(y) = -\int_0^1 log{(x)}\,\vartheta\,x^{\vartheta - 1}\,\,\mathcal{d}x = \frac{1}{\vartheta}$$ $$\mathbb{E}(y^2) = \int_0^1 log^2{(x)}\,\vartheta\,x^{\vartheta - 1}\,\,\mathcal{d}x = \frac{2}{\vartheta^2}$$ $$\sigma_y^2 = \mathbb{E}(y^2) - \left( \mathbb{E}(y) \right)^2 = \frac{1}{\vartheta^2}$$ Finding, in the end, that $$\sqrt{n}(\bar{Y}_n - \frac{1}{\vartheta})\xrightarrow{d}\mathscr{N}\left(0,\frac{1}{\vartheta^2}\right)$$ Now, applying the Delta method, also the following must be true: $$\sqrt{n}\left(\left(\bar{Y}_n\right)^{-1} - \left(\frac{1}{\vartheta}\right)^{-1}\right)\xrightarrow{d}\mathscr{N}\left(0,\frac{1}{\vartheta^2}\cdot\left[\left.\frac{d x^{-1}}{dx}\right|_{x=\frac{1}{\vartheta}} \right]^2\right) = \mathscr{N}\left(0,\frac{1}{\vartheta^2}\cdot \vartheta^4\right)$$ which can be rewritten as $$\sqrt{n}\left(\hat{\vartheta}_n - \vartheta\right)\xrightarrow{d}\mathscr{N}\left(0,\vartheta^2\right)$$

This is the result already found applying the Cramer's Theorem.

[many thanks to kimchi lover and even more to Ian for helping through the whole process]

Related Solutions

Statistics – MLE of Beta Distribution Explained

Additional comments: Your answer seems OK. It may be of interest to know that $\hat \theta$ is not unbiased. One can get a rough idea of the distribution of $\hat \theta$ for a particular $\theta$ by simulating many samples of size $n.$ I don't know of a convenient 'unbiasing' constant multiple. The Wikipedia article I linked in my Comment above gives more information.

Here is a simulation for $n = 10$ and $\theta = 5.$

th = 5;  n = 10
th.mle = -n/replicate(10^6, sum(log(rbeta(n, th, 1))))
mean(th.mle)
## 5.555069   # aprx expectation of th.mle > th = 5.
median(th.mle)
## 5.172145

The histogram below shows the simulated distribution of $\hat \theta.$ The vertical red line is at the mean of that distribution, and the green curve is its kernel density estimator (KDE). According to the KDE, its mode is near $4.62.$

den.inf = density(th.mle)
den.inf$x[den.inf$y==max(den.inf$y)]
## 4.624876

hist(th.mle, br=50, prob=T, col="skyblue2", main="")
abline(v = mean(th.mle), col="red")
lines(density(th.mle), lwd=2, col="darkgreen")

Addendum on Parametric Bootstrap Confidence Interval for $\theta:$

In order to find a confidence interval (CI) for $\theta$ based on MLE $\hat \theta,$ we would like to know the distribution of $V = \frac{\hat \theta}{\theta}.$ When that distribution is not readily available, we can use a parametric bootstrap.

If we knew the distribution of $V,$ then we could find numbers $L$ and $U$ such that $P(L \le V = \hat\theta/\theta \le U) = 0.95$ so that a 95% CI would be of the form $\left(\frac{\hat \theta}{U},\, \frac{\hat\theta}{L}\right).$ Because we do not know the distribution of $V$ we use a bootstrap procedure to get serviceable approximations $L^*$ and $U^*$ of $L$ and $U.$ respectively.

To begin, suppose we have a random sample of size $n = 50$ from $\mathsf{Beta}(\theta, 1)$ where $\theta$ is unknown and its observed MLE is $\hat \theta = 6.511.$

Entering, the so-called 'bootstrap world'. we take repeated 're-samples` of size $n=50$ from $\mathsf{Beta}(\hat \theta =6.511, 0),$ Then we we find the bootstrap estimate $\hat \theta^*$ from each re-sample. Temporarily using the observed MLE $\hat \theta = 6.511$ as a proxy for the unknown $\theta,$ we find a large number $B$ of re-sampled values $V^* = \hat\theta^2/\hat \theta.$ Then we use quantiles .02 and .97 of these $V^*$'s as $L^*$ and $U^*,$ respectively.

Returning to the 'real world' the observed MLE $\hat \theta$ returns to its original role as an estimator, and the 95% parametric bootstrap CI is $\left(\frac{\hat\theta}{U^*},\, \frac{\hat\theta}{L^*}\right).$

The R code, in which re-sampled quantities are denoted by .re instead of $*$, is shown below. For this run with set.seed(213) the 95% CI is $(4.94, 8.69).$ Other runs with unspecified seeds using $B=10,000$ re-samples of size $n = 50$ will give very similar values. [In a real-life application, we would not know whether this CI covers the 'true' value of $\theta.$ However, I generated the original 50 observations using parameter value $\theta = 6.5,$ so in this demonstration we do know that the CI covers the true parameter value $\theta.$ We could have used the probability-symmetric CI with quantiles .025 and .975, but the one shown is a little shorter.]

set.seed(213)
B = 10000;  n = 50;  th.mle.obs=6.511
v.re = th.mle.obs/replicate(B, -n/sum(log(rbeta(n,th.mle.obs,1))))
L.re = quantile(v.re, .02);  U.re = quantile(v.re, .97)
c(th.mle.obs/U.re, th.mle.obs/L.re)
##     98%       3% 
## 4.936096 8.691692

Derive asymptotic distribution of the ML estimator

The relationship between $X_i$ and $Y_i = -\log X_i$ is understood by transformation: $$f_{Y_i}(y) = f_{X_i}(e^{-y}) \left|\frac{d}{dy} \left[e^{-y}\right]\right| = (\theta+1) (e^{-y})^\theta e^{-y} = (\theta+1) e^{-(\theta+1)y}, \quad y > 0,$$ thus $$Y_i \sim \operatorname{Exponential}(\theta+1),$$ where $\theta+1$ is a rate parameter; and to create a pivotal quantity, it is easy to see that the distribution of $2(\theta+1) Y$ is exponential with rate $1/2$; that is to say, chi-square with $2$ degrees of freedom.

Best Answer

Related Solutions

Statistics – MLE of Beta Distribution Explained

Derive asymptotic distribution of the ML estimator

Related Question