Solved – General theorems for consistency and asymptotic normality of maximum likelihood

maximum likelihoodreferences

I'm interested in a good reference for results concerning asymptotic properties of maximum likelihood estimators. Consider a model $\{f_n(\cdot \mid \theta): \theta \in \Theta, n \in \mathbb N\}$ where $f_n(\mathbf x \mid \theta)$ is an $n$-dimensional density, and $\hat \theta_n$ is the MLE based on a sample $X_1, \ldots, X_n$ from $f_n(\cdot \mid \theta_0)$ where $\theta_0$ is the "true" value of $\theta$. There are two irregularities I'm interested in.

  1. The data $X_1, \ldots, X_n$ are not iid and, as a result, the Fisher information about $\theta$ accrues at a rate slower than $n$.
  2. $\Theta$ is a bounded set, and with positive probability $\hat \theta_n$ lies on the boundary. The boundary corresponds to a "simpler" model, and so there is particular interest in whether or not $\theta_0$ lies on the boundary.

My particular questions are

  1. Letting $J_n(\theta)$ denote the observed Fisher information corresponding to $\theta$, and suppose $\theta_0$ lies in the interior of $\Theta$. Under what conditions is $$\left[J_n(\hat \theta_n)\right]^{1/2}(\hat \theta_n – \theta_0)$$ asymptotically normal as $n \to \infty$? In particular, are the regularity conditions similar to the usual ones, with the relevant modification being $J_n(\hat \theta_n) \to \infty$ in some sense?

  2. Suppose instead that $\theta_0$ is on the boundary, and again recall that $\hat \theta_n = \theta_0$ happens with positive probability – for concreteness, in a mixed effects model $Y_{ij} = \mu + \beta_i + \epsilon_{ij}$ we can have $\hat \sigma_{\beta}^2 = 0$. Under what conditions does $\hat \theta_n \to \theta_0$ (almost surely or in probability) and under what conditions will $\hat \theta_n = \theta_0$ eventually (this probably fails for the mixed effects model, but corresponds to "oracle" properties for the LASSO and related estimators, so perhaps it is too much to ask for general results)?

Again, just a pointer to a text with results at this level of generality would be greatly appreciated.

Best Answer

References you can start from:

For the case where the true parameter lies on the boundary:
Moran (1971) "Maximum likelihood estimation in nonstandard conditions"

Steven G. Self and Kung-Yee Liang (1987) "Asymptotic Properties of Maximum Likelihood Estimators and Likelihood Ratio Tests Under Nonstandard Conditions"

Ziding Feng and Charles E. McCulloch (1990) "Statistical Inference Using Maximum Likelihood Estimation and the Generalized Likelihood Ratio when the True Parameter Is on the Boundary of the Parameter Space"

For non-identical but independent r.v.'s:
Bruce Hoadley (1971) "Asymptotic Properties of Maximum Likelihood Estimators for the Independent Not Identically Distributed Case"

For dependent r.v.'s:
Martin J. Crowder (1976) "Maximum Likelihood Estimation for Dependent Observations"

Also

Huber, P. J. (1967). "The behavior of maximum likelihood estimates under nonstandard conditions". In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 1, pp. 221-233).

Update 17-03-2017: as suggested in a comment, the following paper may be referenced here

Andrews, D. W. (1987). Consistency in nonlinear econometric models: A generic uniform law of large numbers. Econometrica: Journal of the Econometric Society, 1465-1471.

Related Question