More specifically, why do the likelihood ratio tests have asymptotically a $\chi^2$ distribution if the models are nested, but this is no longer the case for the not-nested models? I understand that this follows from the Wilks' theorem, but unfortunately, I don't understand its proof.
Likelihood Ratio Tests – Why They Can’t Be Used for Non-Nested Models
likelihood-rationested-models
Related Solutions
This is not correct in two ways:
- (Ordinary) likelihood ratio test can only be used to compare nested models;
We cannot compare mean models under REML.(This is not the case here, see @KarlOveHufthammer's comments below.)
In the case of using ML, I am aware of using AIC or BIC to compare the non-nested models.
R.V. Foutz and R.C. Srivastava has examined the issue in detail. Their 1977 paper "The performance of the likelihood ratio test when the model is incorrect" contains a statement of the distributional result in case of misspecification alongside a very brief sketch of the proof, while their 1978 paper "The asymptotic distribution of the likelihood ratio when the model is incorrect" contains the proof -but the latter is typed in old-fashioned type-writer (both papers use the same notation though, so you can combine them in reading). Also, for some steps of the proof they refer to a paper by K.P. Roy "A note on the asymptotic distribution of likelihood ratio" from 1957 which does not appear to be available on-line, even gated.
In case of distributional misspecification, if the MLE is still consistent and asymptotically normal (which is not always the case), the LR statistic follows asymptotically a linear combination of independent chi-squares (each of one degree of freedom)
$$-2\ln \lambda \xrightarrow{d} \sum_{i=1}^{r}c_i\mathcal \chi^2_i$$
where $r=h-m$. One can see the "similarity": instead of one chi-square with $h-m$ degrees of freedom, we have $h-m$ chi-squares each with one degree of freedom. But the "analogy" stops there, because a linear combination of chi-squares does not have a closed-form density. Each scaled chi-square is a gamma, but with a different $c_i$ parameter that leads to a different scale parameter for the gamma -and the sum of such gammas is not closed-form, although its values can be calculated.
For the $c_i$ constants, we have $c_1 \geq c_2\geq ...c_r \geq0$, and they are the eigenvalues of a matrix... which matrix? Well, using the authors notation, set $\Lambda$ to be the Hessian of the log-likelihood and $C$ to be the outer product of the gradient of the log-likelihood (in expectational terms). So $V = \Lambda^{-1} C (\Lambda')^{-1}$ is the asymptotic variance-covariance matrix of the MLE.
Then set $M$ to be the $r \times r$ upper diagonal block of $V$.
Also write $\Lambda$ in block form
$$\Lambda =\left [\begin {matrix} \Lambda_{r\times r} & \Lambda_2'\\ \Lambda_2 & \Lambda_3\\ \end{matrix}\right]$$
and set $W = -\Lambda_{r\times r}+\Lambda_2'\Lambda_3^{-1}\Lambda_2$ ($W$ is the negative of the Schur Complement of $\Lambda$).
Then the $c_i$'s are the eigenvalues of the matrix $MW$ evaluated at the true values of the parameters.
ADDENDUM
Responding to the valid remark of the OP in the comments (sometimes, indeed, questions become a springboard for sharing a more general result, and themselves may be neglected in the process), here is how Wilks's proof proceeds:
Wilks starts with the joint normal distribution of the MLE, and proceeds to derive the functional expression of the Likelihood Ratio. Up to and including his eq. $[9]$, the proof can move forward even if we assume that we have a distributional misspecification: as the OP notes, the terms of the variance covariance matrix will be different in the misspecification scenario, but all Wilks does is take derivatives, and identify asymptotically negligible terms. And so he arrives at eq. $[9]$ where we see that the likelihood ratio statistic, if the specification is correct, is just the sum of $h-m$ squared standard normal random variables, and so they are distributed as one chi-square with $h-m$ degrees of freedom: (generic notation)
$$-2\ln \lambda = \sum_{i=1}^{h-m}\left(\sqrt n\frac{\hat \theta_i - \theta_i}{\sigma_i}\right)^2 \xrightarrow{d} \mathcal \chi^2_{h-m}$$
But if we have misspecification, then the terms that are used in order to scale the centered and magnified MLE $\sqrt n(\hat \theta -\theta)$ are no longer the terms that will make the variances of each element equal to unity, and so transform each term into a standard normal r.v and the sum into a chi-square.
And they are not, because these terms involve the expected values of the second derivatives of the log-likelihood... but the expected value can only be taken with respect to the true distribution, since the MLE is a function of the data and the data follows the true distribution, while the second derivatives of the log-likelihood are calculated based on the wrong density assumption.
So under misspecification we have something like $$-2\ln \lambda = \sum_{i=1}^{h-m}\left(\sqrt n\frac{\hat \theta_i - \theta_i}{a_i}\right)^2$$ and the best we can do is to manipulate it into
$$-2\ln \lambda = \sum_{i=1}^{h-m}\frac {\sigma_i^2}{a_i^2}\left(\sqrt n\frac{\hat \theta_i - \theta_i}{\sigma_i}\right)^2 = \sum_{i=1}^{h-m}\frac {\sigma_i^2}{a_i^2}\mathcal \chi^2_1$$
which is a sum of scaled chi-square r.v.'s, no longer distributed as one chi-square r.v. with $h-m$ degrees of freedom. The reference provided by the OP is indeed a very clear exposition of this more general case that includes Wilks' result as a special case.
Best Answer
Well, I can give a non-rigorous answer from a non-statistician. The Likelihood ratio method relies on the fact that the denominator max likelihood gives a results always at least as good as the numerator max likelihood because the numerator Hypothesis corresponds to a subset of the denominator hypothesis. As a result, ratio is always between 0 and 1.
If you would have non-nested hypothesis (like testing 2 different distributions), likelihood ratio could be > 1 => -1 * log likehood ratio could be < 0 => it is certainly not a chi2 distribution.