Likelihood Ratio Test for Common Variance from Two Normal Distribution Samples

hypothesis testinglikelihood-ratiomathematical-statisticsnormal distributionself-study

$\newcommand{\szdp}[1]{\!\left(#1\right)}
\newcommand{\szdb}[1]{\!\left[#1\right]}$
Problem Statement: Let $S_1^2$ and $S_2^2$ denote, respectively, the variances of
independent random samples of sizes $n$ and $m$ selected from normal
distributions with means $\mu_1$ and $\mu_2$ and common variance $\sigma^2.$
If $\mu_1$ and $\mu_2$ are unknown, construct a likelihood ratio test of
$H_0: \sigma^2=\sigma_0^2$ against $H_a:\sigma^2=\sigma_a^2,$ assuming that
$\sigma_a^2>\sigma_0^2.$

Note 1: This is Problem 10.89 in Mathematical Statistics with Applications, 5th Ed., by Wackerly, Mendenhall, and Sheaffer.

Note 2: This is cross-posted here.

My Work So Far: Let $X_1, X_2,\dots,X_n$ be the sample from the normal distribution
with mean $\mu_1,$ and let $Y_1, Y_2,\dots,Y_m$ be the sample from the normal
distribution with mean $\mu_2.$ The likelihood is
\begin{align*}
L(\mu_1,\mu_2,\sigma^2)
=\szdp{\frac{1}{\sqrt{2\pi}}}^{\!\!(m+n)}
\szdp{\frac{1}{\sigma^2}}^{\!\!(m+n)/2}
\exp\szdb{-\frac{1}{2\sigma^2}\szdp{\sum_{i=1}^n(x_i-\mu_1)^2
+\sum_{i=1}^m(y_i-\mu_2)^2}}.
\end{align*}
We obtain $L\big(\hat{\Omega}_0\big)$ by replacing $\sigma^2$ with
$\sigma_0^2$ and $\mu_1$ with $\overline{x}$ and $\mu_2$ with $\overline{y}:$
\begin{align*}
L\big(\hat{\Omega}_0\big)
=\szdp{\frac{1}{\sqrt{2\pi}}}^{\!\!(m+n)}
\szdp{\frac{1}{\sigma_0^2}}^{\!\!(m+n)/2}
\exp\szdb{-\frac{1}{2\sigma_0^2}\szdp{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}}.
\end{align*}
The MLE for the common variance in
exactly this scenario is:
$$\hat\sigma^2=\frac{1}{m+n}\szdb{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}.$$
So this
estimator plugged into the likelihood yields
\begin{align*}
L\big(\hat{\Omega}\big)
&=\szdp{\frac{1}{\sqrt{2\pi}}}^{\!\!(m+n)}
\szdp{\frac{1}{\hat\sigma^2}}^{\!\!(m+n)/2}
\exp\szdb{-\frac{1}{2\hat\sigma^2}\szdp{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}}.
\end{align*}
It follows that the ratio is
\begin{align*}
\lambda
&=\frac{L\big(\hat{\Omega}_0\big)}{L\big(\hat{\Omega}\big)}\\
&=\szdp{\frac{\hat\sigma^2}{\sigma_0^2}}^{\!\!(m+n)/2}
\exp\szdb{\frac{(\sigma_0^2-\hat\sigma^2)(m+n)}{2\sigma_0^2}}.\\
-2\ln(\lambda)
&=(m+n)\szdb{\frac{\hat\sigma^2}{\sigma_0^2}
-\ln\szdp{\frac{\hat\sigma^2}{\sigma_0^2}}-1}.
\end{align*}
Now the function $f(x)=x-\ln(x)-1$ first decreases, then increases.
It has a global minimum of $0$ at $x=1.$
Note also that the original inequality becomes:
\begin{align*}
\lambda&<k\\
2\ln(\lambda)&<2\ln(k)\\
-2\ln(\lambda)&>k'.
\end{align*}
As the test is for $\sigma_a^2>\sigma_0^2,$ we will expect the estimator
$\hat\sigma^2>\sigma_0^2.$ We can, evidently, use Theorem 10.2 and claim that
$-2\ln(\lambda)$ is $\chi^2$ distributed with d.o.f. $1-0.$ So we reject
$H_0$ when
$$(m+n)\szdb{\frac{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}{(m+n)\sigma_0^2}
-\ln\szdp{\frac{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}{(m+n)\sigma_0^2}}-1}
>\chi^2_{\alpha},$$
or
$$\frac{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}{\sigma_0^2}
-\ln\szdp{\frac{\sum_{i=1}^n(x_i-\overline{x})^2
+\sum_{i=1}^m(y_i-\overline{y})^2}{\sigma_0^2}}-(m+n)
>\chi^2_{\alpha}.$$

My Questions:

Is my answer correct?
My answer is not the book's answer. The book's answer is simply that
$$\chi^2=\frac{(n-1)S_1^2+(m-1)S_2^2}{\sigma_0^2}$$
has a $\chi_{(n+m-2)}^2$ distribution under $H_0,$ and that we reject $H_0$ if
$\chi^2>\chi_a^2.$ How is this a likelihood ratio test? It's not evident that they went through any of the steps of forming the likelihood ratio with all the necessary optimizations. Their estimator is not the MLE for $\sigma^2,$ is it?

Best Answer

Let $x_1,...,x_n\sim N(\mu_1,\sigma^2)$ and $y_1,...,y_m\sim N(\mu_2,\sigma^2)$. Under $H_0$, the likelihood function is

$$L_0=\left(\frac{1}{\sqrt{2\pi\sigma_0^2}}\right)^nexp\left( \frac{-1}{2\sigma_0^2}\left( \sum_{i=1}^{n}(x_i-\bar{x})^2 + \sum_{i=1}^{m}(y_i-\bar{y})^2 \right) \right)=\left(\frac{1}{\sqrt{2\pi\sigma_0^2}}\right)^nexp\left( \frac{-1}{2\sigma_0^2}\left( (n-1)S_1^2 + (m-1)S_2^2 \right) \right).$$

Under $H_a$ we get:

$$L_a=\left(\frac{1}{\sqrt{2\pi\sigma_a^2}}\right)^nexp\left( \frac{-1}{2\sigma_a^2}\left( (n-1)S_1^2 + (m-1)S_2^2 \right) \right).$$

Inside the latter exponent, we have the same sum of $S_1^2$ and $S_2^2$ but multiplied by $\frac{1}{2\sigma_a^2}$, which can be presented as $\frac{\sigma_0^2}{\sigma_a^2}\frac{1}{2\sigma_0^2}$. The LRT is then

$$\lambda=\frac{L_0}{L_a}=\left( \frac{\sqrt{\sigma_a}}{\sqrt{\sigma_0}} \right)^nexp\left( \frac{-1}{2\sigma_0^2}( (n-1)S_1^2 + (m-1)S_2^2) + \frac{\sigma_0^2}{\sigma_a^2}\frac{1}{2\sigma_0^2}( (n-1)S_1^2 + (m-1)S_2^2) \right)=\left( \frac{\sqrt{\sigma_a}}{\sqrt{\sigma_0}} \right)^nexp\left(-\frac{1}{2} \left( 1- \frac{\sigma_0^2}{\sigma_a^2}\right)\frac{1}{\sigma_0^2}( (n-1)S_1^2 + (m-1)S_2^2) \right).$$

Given that $\sigma_a>\sigma_0$, the expression $\left( \frac{\sqrt{\sigma_a}}{\sqrt{\sigma_0}} \right)^n$ is larger that 1 and $\left( 1- \frac{\sigma_0^2}{\sigma_a^2}\right)$ is positive, for all $n$. Thus, our test depends on $\frac{1}{2\sigma_0^2}( (n-1)S_1^2 + (m-1)S_2^2)$ (note I ignored the minus 0.5 there, as it is a constant), which is a $\chi^2$ variable with $n+m-2$ degrees of freedom (why? simple to find).

Ultimately, our test is in the form of

$$\left\{ \frac{1}{\sigma_0^2}\left( (n-1)S_1^2 + (m-1)S_2^2\right) > \chi^2_{\alpha,~n+m-2} \right\}$$

Related Solutions

Uniformly Most Powerful Test for Weibull Distribution

You've got $L=\frac{\theta_a^n}{\theta_0^n}\,exp\left({-\frac{\theta_a-\theta_0} {\theta_0\theta_a}\sum_{i=1}^ny_i^m}\right)<k$, which is good. Now we take $log$ of both sides:

$$n log\left(\frac{\theta_a}{\theta_0}\right)+\left(\frac{\theta_0-\theta_a}{\theta_0\theta_a}\right)\sum_{i=1}^{n}{y_i^m} < log(k)$$

and so the test itself is in the form: $$\left\{ \sum_{i=1}^{n}{y_i^m} < c \right\}$$

(rejecting if $\sum_{i=1}^{n}{y_i^m} > c$).

Now, for part (b), there's something to note here: $y^m$ has an exponential distribution, and so the $\sum{y^m_i}\sim \Gamma(n,\theta)$. Under the null we get that $\frac{2\sum_{i=1}^{n}{y_i^m}}{\theta_0} > \frac{2c}{\theta_0}$ has a $\chi^2$ distribution with $2n$ degrees of freedom (look for the relation between gamma and chi-squared).

Now let's solve (b):

$$\theta_0=100,\theta_a=400,\alpha=0.05,\beta=0.05$$

When $H_0$ is true, we get $\alpha$ using:

$$\alpha=P\left(\frac{2\sum_{i=1}^{n}{y_i^m}}{100} > \chi^2_{0.05}\right)=0.05.$$

When $H_a$ is true, we get $\beta$ using:

$$\beta=P\left(\frac{2\sum_{i=1}^{n}{y_i^m}}{100} \le \chi^2_{0.05} \middle| \theta=400\right)=P\left(\frac{2\sum_{i=1}^{n}{y_i^m}}{400} \le \frac{1}{4}\chi^2_{0.05} \middle| \theta=400\right)=P\left(\chi^2\le\frac{1}{4}\chi^2_{0.05}\right)=0.05$$

So, we need to find the row in $\chi^2$ table where $\frac{1}{4}\chi^2_{0.05}=\chi^2_{0.95}$:

You can see that for $12$ degrees of freedom, $\chi^2_{0.95}=5.226$ and $\chi^2_{0.05}=21.03$, which is the closest we get for achieving $\frac{1}{4}\chi^2_{0.05}=\chi^2_{0.95}$. Recall that this has $2n$ degrees of freedom, so the appropriate sample size is $6$.

Binomial Distribution: Likelihood Ratio Test for Equality of Several Proportions

In the following, I'm going to use the numbers supplied in the exercise. The null hypothesis is $H_0:p_1=p_2=p_3=p_4=p$. As you wrote, the likelihood function is $$ L(\mathbb{p})=\prod_{i=1}^4{200\choose y_i}p_i^{y_i}(1-p_i)^{200-y_i} $$ where $y_i$ are the number of voters favoring $A$ in ward $i$. Under $H_0$, the MLE of $p$ is $\hat{p}=\sum_{i=1}^{4}y_i/800$. Otherwise, we have $\hat{p}_i=y_i/200$ for $i=1, 2, 3, 4$. Plugging these into the likelihood function and forming the ratio, we get $$ \lambda = \dfrac{\left(\dfrac{\sum y_i}{800}\right)^{\sum y_i}\left(1 - \dfrac{\sum y_i}{800}\right)^{800 - \sum y_i}}{\prod_{i=1}^{4}\left(\dfrac{y_i}{200}\right)^{y_i}\left(1 - \dfrac{y_i}{200}\right)^{200 - y_i}}. $$ According to theorem 10.2 in the book, for large $n$, $-2\log(\lambda)$ has approximately a $\chi^2$ distribution with $\nu$ degrees of freedom when $H_0$ is true, where $\nu$ is the difference between the number of freely varying parameters in $\Omega$ and the number of such parameters in $\Omega_0$. Here, we have $4 - 1 = 3$ degrees of freedom.

Using the data provided in the book, we have $y_1 = 76, y_2 = 53, y_3 = 59, y_4 = 48$. According to the formula above, I get $-2\log(\lambda)=10.54$ and a $p$-value of $0.015$. This is very similar to what Rs prop.test gives:

y <- c(76, 53, 59, 48)
prop.test(y, rep(200, 4), correct = FALSE)

X-squared = 10.722, df = 3, p-value = 0.01333
alternative hypothesis: two.sided
sample estimates:
prop 1 prop 2 prop 3 prop 4 
 0.380  0.265  0.295  0.240

Best Answer

Related Solutions

Uniformly Most Powerful Test for Weibull Distribution

Binomial Distribution: Likelihood Ratio Test for Equality of Several Proportions

Related Question