I was reading about the Neyman-Pearson Lemma (https://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma).
Let $X_1, X_2, \ldots, X_n$ be a random sample from a distribution with unknown parameter $\theta$. Consider two hypotheses:
- Null hypothesis $H_0$: $\theta = \theta_0$
- Alternative hypothesis $H_1$: $\theta = \theta_1$ ( $\theta_1 \neq \theta_0$)
Let $f(x|\theta)$ denote the probability density function (or probability mass function) of $X_1, X_2, \ldots, X_n$.
The Neyman-Pearson lemma states that the likelihood ratio test (LRT) is the most powerful test at level $\alpha$, where the LRT rejects $H_0$ if and only if:
$$\frac{L(\theta_1 | \mathbf{x})}{L(\theta_0 | \mathbf{x})} > k,$$
where $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ is the observed sample, $L(\theta|\mathbf{x}) = \prod_{i=1}^n f(x_i|\theta)$ is the likelihood function, and $k$ is a constant chosen such that the test has level $\alpha$.
My Question: This might sound like a rhetorical question – but what other possible tests could exist? The Neyman-Pearson Lemma is said to be very important as it shows us what is the most powerful statistical test – but it seems to me that the test indicated by the Neyman-Pearson is practically the only test that exists. What other tests can be considered as competitors?
Thanks!
Best Answer
This is an belated answer to this common question (in Statistics). Recall that if $\{P_\theta:\theta\in \Theta\}$ is a population (family of probability distributions on a common space $(\mathbb{X},\mathscr{B})$, then a testing statistic $T(X)$ for the test $H_0: \theta\in\Theta_0$ v.s. $H_1: \theta\in\Theta_1$, where $\Theta=\Theta_0\cup\Theta_1$, $\emptyset=\Theta_0\cap \Theta_1$, is a uniformly most powerful test(UMP) of size $0\leq\alpha\leq 1$ if
In particular, if $\Theta=\{\theta_0,\theta_1\}$, and $P_{\theta_j}=f_j\,d\mu$, $j\in\{0,1\}$, $f_0\leq f_1$, where $\mu$ is a $\sigma$-finite measure on $(\mathbb{X},\mathscr{B})$ and $1\leq \alpha\leq 1$, Neyman-Paerson's theorem states that the test statistic $\psi(X)$ given by \begin{align} \psi(X) = \left\{\begin{array}{lcr} 1 & \text{if} & f_1(X)>k f_0(X)\\ \gamma & \text{if} & f_1(X)=k f_0(X)\\ 0 & \text{if} & f_1(X)<k f_0(X) \end{array} \right. \tag{1}\label{one} \end{align} where $k$ and $\gamma$ are chosen so that $\alpha=\mathbb{E}_0[\psi(X)]$, is a UMP test of level $\alpha$ for the test $H_0: \theta=\theta_0$ versus $H_1:\theta=\theta_1$. Furthermore, if $T(X)$ is another statistic with $0\leq T(X)\leq 1$, we have that $T(X)$ is UMP of level $\alpha$ iff $\mathbb{E}_0[T(X)]=\alpha$ and on $\{f_1(X)\neq k f_0(X)\}$ $T(X)=\psi(X)$ $\mu$-almost everywhere.
The test $\psi(X)$ is a randomized test when $\mu(f_1(X)=k f_0(X))>0)$. Neyman-Pearson's theorem implies that outside $\{f_(X)=k f_0(X)\}$, which could be empty $\mu$-almost everywhere, any UMP test at level $\alpha$ coincides with $\psi(X)$. However this does not imply that the Neyman-Pearson statistic is the only UMP. Here is one example.
Example: Suppose $0<\theta_0<\theta_1$ and $$f_{\theta_j}(\mathbf{x})=\frac1{\theta^n_j}\mathbb{1}_{(0,\theta_j)}(x_1)\cdot\ldots\cdot\mathbb{1}_{(0,\theta_j)}(x_n)=\frac1{\theta^n_j}\mathbb{1}_{(0,\theta_j)}(x_{(n)})$$ where $x_{(n)}=\max_{1\leq k\leq n}x_k$. This is the case of $n$-i.i.d. uniform random variables on $(0,\theta_j)$, $j\in\{0,1\}$. The dominating measure $\mu$ is Lebesgue's measure on the line. Notice that $$\frac{f_1(\mathbf{x})}{f_0(\mathbf{x})}=\Big(\frac{\theta_0}{\theta_1}\Big)^n\mathbb{1}_{(0,\theta_0)}(x_{(n)})+\infty\cdot\mathbb{1}_{[\theta_0,\theta_1)}(x_{(n)})$$ It is easy to check that the size $\alpha$ Neyman-Peason statistic \eqref{one} for $H_0:\theta=\theta_0$ versus $H_1:\theta=\theta_1$ is given by \begin{align} \psi(X) = \left\{\begin{array}{lcr} 1 & \text{if} & X_{(n)}>\theta_0\\ \alpha & \text{if} & X_{(n)}<\theta_0 \end{array} \right. \tag{2}\label{two} \end{align} The power of $\psi(X)$ against $\theta_1$ is given by $$E_1[\psi(X)]=1-(1-\alpha)n\theta^{-n}_0\int^{\theta_1}_{\theta_0}x^{n-1}\,dx=1-\big(\frac{\theta_0}{\theta_1}\big)^n(1-\alpha)$$
Observe that the ration $f_1/f_0$ is a monotone nondecreasing function of the (sufficient) statistic $X_{(n)}$. Define the test statistic \begin{align} \psi(X) = \left\{\begin{array}{lcr} 1 & \text{if} & X_{(n)}>c\\ 0 & \text{if} & X_{(n)}<c \end{array} \right. \tag{3}\label{three} \end{align} where $\alpha=\mathbb{P}_0[X_{(n)}>c]$, that is, $c$ is a solution to $\alpha=n\theta^{-n}_0\int^{\theta_0}_c x^{n-1}\,dx$, that is $c=(1-\alpha)^{1/n}\theta_0$. The power of the test $\tilde{\psi}(X)$ against $\theta_1$ is $$E_1[\tilde{\psi}(X)]=n\theta^{-n}_1\int^{\theta_1}_cx^{n-1}\,dx=1-\big(\frac{\theta_0}{\theta_1}\big)^n(1-\alpha)$$
It follows that $\tilde{\psi}$ is also a UPM test. I leave it to the OP to check that $\psi(X)$ and $\tilde{\psi}(X)$ differ on a set of (Lebesgue) measure $\mu(\psi(X)\neq \tilde{\psi}(X))>0$. However, on $\{f_1(X)\neq kf_0(X)\}=\{X_{(n)}<\theta_0\}$ we do have that $\tilde{\psi}(X)=\psi(X)$ $\mu$-a.s. as stated by the theorem.