I am trying to understand why I can't simultaneously minimize Type I error and Type II error simultaneously. A non-randomized Hypothesis test is essentially a deterministic partition of the Range space of our Random Variable.
$$R_0=\{x\in R: Reject Null\}\hspace{4mm}R_1=\{x\in R: Accept Null\}$$
where $R_0\cup R_1=R$
If I try to minimize the event of a type I Error then I can essentially remove some $x$ from $R_0$ to $R_1$ then have the new partition of $R$ as $R_0^{'}\cup R_1^{'}=R$
if
$$P_{\theta_0}(x\in R_0)=\alpha$$
$$P_{\theta_a}(x\in R_1)=\beta$$
then the new rejection region, $R_0^{'}$ post-transferring of x will have
$$P_{\theta_0}(x\in R_0^{'})\leq\alpha$$
$$P_{\theta_a}(x\in R_1^{'})\geq\beta$$
Thus In attempting to reduce the event of Type I error I have increased the event of a Type II error. But I have read that this trade-off only holds in fixed sample size why is that. Is an increase in sample size decrease both events simultaneously?
Best Answer
In all hypothesis tests, we calculate a $p$-value based on a test statistic calculated from our sample. Usually, we determine a threshold (significance level) $\alpha$ such that if $p < \alpha$, we reject the null hypothesis $H_0$. Note that our $p$-value is calculated under the assumption that $H_0$ is true, and represents the probability of the given parameter (e.g. mean) of the sample being observed given $H_0$ is true, so $$P(\text{type I error}) = P(\text{reject }H_0 \text{ when } H_0 \text{ is true}) = \alpha$$ as we only reject $H_0$ when $p \leq \alpha$. The point to note here is that the probability of a type I error only depends on the significance level.
The probability of a type II error will depend on the population parameters and the sample size, since we make such an error when our observed sample fails to lie in what you have called $R_0$ - the $p$-values which will force us to reject $H_0$. In most hypothesis tests, the distribution of the test statistic calculated for a sample becomes more and more concentrated as we increase the sample size. For example, in hypothesis tests for the population mean, increasing the sample size decreases the standard deviation of the sample mean. Hence, the test is more sensitive to extreme values of the calculated test statistic and is more likely to give extreme $p$-values (upon which we would reject the null hypothesis). If you have a specific hypothesis testing procedure in mind, I would be happy to flesh out the details for that procedure for you.
The point to note here is that the probability of a type II does not only depend on the significance level, and in nearly all cases decreases with sample size. So one can simultaneously minimize the two errors by decreasing the significance level and increasing the sample size.