I don't know why but I find textbooks to be so inconsistent on describing the types of hypothesis tests there are (at an intro level); I want to organize my thoughts a little bit.
When you repeatedly take samples of size $n$ from a certain population distribution with parameters mean $\mu$ and variance $\sigma^2$, you get a sampling distribution of the sample statistic, in our case as with most intro books the sample statistic in question is the sample mean $\bar{x}$.
Such sampling distribution can be normal or non-normal. Even if it's non-normal, CLT says if $n$ is very large then the sampling distribution is approximately normal. So far so good?
Case 1: sample distribution is normal, regardless of $n$: we use the z-score and a normal table to do a hypothesis test:
$$
z=\frac{\bar{x}-\mu}{\sigma}
$$
Case 2: sample distribution is non-normal, but $n$ is large: we use the z-score and a normal table to do a hypothesis test, but the SE is slightly different:
$$
z=\frac{\bar{x}-\mu}{\sigma /\sqrt{n}}
$$
Case 3: sample distribution is non-normal, and $n$ is small: we use the t-stat and the t-table to do a hypothesis test, and the SE involves the sample deviation $\hat{s}$ instead of $\sigma$ the population deviation:
$$
t = \frac{\bar{x}-\mu}{\hat{s}/\sqrt{n}}
$$
Questions at this point is: do I have case 1 & 2 correct? My book is a little confusing and I'm not sure if they should be different, or if the both use $\sigma/\sqrt{n}$ as the SE. Also, are we indeed looking at whether the sampling distribution is normal, or was I supposed to look at whether the population distribution is normal or not?
I have further questions for when you start testing differences in means but I'll put that in a separate question here.
Best Answer
To really understand the issue, we first revisit what happens when observations in a sample are drawn from a normal distribution. Let $$X_i \sim \operatorname{Normal}(\mu, \sigma^2), \quad i = 1, 2, \ldots$$ be a sequence of independent and identically distributed normal random variables with mean $\mu$ and variance $\sigma^2$. Then the sample mean $$\bar X = \frac{1}{n} \sum_{i=1}^n X_i$$ from a sample of size $n$ is also exactly normally distributed: $$\bar X \sim \operatorname{Normal}(\mu, \sigma^2/n).$$ Note that the mean is the same, but now the variance is $\sigma^2/n$, rather than $\sigma^2$. This makes intuitive sense: if you average many observations together, the variance of the average should decrease with more data. Formally, we observe that $$\operatorname{Var}[\bar X] \overset{\text{iid}}{=} \frac{1}{n^2} \sum_{i=1}^n \operatorname{Var}[X_i] = \frac{1}{n^2} n \sigma^2 = \frac{\sigma^2}{n}.$$ Therefore, the expression $$Z = \frac{\bar X - \mu}{\sigma/\sqrt{n}} \sim \operatorname{Normal}(0, 1) \tag{1}$$ is a pivotal quantity. There is no asymptotic approximation or use of the Central Limit Theorem here.
Now, if the $X_i$ are not normal, but $n$ is large, then the above quantity $(1)$ is approximately standard normal via the CLT (except in certain special cases for the distribution of the $X_i$, where the CLT does not hold). Note that $\mu$ and $\sigma^2$ are the mean and variance of the non-normal $X_i$.
Thus far, we have not spoken about known versus unknown parameters. We see that if $\mu$ and $\sigma$ are known, we can directly calculate a realization of $Z$ from the sample mean $\bar X$. For example, if I said that the $X_i$ follow a gamma distribution with shape $a = 5$ and rate $b = 4$, then the mean and variance are $$\mu = \frac{a}{b} = \frac{5}{4}, \\ \sigma^2 = \frac{a}{b^2} = \frac{5}{16},$$ and if we observe in a sample of size $n = 50$ the sample mean $\bar X = 1.32906$, all we need to do to calculate $Z$ is write $$Z = \frac{1.32906 - 5/4}{\sqrt{5/16}/\sqrt{50}} \approx 1.0000.$$ But what if we did not know $\sigma^2$? Then it must be estimated from the sample itself. Doing this incurs a penalty, because we are using the data to estimate both the true mean $\mu$ and the true standard deviation $\sigma$. If an inference is to be performed on the true mean, then for a hypothesis test that assumes under the null $$H_0 : \mu = \mu_0$$ we have the conditional test statistic $$Z \mid H_0 = \frac{\bar X - \mu_0}{s/\sqrt{n}} \sim t_{n-1},$$ where $s$ is the sample standard deviation and $t_{n-1}$ is the Student's $t$ distribution with $n-1$ degrees of freedom. Notice here that this is a true statistic: $Z \mid H_0$ is only a function of the random variables $X_i$ in the sample that allow us to compute $\bar X$ and $s$. $\mu_0$ is assumed as part of performing the hypothesis test, and $n$ is known.
In summary, there really are only two cases. When the distribution of the $X_i$ is normal, the statistic $\bar X$, which is the sampling distribution of the sample mean, is also normal, with no use of CLT. When the distribution of the $X_i$ is not normal, the statistic $\bar X$, which again is the sampling distribution of the sample mean, is not normal, but under certain conditions, is asymptotically normal via CLT. What you do with this when constructing a hypothesis test is a separate issue, which relates to whether the parameters of the population distribution are known or unknown.
When the population parameters are known, there is no inference needed. When the population variance is known, then a hypothesis test can be constructed to make a statistical inference about the unknown mean based on the observed sample mean, because the sampling distribution of the sample mean is either exactly normal, or approximately normal with sufficient sample size. When the population variance is unknown, then irrespective of whether the population is normal or not, the test statistic for the location parameter cannot be computed without estimating this variance through the sample. It is this estimation that changes the distribution of the test statistic from normal to Student's $t$.