Hypothesis Testing – Understanding Test Statistics and Their Sampling Distribution

distributionshypothesis testingself-studyt-test

I am struggling a bit with the real meaning of the test statistic and their sampling distribution, but I cannot find my answer. So, I am looking for clarifications. Sorry in advance if it seems very basic.

To illustrate my question and to keep it simple, I will take the example of a one-sample t-test: $t= \frac{\overline{x} – \mu_0}{se}$

When we say that test statistics have their sampling distributions, does it mean that (in this case), the sampling distribution of the t-test is drawn from sampling distribution of the mean $\overline{X}$? Or is it drawn from the distribution of the null hypothesis assuming the mean of our sample $\overline{x}$ is fixed?

Best Answer

I feel it could bring some value to illustrate @Lewian's answer and @Henry's comment for people like me who are more "visual". I will take the example of the same t-test used in the question $\frac{\bar X-\mu_0}{S/\sqrt{n}}$, but focusing more on big picture.

Let's assume that we have a random sample (people's height in cm for example) of size $n = 20$ from a given country population. We want to determine whether the population mean is different from $\mu_0 = 168$ (for the sake of the exercise). We will simulate our sample in R as follows:

set.seed(1)
n <- 20
X <- sample(150:200, n, replace=TRUE)
X

X = {153, 188, 150, 183, 172, 192, 163, 167, 200, 182, 170, 170, 191, 195, 159, 156, 158, 164, 170, 186}

Numerator $\overline{X} - \mu_0$

Our sample is only one possible sample out of the many we could have drawn from our population. It has a mean $\overline{x}$ (actual value) and a standard deviation $s$ (notation for both in lower case that represent observed data). If we imagine that we could get a significant number of random samples (of the same size) from our population of interest, we would be able to calculate the mean for each of them. This distribution of sample means is called the sampling distribution of the means. In our t-test, $\overline{X}$ denotes the random variable that represents this distribution.

The Central Limit Theorem states that, given a sufficiently large sample size, the sampling distribution of the mean for a variable will approximate a normal distribution. We can empirically illustrate that by simulating this sampling distribution of the mean through bootstrapping :

enter image description here

Since $\mu_0$ is fixed (in our t-test we assume that the null hypothesis is true), then $\overline{X}-\mu_0$ is also a random variable. Same distribution as above (normal), but centered on the effect (the difference between the population value and the null hypothesis - illustration example below).

enter image description here

Denominator $S/\sqrt{n}$

The denominator is actually the standard error of the mean which measures the variability of sample means in the sampling distribution of means. $S$ is the estimate of the standard deviation of the population. It is also a random variable and $S^2$ follows a chi-square distribution.

Ratio: t-test $\frac{\overline{X} - \mu_0}{S/\sqrt{n}}$

The numerator is a random variable with a normal distribution, the denominator is a random variable with a scaled chi-square distribution. Both are independent, so it means that their ratio is also a random variable that follows a t-distribution (see "Characterization" in https://en.wikipedia.org/wiki/Student%27s_t-distribution).

Not sure if it helps... but I tried ;-)

Related Question