Hypothesis Testing – Difference Between Null Distribution and Sampling Distribution

distributionshypothesis testingsampling

I am asking this questions for clarifications on the terms between "null distribution" and "sampling distribution". I find that when some one says null distribution, actually they mean the same thing as when others say sampling distribution.

In this hypothesis testing article [1], you can see the following description of an example

We are considering a random variable Y which is normally distributed. (This is one of the model assumptions.)

Our null hypothesis is: The population mean µ of the random variable Y is a certain value µ0.
For simplicity, we will discuss a one-sided alternative hypothesis: The population mean µ of the random variable Y is greater than µ0. (i.e., µ > µ0)

Another model assumption says that samples are simple random samples. We have data in the form of a simple random sample of size n.

To understand the idea behind the hypothesis test, we need to put our sample of data on hold for a while and consider all possible simple random samples of the same size n from the random variable Y.

For any such sample, we could calculate its sample mean ȳ and its sample standard deviation s. We could then use ȳ and s to calculate the t-statistic t = (ȳ – µ0)/(s/√n)

Doing this for all possible simple random samples of size n from Y gives us a new random variable, Tn. Its distribution is called a sampling distribution.

The mathematical theorem associated with this inference procedure (one-sided t-test for population mean) tells us that if the null hypothesis is true, then the sampling distribution has what is called the t-distribution with n degrees of freedom.

I have no problem with understanding the content, my main concern is about the term "sampling distribution". Here the so-called sampling distribution refers to the test statistic distribution, given the null hypothesis is true. It is a theoretical distribution. According to wikipedia [2], null hypothesis seems to mean the same thing. I have read many lectures notes on statistics, I find both terms coexist. But if you search for sampling distribution, you get many more results.

Could someone clarify my doubts– Do null distribution and sampling distribution mean the same thing?

Reference:
[1] http://www.ma.utexas.edu/users/mks/statmistakes/hyptest.html
[2] http://en.wikipedia.org/wiki/Null_distribution

Best Answer

'Null distribution' is short for the sampling distribution of a statistic under the null hypothesis. 'Sampling distribution' you have to understand from the context: in the context you describe it also means the sampling distribution of a statistic under the null hypothesis, but in another context it could refer to the sampling distribution of a statistic under an alternative hypothesis.

Related Solutions

Solved – How does the sampling distribution of sample means approximate the population mean

I think you might be confusing the expected sampling distribution of a mean (which we would calculate based on a single sample) with the (usually hypothetical) process of simulating what would happen if we did repeatedly sample from the same population multiple times.

For any given sample size (even n = 2) we would say that the sample mean (from the two people) estimates the population mean. But the estimation accuracy -- that is, how good a job we've done of estimating the population mean based on our sample data, as reflected in the standard error of the mean -- will be poorer than if we had a 20 or 200 people in our sample. This is relatively intuitive (larger samples give better estimation accuracy).

We would then use the standard error to calculate a confidence interval, which (in this case) is based around the Normal distribution (we'd probably use the t-distribution in small samples since the standard deviation of the population is often underestimated in a small sample, leading to overly optimistic standard errors.)

In answer to your last question, no we don't always need a Normally distributed population to apply these estimation methods -- the central limit theorem indicates that the sampling distribution of a mean (estimated, again, from a single sample) will tend to follow a normal distribution even when the underlying population has a non-Normal distribution. This is usually appropriate for "bigger" sample sizes.

Having said that, when you have a non-Normal population that you're sampling from, the mean might not be an appropriate summary statistic, even if the sampling distribution for that mean could be considered reliable.

Hypothesis Testing – Understanding Test Statistics and Their Sampling Distribution

I feel it could bring some value to illustrate @Lewian's answer and @Henry's comment for people like me who are more "visual". I will take the example of the same t-test used in the question $\frac{\bar X-\mu_0}{S/\sqrt{n}}$, but focusing more on big picture.

Let's assume that we have a random sample (people's height in cm for example) of size $n = 20$ from a given country population. We want to determine whether the population mean is different from $\mu_0 = 168$ (for the sake of the exercise). We will simulate our sample in R as follows:

set.seed(1)
n <- 20
X <- sample(150:200, n, replace=TRUE)
X

X = {153, 188, 150, 183, 172, 192, 163, 167, 200, 182, 170, 170, 191, 195, 159, 156, 158, 164, 170, 186}

Numerator $\overline{X} - \mu_0$

Our sample is only one possible sample out of the many we could have drawn from our population. It has a mean $\overline{x}$ (actual value) and a standard deviation $s$ (notation for both in lower case that represent observed data). If we imagine that we could get a significant number of random samples (of the same size) from our population of interest, we would be able to calculate the mean for each of them. This distribution of sample means is called the sampling distribution of the means. In our t-test, $\overline{X}$ denotes the random variable that represents this distribution.

The Central Limit Theorem states that, given a sufficiently large sample size, the sampling distribution of the mean for a variable will approximate a normal distribution. We can empirically illustrate that by simulating this sampling distribution of the mean through bootstrapping :

Since $\mu_0$ is fixed (in our t-test we assume that the null hypothesis is true), then $\overline{X}-\mu_0$ is also a random variable. Same distribution as above (normal), but centered on the effect (the difference between the population value and the null hypothesis - illustration example below).

Denominator $S/\sqrt{n}$

The denominator is actually the standard error of the mean which measures the variability of sample means in the sampling distribution of means. $S$ is the estimate of the standard deviation of the population. It is also a random variable and $S^2$ follows a chi-square distribution.

Ratio: t-test $\frac{\overline{X} - \mu_0}{S/\sqrt{n}}$

The numerator is a random variable with a normal distribution, the denominator is a random variable with a scaled chi-square distribution. Both are independent, so it means that their ratio is also a random variable that follows a t-distribution (see "Characterization" in https://en.wikipedia.org/wiki/Student%27s_t-distribution).

Not sure if it helps... but I tried ;-)

Best Answer

Related Solutions

Solved – How does the sampling distribution of sample means approximate the population mean

Hypothesis Testing – Understanding Test Statistics and Their Sampling Distribution

Related Question