I'm re-learning statistics and got confused by the idea of taking the standard error from 1 sample's worth of data (standard deviation of sample divided by the square root of sample size, i.e. $\frac{s}{\sqrt{n}}$). As I understand it, the standard error is the spread of many sample means in an attempt to gauge how precise (not accurate) our estimate of the population mean is, but what if there's just the one sample? Usually experiments can't or just aren't repeated and only have 1 sample from a population, no? So, how can I pretend to know the spread of many means from samples of a population that I only did hypothetically?
If this is an example of a frequentist's versus a Bayesian approach to statistics, I'd just like to learn the frequentists side of the argument (just to understand what it is), please. Thanks in advance!
Best Answer
Very short
A sample is not just one sample but contains many individual observations. Each of the observations can be considered as a sample (is there a difference between '$n$ samples of size 1' and '1 sample of size $n$'?). So you actually have multiple samples that can help to estimate the standard error in sample means.
In order to estimate the variance of the mean of samples, would you rather have a sample of size one million or multiple (say a hundred) samples of ten?
A bit longer
A sample will almost never be picked such that it perfectly matches the population. Sometimes a sample might pick relatively low values, sometimes a sample might pick relatively high values.
The variation in the sample mean, due to these random variations in picking the sample, is related to the variation in the population that is sampled. If the population has a wide spread in high and low values, than the deviations in a random samples with relatively high/low values will be corresponding to this wide spread and they will be large.
The error/variation in the means of samples relates to the variance of the population. So we can estimate the former with the help of an estimate of the latter. We can estimate the variance of sample.means by the variance of the population. And for this estimate of the variance of the population, one single sample is sufficient.
In formula form
The variance of the sample means $\sigma_n$ where the samples are of size $n$ is related to the variance of the population $\sigma$ $$\sigma_n = \frac{\sigma}{\sqrt{n}}$$
So an estimate of $\sigma$, for which a single sample is sufficient, can also be used to estimate $\sigma_n$.