I am quite confused in these terminologies (especially but not limited to regression)
I do understand what Variance and Standard Deviation means, they measure the dispersion / variability of the data.
However, according to my understanding, Standard Error $= \frac{s}{\sqrt{n}}$ where s is the sample standard deviation.
But in regression (for simplicity, here refer to Simple Linear Regression but MLR shall be of the same fashion) $y = \beta_0 + \beta_{1}x + e$.
Variance of $\hat\beta_1$ = $\frac{\sigma^2}{SXX}$
And while we are doing confidence interval for $\hat\beta_1$, the SE we use is simply the square root of Var($\hat\beta_1$) = $\frac{\sigma}{\sqrt{SXX}}$ without needing to divide by $\sqrt{n}$
My questions:
1) Is there a difference between normal Standard Error (of mean) that we talk about (i.e. $\frac{s}{\sqrt{n}}$) and the SE we talk in regression.
2) I suppose, $\hat\beta_1$ is not a mean but purely an estimator of the parameter $\beta_1$, so why do we use SE when we are constructing confidence interval of $\hat\beta_1$?
3) What about confidence interval for predicted $y$ value and fitted $y$ value respectively?
Best Answer
The term "standard error" refers to the standard deviation of a statistic that is calculated. So, you can calculate a standard error for a mean--because the mean is a statistic. You can also calculate a standard error for a parameter estimate like $\hat{\beta}$.
We say standard error instead of standard deviation to distinguish between a value that's calculated from repeated observations and an estimate that's based on a theory about the distribution.
We only have one observation for $\hat{\beta}$, and we have mathematical theory to derive its sampling error--so we call that the standard error.
We have more than one observation of a variable X, and we calculate the sampling error based on that observed data--so we call that statistic the standard deviation.