Solved – t-statistic for linear regression

hypothesis testinglinearp-valueregression

I know how to calculate t-statistics and p-values for linear regression, but I'm trying to understand a step in the derivation. I understand where Student's t-distribution comes from, namely I can show that if $Z$ is a random variable drawn from a standard normal distribution $Z \sim \mathcal{N}(0,1)$ and if $\chi$ is drawn from a $\chi^2_k$ distribution, then the new random variable

$$ T = \frac{Z}{\sqrt{\frac{\chi^2}{k}}} $$

will be drawn from a t-distribution with $k$ degrees of freedom.

The part that confuses me is the application of this general formula to the estimates for the coefficients of a simple linear regression. If I parametrize the linear model as $y = \alpha + \beta x + \varepsilon$, with $\varepsilon$ a random variable with zero mean characterizing the errors, then the best estimate for $\beta$ is

$$ \hat{\beta} = \frac{\bar{x y} – \bar{x} \bar{y} }{\bar{x^2} – \bar{x}^2 } \, , $$

where the bar's indicate sample means. The standard error of $\hat{\beta}$ is

$$ SE(\hat{\beta})^2 = \frac{\sigma^2}{n \left(\bar{x^2} – \bar{x}^2 \right) } \,. $$

Here $\sigma = \sqrt{\operatorname{Var}(\varepsilon)}$. The part I am confused about is why

$$ t \equiv \frac{\hat{\beta}}{SE(\hat{\beta})} $$

is taken to be drawn from a t-distribution, assuming the null hypothesis. If I could write $t$ in the form of the above variable $T$, cleanly identifying the $Z$ and the $\chi$ variables, then everything would be clear.

Best Answer

$$ Z = \frac{\widehat\beta - \beta}{\left( \dfrac \sigma { \sqrt{ n \left( \,\overline{(x^2)} - (\,\overline{x}\,\right)^2}} \right)} \sim \mathrm{N}(0,1). $$ And $$ (n-2) \frac{\widehat{\sigma}^2}{\sigma^2} \sim \chi^2_{n-2}. $$ Notice that $\sigma$ appears in both the numerator and the denominator of $Z/\sqrt{\chi^2_k/k}$ and cancels out.

Independence of these two things is seen by observing that the vector of residuals is independent of the vector of fitted values. To see that, find the covariance between the vector of residuals and the vector of fitted values, and recall that if two random vectors are jointly normally distributed then they are independent if they are uncorrelated.

(The whole story of why these things have the distributions asserted here would take somewhat longer.)

Related Question