I want to show that in a simple linear regression ($y = X\beta + \epsilon$), when the mean is subtracted from any feature (i.e. some column of the matrix $X$), the t-statistics of all the non-intercept coefficients do not change. The t-statistic of any coefficient is given by $\frac{\hat{\beta_j}}{\hat{\sigma}\sqrt{\nu_j}}$. It is clear to me that $\hat{\sigma}$ (square root of residual sum of square) does not change when the mean is removed from any feature. It is also clear to me that $\hat{\beta_j}$ does not change.
Any hints on how to show that $\nu_j$ does not change? Note that $\nu_j$ is the $j$-th diagonal element of the matrix $(X^TX)^{-1}$. Also note that the matrix $X$ includes the intercept term i.e. the first column of $X$ is all ones.
In fact, through testing with examples, I think the following is true: Let $X$ be any $n \times p$ full-ranked matrix where $n \ge p$. Let the first column of $X$ be all ones. Let $A$ be an $n \times p$ matrix such that all columns are zero except the last column, which is a column of all $\mu$. Then, for any real $\mu$, $[(X^TX)^{-1}]_{jj} = [((X-A)^T(X-A))^{-1}]_{jj}$ $\forall j>1$, where $jj$ denotes the $j$th diagonal element of the matrix.
Best Answer
Write $X=(x_{ij})_{i=1,...,n; j=1,...,p}$. The first column ($j=1)$ contains only $(1,1.,,,1)\in R^n$ because this is the intercept. Say, you center all the columns $j=2,...,p$, i.e. you build the matrix $\tilde X = (\tilde x_{ij})$ with $\tilde x_{ij} = x_{ij} - \bar x_j$ where $\bar x_j$ is the mean $\frac 1 n \sum_{i=1}^n x_{ij}$. The question is whether the $i$-th element of $(\tilde X^T \tilde X)^{-1}$ is the same as the $i$-th element of $(X^T X)^{-1}$.
So $X e_1$ is the vector of size $n$ with all entries equal to one, and let $\bar x\in R^p$ the vector with entries $0, \bar x_2,...,\bar x_p$. Then the construction of $\tilde X$ is equivalent to $\tilde X = X -(X e_1)\bar x^T=X(I_p - e_1 \bar x^T)$.
Now $\tilde X^T \tilde X = (I_p - \bar x e_1^T) X^TX (I_p - e_1 \bar x^T)$ and one can check that $(I_p - \bar x e_1^T) $ is invertible, because the image of canonical basis vectors are linearly independent: $(I_p - \bar x e_1^T)e_j = e_j$ for $j\ge 2$ and $(I_p - \bar x e_1^T)e_1$ has the first component equal to 1 due to $\bar x$ having 0 in the first component.
For $j\ge 2$, $$e_j^T (\tilde X^T\tilde X)^{-1} e_j = e_j^T (I_p - e_1 \bar x^T)^{-1}(X^T X)^{-1} (I_p - \bar x e_1^T)^{-1} e_j$$ and one can check that $(I_p - \bar x e_1^T)^{-1} e_j = e_j$ because $e_1^T e_j= 0$. This implies that $\tilde X^T\tilde X$ and $X^TX$ have the same diagonal elements for $j\ge 2$.