I am studying about multicollinearity in regression and in the book it says, "if there is severe (but not perfect) multicollinearity, two or more predictor variables are highly correlated, so $X^TX$ is (computationally) difficult to invert. This produces unstable regression estimates and large standard error."
Could anyone explain what makes it computationally difficult? Any mathematical explanation of the fact would be really helpful.
Best Answer
The MLE of the regression coefficient $\beta$ satisfies $$\hat\beta\sim\mathcal{N}(\beta,\sigma^2(X^\text{T}X)^{-1})$$Hence, if two or more columns of $X$ are highly correlated, one or more eigenvalue(s) of $(X^\text{T}X)$ is close to zero and one or more eigenvalue(s) of $(X^\text{T}X)^{-1})$ is very large. This means that there exists a vector $e_1$ with $\|e_1\|=1$ such that $\text{var}(e_1^\text{T}\hat{\beta})$ is very large.