Suppose I am training a linear model. What are the conceptual differences between using a diagonal covariance matrix and the identity? It is clear to me that the difference between a full covariance matrix and a diagonal covariance matrix is that there is no correlation between predictors with the diagonal matrix. I'm not quite sure of the differences between the identity and diagonal matrices though.
Regression – Difference Between Identity and Diagonal Covariance Matrices
correlationcovarianceregression
Related Question
- Solved – Perfect correlation and covariance matrices
- Solved – the difference between the anti-image covariance and the anti-image correlation
- Regression – Difference Between Correlation and Covariance: Is Covariance Only Useful if the Relationship is Linear?
- Solved – How to include off-diagonal elements in covariance matrix in uncertainty of variable
Best Answer
An identity covariance matrix, $\Sigma=I$ has variance = 1 for all variables.
A covariance matrix of the form, $\Sigma=\sigma^2I$ has variance = $\sigma^2$ for all variables.
A diagonal covariance matrix has variance $\sigma^2_i$ for the $i^\text{th}$ variable.
(All three have zero covariances between variates)