Solved – VIF(collinearity) vs Correlation

correlationmulticollinearityvariance-inflation-factor

I am trying to understand the basic difference between both . As per what i have read through various links, previously asked questions and videos –

Correlation means – two variables vary together, if one changes so does the other but it does not imply collinearity or that one can explain the other.

VIF – Inflation in the variance of the regression coefficients ?( due to the col-linearity existing among predictors )

I am still confused around –

Variance inflation we mean is how inflated the regression coefficients are due to two or more collinear predictors ?
High VIF( > 10) implies high correlation but vice versa is not true ? Can this be explained with an example through variables ?

Best Answer

First, I think it is better to use condition indexes rather than VIF to diagnose collinearity. See the work of David Belsley or even (if you want a soporific) my dissertation (that link seems to have vanished; this one should work (I hope).

However, to get to your question: It is possible to have very low correlations among all variables but perfect collinearity. If you have 11 independent variables, 10 of which are independent and the 11th is the sum of the other 10, then correlations will be about 0.1 but collinearity is perfect. So, high VIF does not imply high correlations.

It is also true that you can have pretty high correlations without it creating troublesome collinearity, but this is trickier to show. See the references.

Related Solutions

Solved – How to test and avoid multicollinearity in mixed linear model

For VIF calculation usdm can also be package ( I need to install "usdm")

library(usdm)
df = # Data Frame
vif(df)

If VIF > 4.0 then I generally assume multicollinearity remove all those Predictor Variables before fitting them into my model

Solved – How to evaluate collinearity or correlation of predictors in logistic regression

Variable selection based on "significance", AIC, BIC, or Cp is not a valid approach in this context. Lasso (L1) shrinkage works but you may be disappointed in the stability of the list of "important" predictors found by lasso.

The simplest approach to understanding co-linearity is variable clustering and redundancy analysis (e.g., in the R Hmisc package functions varclus and redun). This approach is not tailored to the actual model you use. Logistic regression uses weighted $X'X$ calculations instead of regular $X'X$ considerations as used in variable clustering and redundancy analysis. But it will be close. To tailor the co-linearity assessment to the actual chosen outcome model, you can compute the correlation matrix of the maximum likelihood estimates of $\beta$ and even use that matrix as a similarity matrix in a hierarchical cluster analysis not unlike what varclus does.

Various data reduction procedures, the oldest one being incomplete principal components regression, can avoid co-linearity problems at some expense of interpretability. In general, data reduction performs better than all stepwise variable selection algorithms because of the direct way that data reduction handles co-linearity.

You can get VIFs in logistic regression. See for example the vif function that can be applied to lrm fits in the R rms package.

Best Answer

Related Solutions

Solved – How to test and avoid multicollinearity in mixed linear model

Solved – How to evaluate collinearity or correlation of predictors in logistic regression

Related Question