I have a large set of predictors (more than 43,000) for predicting a dependent variable which can take 2 values (0 or 1). The number of observations is more than 45,000. Most of the predictors are unigrams, bigrams and trigrams of words, so there is high degree of collinearity among them. There is a lot of sparsity in my dataset as well. I am using the logistic regression from the glmnet package, which works for the kind of dataset I have. My problem is how can I report p-value significance of the predictors. I do get the beta coefficient, but is there a way to claim that the beta coefficients are statistically significant?
Here is my code:
library('glmnet')
data <- read.csv('datafile.csv', header=T)
mat = as.matrix(data)
X = mat[,1:ncol(mat)-1]
y = mat[,ncol(mat)]
fit <- cv.glmnet(X,y, family="binomial")
Another question is:
I am using the default alpha=1, lasso penalty which causes the additional problem that if two predictors are collinear the lasso will pick one of them at random and assign zero beta weight to the other. I also tried with ridge penalty (alpha=0) which assigns similar coefficients to highly correlated variables rather than selecting one of them. However, the model with lasso penalty gives me a much lower deviance than the one with ridge penalty. Is there any other way that I can report both predictors which are highly collinear?
Best Answer
There is a new paper, A Significance Test for the Lasso, including the inventor of LASSO as an author that reports results on this problem. This is a relatively new area of research, so the references in the paper cover a lot of what is known at this point.
As for your second question, have you tried $\alpha \in (0,1)$? Often there is a value in this middle range that achieves a good compromise. This is called Elastic Net regularization. Since you are using cv.glmnet, you will probably want to cross-validate over a grid of $(\lambda, \alpha)$ values.