How do you calculate an estimate for the variance covariance matrix of a logistic regression with elastic net regularization?
Starting from the variance-covariance matrix of a plain vanilla logistic regression, how does the formula need to be augmented:
$\hat{\Sigma}=-\left[\hat{p}(1-\hat{p})XX^{T}\right]^{-1}$
where $\hat{p}=\frac{1}{1+exp(X^{T}\hat{\Theta})}$
If you don't know the answer for elastic net, how would this be implemented for a logistic regression with ridge regularization?
Update: This link suggests OLS ridge is:
M = $(X'X+λI)^{−1}X'$
var(β)= ${\sigma}^{2}MM'$
Best Answer
As stated in this paper, Eq. (12), the $(i, j)$ element of the variance-covariance matrix for logistic regression is $\frac{\partial^2l(\hat\beta)}{\partial\hat\beta_i\partial\hat\beta_j}$, where $l(\hat\beta)$ is the log-conditional likelihood of coefficients $\hat\beta$ given the observed data.
As stated in this paper, Eq. 2.1, the log-conditional likelihood for ridge-regularized logistic regression with parameter $\lambda$ is defined as $l^\lambda(\hat\beta) := l(\hat\beta) - \lambda\hat\beta'\hat\beta$.
Hence, if we define $V(\hat\beta)$ to be the variance-covariance matrix of non-regularized logistic regression, then the variance-covariance matrix of ridge-regularized logistic regression is $V^\lambda(\hat\beta) = V(\hat\beta) - 2\lambda I$.