Solved – Fit logistic regression with linear constraints on coefficients in R

Question

This is (I think) a routine question, but I haven't seen an answer here. In the past I had to fit logistic regression models with linear constraints on the coefficients, in the case when some of the predictors were categorical. Until now I've been using glmnet, which by the way only allows bound constraints ($a_i\leq\beta_i\leq b_i $). However, apart from the limitation on the type of constraints, I find it annoying to be forced to use glmnet matrix/column specification for data, instead than a more expressive formula as done in glm. Also, most of the times I only needed to fit an unregularized logistic regression model ($\lambda=0$). However, glmnet help warns against fitting the model for a single value of $\lambda$. Thus I had to fit a sequence of models, parameterized by a sequence of $\lambda$, and then extract the model corresponding to $\lambda=0$. It gets boring the $n-$th time you have to do it.

For all these reasons I'm looking into other solutions to fit unregularized logistic regression models, with linear constraints on the coefficients. One could of course derive the objective function for logistic regression (as done here) and then use an optimizer in R which supports convex optimization problems and linear constraints (any suggestions?). However, this is complicated and I'm not sure how I would setup the optimization to take into account the categorical predictors. Is there a simpler way to achieve my goal? If not, can you show me in detail how to do this with optimization ?

Answer 1

The question is not clear to me. But if you want to use formula instead of matrix input. You can use caret pacakge, which provide universal interface to many packages including glmnet. Or you can use model.matrix to convert formula to matrix form.

If you want to fit a logistic regression with $\lambda=0$, you can directly use traditional glm. If you want to manually implement it, my answer here has the code to to calculate the objective function and gradient of logistic loss and use BFGS to optimize.

---

For the revised question: projected gradient descent may work well, since the "box constraints" is easy to solve. My answer here gives some details on projected gradient descent. (I believe projected gradient descent is used by glmnet).

Solving constrained optimization problem: projected gradient vs. dual?

Additional reference for projected gradient descent, book from the author of glmnet: Statistical Learning with Sparsity page 120