[Math] Quadratic Form with $ {L}_{1} $ Regularization and Box Constraints – Coordinate Descent Iteration

convex optimizationoptimization

Coordinate descent is a powerful method for solving optimization problems like

$$\min_x \tfrac{1}{2}x^T A x + b^T x + \lambda ||x||_1$$

where $A$ is symmetric and positive definite, $\lambda>0$ and $||\cdot||_1$ is the 1-norm on ${\bf R}^n$.

Say that I want to solve a version of the problem with constraints, for example coordinate-wise inequality constraints

$$l_i \leq x_i \leq u_i$$

or equality constraints

$$x_i = c_i$$

for some or all coordinates. In principle these could be general linear equality or inequality constraints, but I'd be happy with coordinate-wise constraints.

For unconstrained coordinate descent there is a simple algorithm

while not converged:
    for i = 1:n
        set x(i) to the value that minimizes the obj function, holding
        other coordinates fixed.

What is the corresponding algorithm with coordinate-wise equality or inequality constraints?

Best Answer

Coordinate-wise equality constraints are trivial. Just fix $x_i=c_i$ and skip that coordinate during the search. More complex equality constraints are likely not possible here. After all, it's unlikely that you will be able to hold all of the other coordinates fixed and maintain feasibility.

Coordinate-wise bounds are not difficult to deal with, either. When you are working with coordinate $i$, you simply have to respect those corresponding inequality constraints. So you can descend in an unconstrained fashion, but stop if the descent direction takes you outside of the interval $[l_i,u_i]$.

If, at a given iteration, you find that you cannot take a step in a descent direction without violating the constraint, then you don't: $x_i$ remains fixed, and you move to the next coordinate. So for instance, suppose $x_i\neq 0$; then the partial derivative is $$\frac{\partial f}{\partial x_i} = x^TAe_i + b^Te_i + \lambda \mathop{\textrm{sign}}(x_i)$$ If $x_i=l_i$ and this quantitiy is positive, or if $x_i=u_i$ and this quantity is negative, then the value of $x_i$ will not change in this iteration. If $x_i=0$, of course, then you have a subgradient to deal with, but the point is the same.

More complex linear inequality constraints should be possible to handle as long as they define a polyhedron with a non-empty interior (so, e.g., there cannot be "opposing" inequalities creating an effective equality constraint.) Consider constraints $Fx\leq g$, and suppose that the current point $x$ is feasible (that is, $Fx\leq g$), and we're scheduled to move coordinate $i$. Then we seek $[t_\min,t_\max]$ as follows: $$t_\min=\inf\{t~|~F(x+te_i)\leq g\}, \quad t_\max=\sup\{t~|~F(x+te_i)\leq g\}$$ This can be solved analytically; and because $Fx\leq g$, know that $t_\min\leq 0$ and $t_\max\geq 0$. Then we can define $$l_i = x_i + t_\min, \quad u_i = x_i + t_\max$$ and proceed with a coordinate descent constrained to the interval $[l_i,u_i]$, just as we did above.

Related Question