[Math] Solve Quadratic Programming Problem with Linear Underdetermined Equality Constraints

convex optimizationoptimizationquadratic programming

I have an underdetermined system of equations $ C x = d $ (to be solved for $ x $). I want to find $ x $ such that it makes $ {x}^{T} B x $ minimum. What options do I have to solve this?

$ B $ and $ C $ are sparse matrices, but they can have arbitrary spectrums. If this makes it complicated, assume the simplest case (for example positive-definiteness of $ B $).

The problem can be written as:

$$\begin{align*}
\arg \min_{x} \quad & {x}^{T} B x \\
\text{subject to} \quad & C x = d
\end{align*}$$

Best Answer

The problem is given by:

$$\begin{align*} \arg \min_{x} \quad & {x}^{T} B x \\ \text{subject to} \quad & C x = d \end{align*}$$

Where $ B $ is Positive Semi Definite (PSD) matrix.

Since $ B $ is PSD one could write the problem as:

$$\begin{align*} \arg \min \quad & {\left\| A x - b \right\|}_{2}^{2} \\ \text{subject to} \quad & C x = d \end{align*}$$

Where $ {A}^{T} A = B $ (Since it is PSD it is guaranteed such $ A $ exists) and $ b = \boldsymbol{0} $.

Now, this a simple Least Squares problem with Linear Equality Constraints.

The Lagrangian is given by:

$$ L \left( x, \nu \right) = \frac{1}{2} \left\| A x - b \right\|_{2}^{2} + {\nu}^{T} \left( C x - d \right) $$

From KKT Conditions the optimal values of $ \hat{x}, \hat{\nu} $ obeys:

$$ \begin{bmatrix} {A}^{T} A & {C}^{T} \\ C & 0 \end{bmatrix} \begin{bmatrix} \hat{x} \\ \hat{\nu} \end{bmatrix} = \begin{bmatrix} {A}^{T} b \\ d \end{bmatrix} $$

Since $ B = {A}^{T} A $ and $ b = \boldsymbol{0} $ the above reduces to:

$$ \begin{bmatrix} B & {C}^{T} \\ C & 0 \end{bmatrix} \begin{bmatrix} \hat{x} \\ \hat{\nu} \end{bmatrix} = \begin{bmatrix} \boldsymbol{0} \\ d \end{bmatrix} $$

Now all needed is to solve the above with any Linear System Solver.
If you want to use iterative procedure you may use Linear Least Squares with Linear Equality Constraints - Iterative Solver (With included MATLAB Code).

Related Question