Convert SVM into convex QP

convex optimizationoptimizationquadratic programming

I have the following minimization:

$$\min\limits_{\mathbf{x}}\frac{1}{2}\|\mathbf{x}\|^{2}_{2}+\frac{1}{n}\sum_{i=1}^{n}\max(0,1-\mathbf{x}^{T}\mathbf{a}_{i})$$

The problem asks me to show that it is convex, which is fine and I did without any problems, and then it asks me to rewrite this problem as a convex QP, in order to find its Lagrangian and KKT conditions, The last step of the exercise is to show that the dual problem can also be written as a QP, but with box constraints.

Here are the steps I took in order to convert the primal as a QP:

Let us define $p_i=\max(0,1-\mathbf{x}^{T}\mathbf{a}_{i})$, which can be translated into the constraints:

$$\begin{align}
-p_i&\leq0\\
1-p_i&\leq\mathbf{x}^{T}\mathbf{a}_{i}
\end{align}$$

or

$$\begin{align}
-\mathbf{p}&\leq\mathbf{0}\\
-\mathbf{p}-\mathbf{A}\mathbf{x}&\leq\mathbf{1}
\end{align}$$

And define the vector $\mathbf{z}=\left[\mathbf{x} \quad \mathbf{p}\right]^{T}$

Then we can rewrite the problem as:

\begin{align}
&\min\limits_{\mathbf{z}}\frac{1}{2}\mathbf{z}^{T}\mathbf{P}\mathbf{z}+\mathbf{q}^{T}\mathbf{z}\\
&\text{ s.t. }\quad\mathbf{C}\mathbf{z}\leq \mathbf{b}
\end{align}

Where

$$\mathbf{P}=\begin{bmatrix}\mathbf{I} & \mathbf{0}\\
\mathbf{0} & \mathbf{0}\end{bmatrix},$$

$$\mathbf{q}=\begin{bmatrix}\mathbf{0}\\
\frac{1}{n}\mathbf{1}\end{bmatrix},$$

$$\mathbf{C}=\begin{bmatrix}\mathbf{0} & -\mathbf{I}\\
-\mathbf{A} & -\mathbf{I}\end{bmatrix},$$

and

$$\mathbf{b}=\begin{bmatrix}\mathbf{0}\\
-\mathbf{1}\end{bmatrix}.$$

The constraint is convex, but $\nabla_{z}^{2}f(\mathbf{z})=\mathbf{P}\succeq0$, which is only positive semi-definite. However, in order for my QP to be convex, I would need $\nabla_{z}^{2}f(\mathbf{z})=\mathbf{P}\succ0$, right?

What did I do wrong? Is there another way to formulate this problem in order to achieve the positive definiteness of my Hessian?

Best Answer

A function is convex if its Hessian is positive semidefinite, and strictly convex if its Hessian is positive definite. Your problem is strictly convex in $x$, but linear (and hence, convex) in $p$.

The second block element of $\mathbf{b}$ should be $-\mathbf{1}$.