Are all mixed integer nonlinear problems non-convex? If not, is the problem non-convex? And what solvers can we use

mixed-integer programmingnonlinear optimization

My apologies for asking (basic) questions on mixed-integer nonlinear problems.

Question 1: Are all mixed-integer nonlinear problems non-convex? If not, why do people say it is very hard to solve it? If there are some convex mixed-integer nonlinear problems, then one could employ a general purpose solver such as CVX and MOSEK, correct?

\begin{alignat}{3}
& \underset{ X \in \mathbb{R}^{n \times n}, \ d \in [0,1]^{n \times 1} }{\text{minimize}}
& & -\sum_{m=1}^M d_m \\
& \text{subject to}
& & \quad X^TX = I \\
& & & \quad {\rm Tr}({\rm Diag}(d) X^T A X) \leq \alpha \\
& & & \quad {\rm Diag}(d) – {\rm Diag}(d^2) = 0
\end{alignat}

where $d$ is a binary-valued vector, ${\rm Diag}$ creates a diagonal matrix, and $\alpha \in \mathbb{R}$ is a given constant. $d^2$ operation is component-wise.

Question 2: Isn't the above problem non-convex? One of the reasons is the orthogonality constraint, right? (I have seen people are also referring the set of such a constraint as Stiefel manifold). I think the third constraint is also non-convex, right?

Question 3: Last question – is there any global optimization solver for such a problem?

Best Answer

A problem with integer variables is non-convex pretty much by definition of convexity (a set consisting of isolated points like $\{0,1\}$ is not convex). However, by a slight abuse of language, "mixed-integer convex problem" is used to describe mixed-integer problems where the continuous relaxation is convex. They are already hard because they contain the NP-hard class of mixed-integer linear problems. Like you say, a solver such as MOSEK, and a tool such as CVX can often be used to model and solve such type of problems.

Your problem is not of that form ie. it is even harder because the nonlinear constraints 1 and 2 are non-convex. (Constraint 3 is just a roundabout way to write that $d$ is binary, which you would communicate to the solver more directly anyway). Unless there is some special structure to exploit then you may need a general MINLP solver (see for example list of MINLP solvers in https://en.wikipedia.org/wiki/List_of_optimization_software)

Related Question