[Math] Definition of stationary points for convex optimization

convex optimizationoptimization

I am trying to understand some concepts of convex optimization,
whereby I stumbled upon the following definition of stationary points:

Considering the following constrained optimization problem:

$\min_x f(x), \quad\text{s.t.} \quad x \in C, \quad (P)$

Where f is a continuously differentiable function defined over the closed and convex set $C \subset \mathbb{R}^n$.

Let f be a continuously differentiable function over closed set $C$.
Then $x^* \in C$ is called a stationary point of the minimization problem (P) if $\nabla f(x^*)^T(x-x^*) \geq 0$ for any $x \in C$.

I don't really understand the definition, to my understanding for
a stationary point $\nabla f(x^*)=0$ is given, but in this case defining
$\nabla f(x^*)^T(x-x^*) \geq 0$ would make no sense to me, therefore
I think I am missing a substantial point here.

Best Answer

For me, the best way to intuitively understand this definition is via directional derivatives. A well known result is that for a continuously differentiable function, the directional derivative of $f$ in the direction $d$, which measures its slope along the ray pointing from some point in direction $d$, is: $$ f'(x; d) = \nabla f(x)^T d. $$

The vectors $x - x^*$ for all $x \in C$ point from the point $x^*$ towards all the points of $C$, and the stationarity conditions $$ \forall x \in C \quad \nabla f(x)^T (x - x^*) \geq 0 $$ means that the function's slope along any direction which points from $x^*$ toward a point in $C$ is non-negative. That is, moving a small distance from $x^*$ toward any point in $C$ does not decrease the function values. In other words, at $x^*$ there are no feasible descent directions.

It is also easy to prove that when $x^*$ is an interior point of $C$, then this condition reduces to the well known condition of $\nabla f(x) = 0$, since for very small vectors $d$ both $x^* + d$ and $x^* - d$ are in $C$, and we get: $$ \nabla f(x^*)^T d \geq 0, \quad \nabla f(x^*) (-d) \geq 0. $$

Related Question