For a function $f$ having Hessian $H_f$, I have read at a few places that if $H_f$ is positive-semidefinite at a point, then $f$ will either have a local minima or a saddle point. But $f$ cannot have a local maxima at that point.
I just can't understand why is that so conceptually. Why can't $f$ have a local maxima under any circumstances at that point?
Best Answer
Technically if $f$ is constant, then the Hessian is the zero matrix (which is PSD), so every point is a local maximum and a local minimum, although these are not strict maxima/minima. It is also possible to have a strict local maximum when the Hessian is zero: consider $f(x) = -x^4$ at $x=0$. (Thanks Robert Israel.)
Excluding this edge case of a zero Hessian however, you cannot have a local maximum. Suppose the Hessian $H_f$ at $x_0$ has a strictly positive eigenvalue. Let $v$ be the associated eigenvector. Then the univariate function $g(t) = f(x_0 + tv)$ has second derivative equal to $g''(t) = v^\top H_f v > 0$. Since the curvature of $g$ at zero is positive, there exist points $t$ arbitrarily close to $0$ such that $g(t)>g(0)$. This means $f(x_0 + tv) > f(x_0)$ for such $t$ as well.