I'm following a course on convex optimization by Stephen P. Boyd. https://learning.edx.org/course/course-v1:StanfordOnline+SOE-YCVX0001+1T2020/
My question is about Lecture 3 (https://youtu.be/JrWSn5wW_Z0). Boyd gives a function:
quadratic-over-linear: $f(x,y) = x^2/y,\: y>0$
The Hessian matrix of the function is:
$$\nabla^2f(x,y)=\begin{bmatrix}2/y&&-2x/y^2\\-2x/y^2&&2x^2/y^3\end{bmatrix}=\frac{2}{y^3}\begin{bmatrix}y\\-x\end{bmatrix}\begin{bmatrix}y\\-x\end{bmatrix}^T$$
So the Hessian matrix has rank one. Boyd says at https://youtu.be/JrWSn5wW_Z0?t=1527 :
The fact that it's rank one tells you that at any point on that surface it's
sort of curving in one direction, but not the other.
Alright, so that's what it says. In one direction, it's flat.
In the other direction, it's got positive curvature.
I don't see why it's true for any point. For example, $\nabla^2f(1,1)= \begin{bmatrix}2&&-2\\-2&&2\end{bmatrix}$ . So the function has curvature in both $x$ and $y$ direction.
Maybe Boyd meant to say that this is true for any point $(x_0, y_0)$ where $\nabla f(x_0,y_0) = 0$ ?
In that case the line $x_0=0$ would represent the set of such points, and the Hessian matrix would be $\nabla^2f(0,y_0)= \begin{bmatrix}2/y_0&&0\\0&&0\end{bmatrix}$. This would mean that the function has curvature only in $y$ direction.
So, what am I missing here? Is the claim really true for any point? Is it true only for points where the gradient is zero? Is it true for any function in general?
Best Answer
Short answer.
If you make a parabolic trough out of paper that rests flat on a table on a single line of contact, and imagine walking in any direction that is not that purely flat direction on the table, you will note that you curve upward.
Long answer. It may be helpful to review how curvature is related to real-world experience. We know from experience that as we travel on a curved path on a surface at constant speed the force we experience due to acceleration has a component that pushes us up or down in relation to the surface normal. It reflects the fact that the surface is curved. To test for this curvature in a given tangent direction on the surface $f(x,y)-z=0$ imagine traveling on a parametrized path on the surface that passes through that given tangent direction. The Chain Rule tells you the tangent vector $\vec v =< \dot x, \dot y, \dot z>$ satisfies $ f_x \dot x+ f_y \dot y - \dot z=0$ from which we see that the normal vector direction is that of $N=< f_x,f_y, -1>$. Differentiating again we deduce $Q=f_{xx} \dot x^2+ + 2f_{xy} \dot x \dot y+ f_{yy}\dot y^2 =-\vec A \cdot N$ where $\vec A$ is the acceleration. Thus the Hessian quadratic form $Q= \vec v^t H \vec v$ can be interpreted as a way to measure curvature as we travel in any specified tangent direction $\vec v$.
Diagonalization theory summarized. Symmetric quadratic forms can always be expressed (using eigenvector, eigenvalue theory) in purest form as a linear combination of pure squares of certain orthogonal components of $\vec v$. So in particular our curvature formula can be written as $Q= \kappa_1 (\vec v\cdot e_1)^2 + \kappa_2(\vec v\cdot e_2)^2 $ where $e_1, e_2$ are two mutually orthogonal unit length tangent vectors. These directions are called the principal curvature directions on the surface. In the case where $\kappa_2=0$ but $\kappa_1\ne 0$ we say the surface is flat in that direction. It is actually flat in only that one direction. In any other direction it is curved, because there is a component of $\vec v$ in the direction of $\vec e_1$ where the surface is curving.