I have come across the following statement in https://www.sciencedirect.com/science/article/abs/pii/0031320394001256
The statement (in the right middle of Page 785) is (essentially) the following:
The vector $\boldsymbol{x}\in\mathbb{R^2}$ minimizing $\boldsymbol{x}^\top\boldsymbol{A}\boldsymbol{x}$, where $\boldsymbol{A}$ is a symmetric positive definite two-dimensional matrix is the same as the eigenvector corresponding to the smaller eigenvalue of $\boldsymbol{A}$.
I understand that the eigenvector corresponding to the larger eigenvalue maximizes the quadratic form. The eigenvector corresponding to the second (smallest in this case) eigenvalue, I thought, was the maximizer of the quadratic form that was also orthogonal to the eigenvector corresponding to the first eigenvalue. Is this also the minimizer? I am looking for a formal proof/explanation.
Best Answer
You seem to be very distracted by the $\mathbb{R}^2$ so let's generalize to $\mathbb{R}^n$. Write a unit vector $x = \sum_{i=1}^n c_i v_i$ as a sum of orthonormal eigenvectors of $A$, so that $\| x \|^2 = \sum c_i^2$. We assume that the eigenvalues have been sorted such that $\lambda_1 \ge \dots \ge \lambda_n$. Now write
$$\langle x, Ax \rangle = \left\langle \sum_{i=1}^n c_i v_i, \sum_{i=1}^n c_i \lambda_i v_i \right\rangle = \sum_{i=1}^n \lambda_i c_i^2.$$
Now the minimization problem is: what unit vector $x$ (you need this condition or else the minimum is just given by $x = 0$) minimizes this sum? Equivalently, we want to minimize $\sum \lambda_i c_i^2$ given the constraint $\sum c_i^2 = 1$. This is a very easy Lagrange multiplier calculation and the answer is that we should take $c_n^2 = 1$ and all other $c_i^2 = 0$, with minimum value $\lambda_n$, the smallest eigenvalue.
When $n = 2$ this smallest eigenvalue is $\lambda_2$ which also happens to be the second largest eigenvalue and hence which also solves a maximization problem.
It's not that "the maximization problem becomes a minimization problem," it's that there are two problems, a maximization problem and a minimization problem, and $\lambda_2$ happens to be the answer to both of them, because it happens to be both the smallest and the second largest eigenvalue in this case. But the distinction becomes clear for larger values of $n$ where $\lambda_2 \neq \lambda_n$.
More generally every singular value can be characterized as both the solution to a maximization problem and the solution to a minimization problem; see e.g. this blog post for one way to do this (search for "variational").