I was reading through Convex Optimization lecture notes: https://people.eecs.berkeley.edu/~elghaoui/Teaching/EE227BT/LectureNotes_EE227BT.pdf
On page 75, it describes the $\textbf{Largest Eigenvalue Problem}$ in which for any square matrix $A$ the value of $\max_{x^Tx = 1} x^TAx$ is said to be $\lambda_{max}(A)$ where $\lambda_{max}$ denotes the largest singular value. However, this is the Rayleigh Quotient where the maximum is known to be the largest eigenvalue, not singular value. I first assumed there was a typo in the notes, perhaps $A$ was also positive semi definite for example. But then, I stumbled upon this paper http://www.public.asu.edu/~hdavulcu/wanga14.pdf where for Eq. 5 a similar quadratic objective is given where it is stated that the maximum value also corresponds to the largest singular value.
Can someone relieve me from my confusion. Is it the terms singular value and eigenvalues are sometimes interchangebly used even though they're not same. Thank you.
Best Answer
Yeah, there seems to be a little bit of confusion here, but you seem to have the right idea. This might just boil down to a typo, that the author meant to say eigenvalue but instead typed singular value. You should ask the author about it.
For a symmetric matrix $A$, the singular values are the absolute values of the eigenvalues. So, as you point out, if $A$ is positive semidefinite, they are the same.
To point out where the result stated in the link is wrong, consider: $$A = \begin{pmatrix} -2 & 0 \\ 0 & 1 \end{pmatrix}$$ which has eigenvalues $-2, 1$, is symmetric and therefore has singular values $2, 1$. The quadratic form $x^TAx$ takes the form: $$\begin{pmatrix} x_1 & x_2 \end{pmatrix} \begin{pmatrix} -2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} x_1 & x_2 \end{pmatrix} \begin{pmatrix} -2x_1 \\ x_2 \end{pmatrix} = -2x_1^2 + x_2^2 $$ which for $x_1^2 + x_2^2 = 1$ takes its maximum value $1$ at $(x_1, x_2) = (0,1)$. So, the function never takes the value $2$, the value of the largest singular value.
Note however, that if you would compute $|x^TAx|$, the absolute value of the quadratic form, the maximum would be the largest singular value.
In your second link, note that the formulation is slightly different: Here we have two vectors $x,y$ as input, and we compute $y^TAx$. The value of the largest singular value can be achieved by using the singular value's left- and right-singular vectors.
For the example above, you can take $$y = \begin{pmatrix} -1 \\ 0 \end{pmatrix} \quad x = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$$ to achieve the value of $2$ for $y^T A x$.