The question goes as:
Let $\sigma_1(A) \geq \sigma_2(A) \geq \cdots \geq \sigma_r(A)$ be all (non-zero) singular values of an order $n$ real matrix $A$. Prove:
\begin{align*}
\sigma_k(A) = \sup_{\substack{X \in \mathbb{R}^{k \times n}\\ X \neq 0}}\frac{\sigma_k(XA)}{\sigma_1(X)}, \quad
\sigma_{n + 1 – k}(A) = \inf_{\substack{X \in \mathbb{R}^{k \times n} \\ X \neq 0}}\frac{\sigma_1(XA)}{\sigma_k(X)},
\quad \forall k = 1, 2, \ldots, r.
\end{align*}
Take the first equality as example, by the definition of singular values and the representation of eigenvalues, I am able tho show that
$\sigma_k(A) \leq \sup_{X \in \mathbb{R}^{k \times n}}\frac{\sigma_k(XA)}{\sigma_1(X)}$. It seems difficult to obtain the reverse direction. In particular, I don't know how to use the condition that $A$ is symmetric.
Best Answer
$\newcommand{\real}{\mathbb{R}}$ $\newcommand{\diag}{\mathrm{diag}}$
I will expand @J.Loreaux excellent comment to a more detailed answer for future references. Also, I will take @user8675309's suggestion to allow the singular values to be zero.
For any $1 \leq k \leq r$ and $X \in \real^{k \times n}$, suppose $$X^TX = O\diag(\sigma_1^2(X), \ldots, \sigma_n^2(X))O^T.$$ is the spectral decomposition of $X^TX$, where $O$ is an order $n$ orthogonal matrix. Recall the notation: for two symmetric matrices $A$ and $B$, $A - B \geq 0$ means the matrix $A - B$ is positive semi-definite. It then follows that
\begin{align*} & \Delta_k := \sigma_1^2(X)A^TA - (XA)^T(XA) = \sigma_1^2(X)A^TA - A^TX^TXA \\ = & A^T(\sigma_1^2(X)I_{(n)} - X^TX)A = A^TO\diag(0, \sigma_1^2(X) - \sigma_2^2(X), \ldots, \sigma_1^2(X) - \sigma_n^2(X))O^TA \geq 0. \end{align*}
Together with $A^TX^TXA \geq 0$, by Courant-Fischer Theorem (see this link), we then have for every $1 \leq k \leq r$: \begin{align*} \lambda_k(\sigma_1^2(X)A^TA) = \lambda_k(A^TX^TXA + \Delta_k) \geq \lambda_k(A^TX^TXA), \end{align*} which implies \begin{align*} \sigma_1(X)\sigma_k(A) \geq \sigma_k(XA). \end{align*} This shows \begin{align*} \sigma_k(A) \geq \sup_{\substack{X \in \real^{k \times n} \\ X \neq 0}}\frac{\sigma_k(XA)}{\sigma_1(X)}. \end{align*} The proof to the other inequality is analogous.