I have encountered the following exercise:
Let $\lambda_1(A) \geq \lambda_2(A) \geq \cdots \geq \lambda_n(A)$ be all eigenvalues of the real symmetric matrix $A$. Prove:
$$\sup_{XX^T = I_{(k)}} \lambda_k(XAX^T) = \lambda_k(A).$$
My attempt: by taking $X = (I_{(k)}, 0)P^T$, where $P$ is an order $n$ orthogonal matrix such that $A = P\mathrm{diag}(\lambda_1(A), \ldots, \lambda_n(A))P^T$, I am able to show that LHS $\geq$ RHS. I am stuck with the reverse direction. I think the Courant-Fischer min-max principle should help but haven't worked it out completely.
Best Answer
$\newcommand{\real}{\mathbb{R}}$ $\newcommand{\rank}{\mathrm{rank}}$ $\newcommand{\Im}{\mathrm{Im}}$
Given what I have achieved in the question, it suffices to prove $\lambda_k(A) \geq \lambda_k(XAX^T)$ for any $k \times n$ matrix with $XX^T = I_{(k)}, k = 1, \ldots, n$. We fix $k$ in the subsequent arguments.
Viewing $X^T \in \real^{n \times k}$ as a linear transformation from $\real^k$ to $\real^n$, then $$\dim(\Im(X^T)) = \rank(X^T) = \rank(XX^T) = \rank(I_{(k)}) = k.$$ Moreover, since $XAX^T$ is an order $k$ symmetric matrix, it follows by Courant-Fischer Max-min principle (apply it twice) that \begin{align*} \lambda_k(XAX^T) &= \min_{y \in \real^k\backslash\{0\}}\frac{y^TXAX^Ty}{y^Ty} = \min_{y \in \real^k\backslash\{0\}}\frac{y^TXAX^Ty}{y^TXX^Ty} = \min_{z \in \Im(X^T)\backslash\{0\}}\frac{z^TAz}{z^Tz} \\ &\leq \max_{S: \dim(S) = k}\min_{z \in S\backslash\{0\}}\frac{z^TAz}{z^Tz} = \lambda_k(A). \end{align*}
This completes the proof.