(1) About a week ago I have asked the question and got a beautiful explanation. After that I started to consider the relationship of the singular values of the big matrix $A$ and the small matrix $A_{k,\alpha}$. We will denote the maximal singular value of $A$ by $\sigma_{\max}(A)$.
Specifically, if I have a block matrix
\begin{equation}
A=\left(
\begin{matrix}
0&0&\cdots&0&A_{1,\alpha}&0&\cdots&0\\
0&0&\cdots&0&A_{2,\alpha}&0&\cdots&0\\
\vdots&\vdots&\cdots&\vdots&\vdots&\vdots&\cdots&\vdots\\
0&0&\cdots&0&A_{\alpha,\alpha}&0&\cdots&0\\
\vdots&\vdots&\cdots&\vdots&\vdots&\vdots&\cdots&\vdots\\
0&0&\cdots&0&A_{s,\alpha}&0&\cdots&0
\end{matrix}
\right)_{N\times N}.
\end{equation}
Here, each $A_{k,\alpha},k=1,2,\cdots,s$ is a matrix of $N_k\times N_\alpha$ (Here $N_k$ and $N_\alpha$ could be infinity). Obviously $A_{\alpha,\alpha}$ is a square matrix of $N_{\alpha}\times N_{\alpha}$ and we assume that the diagonal line of $A_{\alpha,\alpha}$ is on the diagonal line of $A$. At first I still concerned the relationship of $\sigma_\max(A)$ and $\sigma_\max(A_{\alpha,\alpha})$ and made the similar guess. However, I have used software to find a counterexample where $$\sigma_{\max}(A)\not=\sigma_{\max}(A_{\alpha,\alpha}).$$ Now I conjectured that $$\sigma_\max(A)\le\sum_{k=1}^s\sigma_{\max}(A_{k,\alpha}).$$ I think it may hold in general but I haven't come up with a proof. I will highly appreciate if anyone could help me!
(2) Under a special assumption that $A$ is a block diagonal matrix
\begin{equation}
A=\left(
\begin{matrix}
A_{N_1}&0&\cdots&0\\
0&A_{N_2}&\cdots&0\\
\vdots&\vdots&\cdots&\vdots\\
0&0&\cdots&A_{N_s}
\end{matrix}
\right)_{N\times N},
\end{equation}
I have the following arguments:
$$\sigma_{\max}(A)=\sqrt{\lambda_{\max}(A^tA)}=\max_{k=1,\cdots,s}\sqrt{\lambda_{\max}(A_{N_k}^tA_{N_k})}=\max_{k=1,\cdots,s}\sigma_{\max}(A_{N_k}).$$
Am I right? Thanks anyway!
Best Answer
For 1, your conjecture is correct, at least as far as finite matrices are concerned. In fact, we have the slightly stronger statement $$ \sigma_{\max}(A)^2 \le \sum_{k=1}^s \sigma_{\max}(A_{k,\alpha})^2. $$ To see that this is the case, note that $\sigma_{\max}(A)^2$ is the maximal eigenvalue of $A^TA$, and that $A^TA$ is block-diagonal with a single non-zero block-entry given by $$ [A^TA]_{\alpha,\alpha} = \sum_{k=1}^s A_{k,\alpha}^TA_{k,\alpha}. $$ The fact that $\lambda_{\max}(\sum_{k}M_k) \leq \sum_k\lambda_{\max}(M_k)$ holds for symmetric matrices $M_1,\dots,M_s$ can be seen by considering the characterization $$ \lambda_{\max}(M) = \max_{\|x\| = 1}x^TMx. $$ Your argument for (2) is correct. In particular, it is indeed the case that $A^TA$ will be block diagonal with diagonal block-entries $$ [A^TA]_{k,k} = A_{N_k}^TA_{N_k}. $$