Consider a random vector $X\in\mathbb{R}^d$ that is distributed according to a multivariate Gaussian $\mathcal{N}(\mu, \Sigma)$, and the random variable $y = \langle a, X \rangle + e$ for some (fixed) vector $a\in\mathbb{R}^d$ and random variable $e\in\mathbb{R}$ that is independent of $X$ and distributed according to a centered univariate Gaussian $\mathcal{N}(0, \sigma_e^2)$.
It is well-known that $y$ is therefore distributed according as a $\mathcal{N}(a^T \mu, a^T \Sigma a + \sigma_e^2)$.
I am interested in the eigenvalues of the covariance matrix of the random vector $F = \begin{bmatrix} X \\ y \end{bmatrix} \in\mathbb{R}^{d+1}$ which also has a multivariate Gaussian distribution, according to the above.
This covariance matrix (let's call it $Z$) is a block matrix, as follows: $$ Z = \begin{bmatrix}
\Sigma & B \\
B^T & \sigma_e^2 + a^T \Sigma a
\end{bmatrix} $$
for some appropriate vector $B$ that is the covariance of $X$ and $y$.
If it is not possible to exactly analytically find the largest eigenvalue of $Z$, then I would like to arrive at a (good) upper bound of it. I suspect that it would look something like $\lambda_\text{max}(\Sigma)$ plus something like $\sigma_e^2 + a^T \Sigma a$.
Best Answer
Firstly, $B=\Sigma a$, because $\sigma$ is a symmetric positive definite matrix, there are an orthonormal matrix $U \in \mathbb{R}^{d\times d}$ and a positive diagonal matrix $\Lambda \in \mathbb{R}^{d\times d}$such that $U^T \Lambda U= \Sigma$, hence $$\begin{align}\begin{bmatrix} \Sigma & B \\ B^T & \sigma_e^2 + a^T \Sigma a \end{bmatrix} &= \underbrace{\begin{bmatrix} U^T & 0 \\ 0 & 1 \end{bmatrix}}_{=:C} \begin{bmatrix} \Lambda & \Lambda Ua \\ a^TU^T\Lambda & \sigma_e^2 + a^T U^T \Lambda Ua \end{bmatrix} \underbrace{\begin{bmatrix} U & 0 \\ 0 & 1 \end{bmatrix}}_{=C^T} \\ &= C^T \underbrace{\begin{bmatrix} \sqrt{\Lambda} & 0\\ a^TU^T\sqrt{\Lambda} & \sigma_e \end{bmatrix}}_{=D} \begin{bmatrix} \sqrt{\Lambda} & \sqrt{\Lambda} Ua \\ 0 & \sigma_e \end{bmatrix} C\\ &=C^TDD^TC \end{align}$$ Because $C$ is also an orthonormal matrix, $\|Z\|=\| DD^T\|= \| D\|^2$, where $\| \cdot \|$ is the usual matrix norm. Besides, $$D=\left\|\begin{bmatrix} \sqrt{\Lambda} & \sqrt{\Lambda} Ua \\ 0 & \sigma_e \end{bmatrix} \right\| \le \left\|\begin{bmatrix} \sqrt{\Lambda} & 0 \\ 0 & \sigma_e \end{bmatrix} \right\|+\left\|\begin{bmatrix} 0 & \sqrt{\Lambda} Ua \\ 0 & 0 \end{bmatrix} \right\| =\max( \| \Lambda \|,\sigma_e)+\sqrt{a^TU^T\Lambda U a}=\sqrt{\max(|\Sigma|,\sigma_e^2)}+\sqrt{ a^T\Sigma a} $$
So this gives a loose upper bound.
$$\|Z\| \le \left(\sqrt{\max(|\Sigma|,\sigma_e^2)}+\sqrt{ a^T\Sigma a}\right)^2$$
P.s: I knowledge on linear algebra is rusty so feel free to correct me.