[Math] Showing Orthogonal Projection Matrix Multiplied by Full-Rank Matrices is Positive-Definite

linear algebramatricesmatrix-rankpositive definiteprojection-matrices

Here is some useful information pertaining to my question:

  • Let $X \in\mathbb{R}^{n \times m}$ and $Z \in\mathbb{R}^{n \times p}$ be full-rank matrices.
  • Define $B = I_n – X(X^{T}X)^{-1}X^{T}$.
  • Assume that the columns of $X$ are linearly independent from the columns of $Z$.

I am trying to show that $Z^{T}BZ$ is positive definite.

My first plan of attack was to (1) show that $Z^{T}BZ$ is an orthogonal projection matrix, (2) prove that $I$ is the only positive definite orthogonal projection matrix, and (3) prove that $Z^{T}BZ=I$. This fell through because I felt that there was not enough information about $Z$ to prove (1).

I am pretty sure that I will ultimately need to show either (i) $z^{T}Z^{T}BZz>0$ $\forall$ $z\in \mathbb{R}^p$ OR (ii) all the eigenvalues of $Z^{T}BZ$ are positive. I think my biggest problem stems from not knowing how to deal with $Z^{T}$ and $Z$… clearly they are important, as it seems that we can only prove that $B$ is positive semidefinite. However, I am unsure how this full-rank matrix can "transform" $B$ from psd to pd.

Any assistance would be greatly appreciated! If it helps, I have already shown that B is an orthogonal projection matrix and that B is psd. 🙂

Best Answer

Hmm, I am a bit surprised that no one went and finished the proof.

So how do we prove that $ ( I - X ( X^T X )^{-1} X^T ) Z $ has linearly independent columns? Assume that it doesn't and show that this results in a contradiction. We will look at $ Z - X ( X^T X )^{-1} X^T Z $ instead.

Let $ x $ be such that $ (Z - X ( X^T X )^{-1} X^T Z) x = 0 $

  • Maybe $ Z x = 0 $. But that can't be because $ Z $ has linearly independent columns. So, $ Z x \neq 0 $.

  • Maybe $ X^T ( Z x ) = 0 $. That can't be since then $ Z x - X ( X^T X )^{-1} X^T Z x = Z x = 0 $ because $ Z $ has linearly independent columns. So, $ X^T Z x \neq 0 $. This also means $ y = ( X^T X )^{-1} X^T Z x \neq 0 $.

This reasoning tells us that $ Z x - X y = 0 $ or, equivalently, $ Z x = X y $ for both $ x \neq 0 $ and $ y \neq 0 $. But that in turn means that the columns of $ Z $ are not linearly independent of the columns of $ X $ since then $$ \left( \begin{array}{c | c} Z & X \end{array} \right) \left( \begin{array}{c} x\\-y \end{array} \right) = 0. $$

Related Question