[Math] Proof projection is invariant to the choice of basis

linear algebramatrices

Assume first $A$ has independent columns. We have a projection matrix $P$ on the range of $A$, i.e. $P_A = A(A^\top A)^{-1}A^\top$. Consider now a matrix $B$ which has the same range as $A$. We have $B = AC$, where $C$ is an invertible matrix. We also have the following.

$P_B = B(B^\top B)^{-1}B^\top = AC(C^\top A^\top AC )^{-1}C^\top A^\top = ACC^{-1}(A^\top A)^{-1} {C^{\top}}^{-1}C^\top A = P_A$

I am interested in having a similar result for the case where A does not have independent columns. Now, $A^\top A$ isn't invertible and the projection matrix becomes $P_A = A(A^\top A)^{\dagger}A^\top$, where we denote by $(\cdot)^\dagger$ the Moore-Penrose pseudoinverse. Consider now two cases:

  1. $B$ also has dependent columns and $B = AC$ for some invertible $C$.
  2. $B$ has independent columns and $B = AC$ for a skinny rectangular matrix $C$ such that $C^\top C = I$, i.e. $C$ selects a minimal subset of columns of $A$ to span the range. Edit: The columns of C are columns of the identity matrix.

In each case, I would like a proof that $A(A^\top A)^{\dagger}A^\top = B(B^\top B)^{\dagger}B^\top$. The problem with the above technique is that I can't simply pull $C$ out of $(C^\top A^\top AC )^{\dagger}$.

Edit: I see that the matrix $A(A^\top A)^{\dagger}A^\top = A A^\dagger$, but again, I don't see how to show that $A A^\dagger = AC (AC)^\dagger$.

Edit: I can also see the following justification: $P_A$ is the projection matrix on the range of $A$, and $P_B$ on the range of $B$. Since the range is the same, they also have to be the same by uniqueness of projection (@martini). But this is ugly, because we have to interpret the matrices as projections. Isn't there an $algebraic$ way to justify this?

Best Answer

If $A^{\top}$ is the transpose of $A$, we may have $A \neq 0$ and $P_A=0$. So, $P_A$ is not the projection on the range of $A$ in this case.

Example: If $A=\left( \matrix {1 & 1+i \\ i &i-1} \right)$, $A^{\top} A=0$, so $(A^{\top}A)^{\dagger}=0$ and $P_A=0$, but $A \neq0$.

So, we suppose $A^{\top}$ is the Hermitian transpose (or conjugate transpose). We have $P_A P_A=P_A$ and $P_A^{\top}=P_A$.

  1. If $B=AC$ with $C$ invertible. We have $A^{\top}A(A^{\top}A)^{\dagger}A^{\top}A=A^{\top}A$. So, $$P_B P_A P_B = B(B^{\top}B)^{\dagger}C^{\top}A^{\top}P_A AC (B^{\top}B)^{\dagger}B $$ $$=B(B^{\top}B)^{\dagger}C^{\top}A^{\top}AC (B^{\top}B)^{\dagger}B=P_B P_B=P_B $$

As $A=BC^{-1}$, we have $P_A P_B P_A=P_A$ too.

If $M$ is a matrix, $M^{\top}M=0 \implies M=0$.

If $M=(Id-P_B)P_A$, $M^{\top}M=P_A(Id-P_B)P_A=P_A-P_A P_BP_A=0$, so $M=0$ and $P_A-P_BP_A=0$, and $P_A=P_B P_A$. By hermitian transposition, $P_A=P_A P_B$.

We have $P_B=P_A P_B=P_B P_A$, so $P_A=P_B$.

Related Question