[Math] Pseudo-inverse of a matrix that is neither fat nor tall

linear algebramatricespseudoinverse

Given a matrix $A\in\mathbb R^{m\times n}$, let us define:

  • $A$ is a fat matrix if $m\le n$ and $\text{null}(A^T)=\{0\}$
  • $A$ is a tall matrix is $m\ge n$ and $\text{range}(A)=\mathbb R^n$

Using the finite rank lemma, we can find that:

  • When $A$ is a fat matrix, its (right) pseudo-inverse is $A^\dagger = A^T(AA^T)^{-1}$
  • When $A$ is a tall matrix, its (left) pseudo-inverse is $A^\ddagger = (A^TA)^{-1}A^T$

My question is what is the pseudo-inverse when $A$ is neither fat nor tall (in the sense of the above definitions), i.e. it is a matrix such that $\text{null}(A^T)\ne \{0\}$ (i.e. the null space is non-trivial) and $\text{range}(A)\ne\mathbb R^n$? An example of such a matrix is:

$$
A = \begin{bmatrix}
1 & 1 & 0 \\
0 & 2 & 0 \\
0 & 0 & 0 \\
0 & 3 & 0
\end{bmatrix}
$$

which clearly does not map to full $\mathbb R^4$ and whose null space is $\text{span}\left\{\begin{bmatrix}0 \\ 0 \\ 1\end{bmatrix}\right\}$.

Best Answer

There is no need to study both the fat matrix and the tall matrix. Let $B = A^t$. In that case, the tall matrix problem for $B$ is the fat matrix problem for $A$ and vice versa.

Thus, we consider only the tall matrix case. Assume that the rank of $A$ is $r$ where $r \leq n$. Let the SVD (singular value decomposition) of $A$ is defined as $A = U \Sigma V^t$ where $U$ is an $m \times r$ matrix with orthogonal columns, $V$ is an $r \times n$ matrix with orthogonal columns and the diagonal $r \times r$ matrix $\Sigma$ contains the $r$ singular values in the diagonal. These diagonal values $\sigma_i$, $i=1 \colon r$ are ordered in the non-increasing order.

The Moore-Penrose pseudo inverse is then given by $$ A^{+} = V \Sigma^{-1} U^t . $$ This pseudo inverse exists even when $A^tA$ is singular.

Numerical analysts will tell you to define the "zero singular values" as the singular values which are "tiny". Tiny means any number which is in the same order as $\|A\|*\epsilon$ {(any) norm of $A$ times the machine precision}. Unfortunately, the Moore-Penrose inverse often depends on the way "tiny" is defined. Mathematicians do not have this problem since zero means zero and nothing else.

For your matrix $A$, there are two non-zero singular values (3.7519 and 0.9610) and hence $r=2$. Since there are no tiny singular values, the Moore-Penrose inverse is well defined.

Related Question