Matrix decomposition is one area of matrices that has always intrigued me. Every time I open a matrix book, I can interestingly follow it till Eigen values and Eigen vectors because they are well motivated.
However, I am not able to understand the motivation of matrix decomposition techniques.
I am just revising on Matrices before I start learning linear and non linear optimization methods for machine learning.
It would be great if people here could motivate the need and benefits of matrix decomposition techniques like LU, QR, Cholesky, SVD etc. Also, any interpretations of matrix decomposition techniques (if any) would be great.
Thanks!
Best Answer
Eigenvalues and eigenvectors lead naturally to the spectral theorem, diagonalization of matrices, and Jordan normal form. This is all indeed very elegant and nice but there are several problems.
First, there is the practical one. How do you actually solve a large system of linear equations on a computer and how do you actually diagonalize a large matrix (that theory tells you is diagonalizable) on a computer. These are very difficult problems since there is a huge gap between the theoretical results and actual computations. That gap is caused of course by rounding errors on a computer. Loads of books are written on the subject as, needless to say, it's of immense importance. Many factorizations of matrices (e.g., LU and QR) are meant to address such issues. To make computations more robust and more efficient.
As for motivating SVD, nothing can be easier. Consider the following questions regarding a linear transformation $T:V\to W$ where $V,W$ are inner product spaces (say over $\mathbb R$) not necessarily of the same dimension (so there is no point of speaking of eigenvalues or Jordan form or any of that):
1) What is the shape of the image under $T$ of the unit sphere in $V$.
2) If $T$ is not invertible, how do you invert it in the best way you can.
3) How do you solve $Tx=b$ when no solution exists.
4) How do you replace $T$ by another transformation of smaller rank (a useful question for data analysis and compression).
the list goes on, but the answer to all of these questions is:SVD.