I am working to implement a Least Squares Estimate using Matrices. However, I seem to produce a Singular Matrix which means I cannot solve the equation. I'm struggling to understand why the matrix is singular (I know about determinants, etc) – is there something obvious that I am missing?
The system I am working with is as follows:
$$ X = \begin{pmatrix}\frac{x_1^2}{y_1^2} & \frac{x_1y_1}{y_1^2} & \frac{x_1}{y_1^2} & \frac{y_1}{y_1^2} & \frac{1}{y_1^2} \\
\frac{x_2^2}{y_2^2} & \frac{x_2y_2}{y_2^2} & \frac{x_2}{y_2^2} & \frac{y_2}{y_2^2} & \frac{1}{y_2^2} \\
\vdots \\
\frac{x_k^2}{y_k^2} & \frac{x_ky_k}{y_k^2} & \frac{x_k}{y_k^2} & \frac{y_k}{y_k^2} & \frac{1}{y_k^2}
\end{pmatrix} \\
P = \begin{pmatrix}
A \\
B \\
C \\
D \\
E
\end{pmatrix}
$$
$ P $ is a vector of unknowns, I am trying to find estimates for $ P $ with the following equation:
$$ P_{est} = \left(X^TX\right)^{-1}X $$
Now, the Matrix $ \left(X^TX\right)^{-1} $ is always singular for my data, I have even tried putting in random data which also gives a singular Matrix. With this in mind I suspect there is a property of the Matrix definition itself which is causing the issue but I can't put my finger on it.
Do any of you clever people know why this is happening? I can and will provide data once I return to work.
Thankyou.
Update – I have scaled the data and I still get a Singular Matrix – what is causing this?
Given the following data:
$$ \begin{matrix}
X & Y \\
13.7284650 & 12.2143600 \\
13.7284650 & 12.2143600 \\
12.2198090 & 12.2098100 \\
12.2077710 & 13.7148230 \\
\end{matrix} $$
Will produce the following Matrix:
$$ X = \begin{pmatrix}
1.263 & 1.124 & 0.092 & 0.082 & 6.703E-3 \\
1.002 & 1.001 & 0.082 & 0.082 & 6.708E-3 \\
0.792 & 0.89 & 0.065 & 0.073 & 5.316E-3 \\
0.723 & 0.85 & 0.059 & 0.07 & 4.863E-3 \\
\end{pmatrix} $$
Transposing gives:
$$ X^T = \begin{pmatrix}
1.263 & 1.002 & 0.792 & 0.723 \\
1.124 & 1.001 & 0.89 & 0.85 \\
0.092 & 0.082 & 0.065 & 0.059 \\
0.082 & 0.082 & 0.073 & 0.07 \\
6.703E-3 & 6.708E-3 & 5.316E-3 & 4.863E-3 \\
\end{pmatrix} $$
Therefore $ X^TX $ gives:
$$ X^TX = \begin{pmatrix}
3.7499 & 3.7425 & 0.2927 & 0.2937 & 0.0229 \\
3.7425 & 3.7804 & 0.2937 & 0.2982 & 0.0231 \\
0.2927 & 0.2937 & 0.0229 & 0.0231 & 1.8001E-3 \\
0.2937 & 0.2982 & 0.0231 & 0.0236 & 1.8249E-3 \\
0.0229 & 0.0231 & 1.8001E-3 & 1.8249E-3 & 1.4184E-4 \\
\end{pmatrix} $$
Which has has determinant 0, and therefore cannot be inverted.
Why is this? I suspect it's something to do with diagonality? Is there a way around this to solve the equation?
Any help/advice appreciated, thankyou.
Best Answer
$X$ is 'fat' (that is has more columns than rows). So you don't want to be computing the 'least squares solution' (there are many such solutions, thus the reason that $X^TX$ is singular).
Let me elaborate, consider the problem: Given some $n\times m$ matrix $A$ and vector $y$, find a vector $x$ such that
$$y=Ax.\quad\quad\quad(*)$$
Assume that $A$ is full rank (that is, rank$(A)=min(n,m)$).
If $A$ is square, there is a unique $x$ that satisfies $(*)$ and it is given by
$$x=A^{-1}y.$$
If $A$ is 'skinny' there will most likely be (for all $y$ except those that lie in some subspace) no vector $x$ that exactly satisfies $(*)$. That is why we compute the 'least squares solution' (or 'least square approximate solution') of $(*)$. That is, the vector $x$ that minimizes the square error between $y$ and $Ax$, $||y-Ax||_{2}^2$. It can be shown that the least square solution is given by
$$x=(A^TA)^{-1}A^Ty.$$
If $A$ is 'fat', then for a single vector $y$ there will be many vectors $x$ that satisfy $(*)$. What people often do in this case is pick the 'minimum norm solution'. That is, the vector $x$ with smallest $||x||_2$ that satisfies $(*)$. It turns out that this can be computed as
$$x=A^T(AA^T)^{-1}y.$$
For a much better and thorough explanation of the above (with the derivations of the different solutions) see lectures 6-9 of this.