Question: Suppose that 4 vectors, v1, v2, v3, and v4 are given in $R^{6}$ and we don’t know whether or not these vectors are linearly independent. Explain how you would find the (projection) matrix which projects onto the subspace S = span {v1, v2, v3, v4}.
Is it correct to say:
First find the basis of S to obtain a linearly independent columns of a matrix A that will span S. Then use the basis vectors as the columns of A. To find the projection matrix we use $P=A(A^{T}A)^{−1}A^{T}$
So I had asked this question somewhere and someone suggested in using least squares. I am assuming because A is 4×6. Then the projection matrix would just be P=Ax. But, if I try finding the least square, x, I would need a $b$ for the equation $A^{T}Ax=A^{T}b$.
Best Answer
A sample problem like this is explored in Solve an overdetermined system of linear equations.
Start with the linear system $\mathbf{A}x = b$ where $$ \mathbf{A}\in\mathbb{C}^{m \times n}_{\rho}, \quad x \in\mathbb{C}^{n}, \quad b \in\mathbb{C}^{m}. $$ The existence of the singular value decomposition is guaranteed: $$ \mathbf{A} = \mathbf{U}\, \Sigma \, \mathbf{V}^{*} = \underbrace{\left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}}} & \color{red}{\mathbf{U}_{\mathcal{N}}} \end{array} \right]}_{m\times m} % \underbrace{\left[ \begin{array}{cc} \mathbf{S}_{\rho \times \rho} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right]}_{m\times n} % \underbrace{\left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}} \end{array} \right]^{*}}_{n\times n} . $$ $\color{blue}{Range\ space}$ entities are colored in blue, $\color{red}{nullspace}$ in red. From that we construct the Moore-Penrose pseudoinverse $$ \mathbf{A}^{\dagger} = \mathbf{V}\, \Sigma^{\dagger} \mathbf{U}^{*}. $$ The four fundamental unitary projectors are $$ \begin{align} % \mathbf{P}_\color{blue}{\mathcal{R}\left( \mathbf{A} \right)} &= \mathbf{A}\mathbf{A}^{\dagger} & % \mathbf{P}_\color{red}{\mathcal{N}\left( \mathbf{A}^{*} \right)} &= \mathbf{I}_{m} - \mathbf{A}\mathbf{A}^{\dagger} \\ % \mathbf{P}_\color{blue}{\mathcal{R}\left( \mathbf{A}^{*} \right)} &= \mathbf{A}^{\dagger}\mathbf{A} & % \mathbf{P}_\color{red}{\mathcal{N}\left( \mathbf{A} \right)} &= \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \\ % \end{align} $$ You are asking about $$ \mathbf{P}_\color{blue}{\mathcal{R}\left( \mathbf{A} \right)} = \mathbf{A}\mathbf{A}^{\dagger} = \mathbf{A}\left( \mathbf{A}^{*}\mathbf{A}\right)^{-1}\mathbf{A}^{*}. $$
The connection between the pseudoinverse and the normal equations solutions is shown in SVD and linear least squares problem