While I was reading a statistics paper, I came across one statement that I don't understand (I just have basic linear algebra knowledge).
Assume (in the context of regressions), we have a $n\times p$ data matrix $X$, assuming that $X$ is invertible and $n>p$. The paper states
"$U \in \mathbb{R}^{n \times p}$ is an orthonormal matrix whose column space is orthogonal to that of $X$ s.t. $U^TX=0$": such matrix exists if $n\geq 2p$.
I don't understand where the last statement comes from.
I know that the nullspace of $X$ has dimension $n-rank(X)=n-p$ in full rank case and $U$ is the orthonormal basis of the null space of $X$. But I don't get the link why $U$ only exists, if $n\geq p +rank(X)$, i.e. $n\geq 2p$.
Best Answer
Note: it is not standard to refer to the matrix $X$ as "invertible" unless $X$ is a square matrix. Presumably, "$X$ is invertible" refers in this case to the fact that $X$ has full rank. In this case, because $X$ is $n \times p$ with $n > p$, this means that $X$ has rank $p$ (i.e. has "full column rank").
As you say, "$U \in \mathbb{R}^{n \times p}$ is an orthonormal matrix whose column space is orthogonal to that of $X$". The fact that the column space is orthogonal to that of $X$ is equivalent to the statement that $U^TX = 0$. Because $U$ is $n \times p$ with orthonormal columns, the dimension of its column space is $p$. Because the column space of $U$ is orthogonal to that of $X$, the column space of $U$ must be a subspace of the orthogonal complement to the column space of $X$. The column space of $X$ is a $p$-dimensional subspace of $\Bbb R^n$, which means that its orthogonal complement has dimension $n-p$.
Putting all this together leads us to the following conclusion: $$ \operatorname{col}(U) \subseteq \operatorname{col}(X)^\perp \implies\\ \dim(\operatorname{col}(U)) \leq \dim (\operatorname{col}(X)^\perp) \implies\\ p \leq n-p \implies\\ 2p \leq n. $$