Recall:
- Let $A$ be an $m \times n$ matrix with real entries. The null space of $A$ is,
$$\operatorname{nullspace}(A) = \{X \in \mathbb{R}^n: AX=0\}\tag{1}$$
and the dimension of the null space is $\operatorname{null}(A)$.
- The rank of $A$ is given by,
$$\dim (\operatorname{rowspace}(A)) = \dim (\operatorname{colspace}(A))= \operatorname{rank}(A)\tag{2}$$
where the row space is the subspace of $\mathbb{R}^n$ spanned by the rows, and the column space is the subspace of $\mathbb{R}^m$ spanned by the columns.
- The Rank-Nullity Theorem states: For any $m \times n$ matrix $A$,
$$\operatorname{rank}(A)+\operatorname{null}(A)=n\tag{3}$$
1) Find the dimension of the homogeneous system of linear equations,
\begin{align*}
2x+2y+z&=0\\
3x+3y-2z&=0\tag{4}\\
x+y-3z&=0
\end{align*}
Let the coefficient matrix be:
$$C=\left[\begin{array}{ccc}
2 & 2 & 1 \\
3 & 3 & -2\\
1 & 1 & -3 \\\end{array}\right]$$
Now form the Augmented Matrix:
$$C'=\left[\begin{array}{ccc|c}
2 & 2 & 1 & 0\\
3 & 3 & -2 & 0\\
1 & 1 & -3 & 0\\\end{array}\right]$$
This reduces to row-echelon form:
$$C'_r=\left[\begin{array}{ccc|c}
1 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\\end{array}\right]$$
So the system of equations (4) reduces to $x+y=0$, and $z=0$, allowing us to parametrise the solution set as:
$$W=\{(-x,x,0):x\in\mathbb{R}\}\tag{5}$$
Thus the dimension of the solution set is $1$, hence $\dim(W)=1$, since its solution set forms a line through the origin in $\mathbb{R}^3$. Here the nullity of $C$ is the dimension of the null space of $C$, which is the same as the dimension of the solution space of $CX = 0$. That is $\operatorname{nullspace}(C)=W$.
Note the rank of $C$ is given by
$$\dim (\operatorname{rowspace}(C)) = \dim (\operatorname{colspace}(C))=\operatorname{rank}(C)=2$$
and so by (3) we can find,
$$\operatorname{null}(C)=3-\operatorname{rank}(C)=3-2=1$$
and this is what we want, the dimension of the null space, as defined in (1), as $W$ is exactly the null space of $C$, and so $\dim(W)=1$.
- To answer the question in terms of the kernel of a linear transformation, $\theta$:
Let $\theta\colon \mathbb{R}^3\rightarrow\mathbb{R}^3$, be defined
by:
$$\theta\colon
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
2x+2y+z \\
3x+3y-2z \\
x+y-3z \\
\end{bmatrix}
$$
Note that:
$$\dim(\mathbb{R}^3)=\dim(\ker(\theta))+\dim(\operatorname{im}(\theta))$$
and that:
$$\operatorname{null}(C)=\dim(\ker(\theta))\qquad
\operatorname{rank}(C)=\dim(\operatorname{im}(\theta))$$
where $C$ is the $3\times3$ matrix of the linear transformation $\theta$. The kernel of $\theta$, is given by:
\begin{align*}
\ker(\theta)
&=
\left\{
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
: 2x+2y+z =3x+3y-2z =x+y-3z =0
\right\}\\
&=
\left\{
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
:
\begin{bmatrix}
2 & 2 & 1 \\
3 & 3 & -2 \\
1 & 1 & -3 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
0 \\
0 \\
0 \\
\end{bmatrix}
\right\}\\
&=
\left\{
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
:
\begin{bmatrix}
1 & 1 & 0\\
0 & 0 & 1 \\
0 & 0 & 0 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
0 \\
0 \\
0 \\
\end{bmatrix}
\right\}\\
&=
\left\{
\begin{bmatrix}
-x \\
x \\
0 \\
\end{bmatrix}:x\in\mathbb{R}
\right\}
\end{align*}
The dimension of the whole space is $\dim(\mathbb{R}^3)=3$, so by Rank-Nullity:
$$\dim(\mathbb{R}^3)=\operatorname{rank}(C)+\operatorname{null}(C)=2+1=3$$
Since the kernel of a matrix transformation is simply the null space of the matrix, and $W=\ker(\theta)$, then $\dim(W)=\dim(\ker(\theta))=1$.
Note that $W$ is your solution set, it is the thing you have to work out to give (5), it is not a linear transformation itself that you can use The Rank-Nullity Theorem on to find its dimension; rather it is the kernel of some linear transformation whose dimension is seen by the size of the subspace of $\mathbb{R}^3$ it spans, which in this case is a line through the origin isomorphic to $\mathbb{R}^1$.
2) Define $V = \{X \in \mathbb{R}^3 \mid AX=0 \}$. Hence $\dim V$ is the nullity of $A$, or the kernel of the linear transformation $\phi\colon \mathbb{R}^3\rightarrow\mathbb{R}^3$ defined by,
$$\phi\colon
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
x+y+z \\
3x-y+z \\
x+5y+3z \\
\end{bmatrix}
$$
The kernel of the linear transformation $\phi$, is given by:
\begin{align*}
\ker(\phi)
&=
\left\{
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
:
\begin{bmatrix}
1 & 1 & 1 \\
3 & -1 & 1 \\
1 & 5 & 3 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
0 \\
0 \\
0 \\
\end{bmatrix}
\right\}\\
&=
\left\{
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
:
\begin{bmatrix}
1 & 0 & \tfrac12 \\
0 & 1 & \tfrac12 \\
0 & 0 & 0 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
0 \\
0 \\
0 \\
\end{bmatrix}
\right\}\\
&=
\left\{
\begin{bmatrix}
-x \\
-x \\
2x \\
\end{bmatrix}:x\in\mathbb{R}
\right\}
\end{align*}
Where we have reduced $A$ to row-echelon form in the second step. Hence the solution set is $x+\tfrac12 z=0$, $y+\tfrac12 z=0$, so $x=y$, and $2x=-z$. Here the nullity of $A$ is the dimension of null space of $A$, or $\dim(\ker(\phi))=1$, which is the same as the dimension of the solution space of $AX = 0$, which is $\dim(V)=1$, since $V=\ker(\phi)$.
Rank is the maximum number of linearly independent columns of $A$, and dim($A$) is the number of columns of $A$. Same goes for $A^TA$. Since they both have $n$ columns, the proof you linked is correct.
If you look at $A$ and $A^TA$ as linear transformations, then we have $A: \mathbb R^n \rightarrow \mathbb R^m$, and $A^TA: \mathbb R^n \rightarrow \mathbb R^n$. So the right hand side of the dimension theorem is equal to $n$ for both $A$ and $A^TA$.
Best Answer
Let $Z_k={\rm ker} T^k$, $k\geq 0$ be the sequence of kernel spaces. One has $$Z_0 =\{0\} \subset Z_1 \subset Z_2 ...$$ Let $d_k = {\rm dim\ } Z_k$. Then one has the following inequality for $k\geq 1$: $$ d_{k+1}-d_k \leq d_k-d_{k-1}$$ In your case this yields $11-9\leq 9-d_2\;$ or $\;d_2\leq 7$ and $\;9-d_2\leq d_2-4\;$ or $\;d_2\geq 6.5$. The unique integer solution is thus $d_2=7$ and by the kernel-rank theorem we get ${\rm rank}\; T^2=4$ in accordance with the statement of Alex.
I imagine the above inequality is well-known by specialists but I don't have a reference for it. To prove it note that since $T Z_{k+1}\subset Z_k$ and $T Z_k\subset Z_{k-1}$ we have a well-defined map between quotients: $$ \widehat{T} : Z_{k+1}/Z_k \rightarrow Z_k/Z_{k-1} .$$ I claim this map is injective. If $w\in Z_{k+1} \setminus Z_k = Z_{k+1} \setminus T^{-1}(Z_{k-1})$ then the last expression shows indeed that $Tw\in Z_k\setminus Z_{k-1}$. As the map is injective dimensions must increase so $$ d_{k+1}-d_k = {\dim\;} Z_{k+1}/Z_k \leq \dim Z_k/Z_{k-1}=d_k-d_{k-1}.$$ If you don't fancy quotient spaces you may concoct a (slightly longer) proof using complements, i.e. writing $Z_{k+1} = Z_k \oplus W_k$ and look at how $T$ acts upon $W_k$.
About generality: The above inequality (showing concavity of $k\mapsto d_k$) holds in a space of any dimension (also infinite, as long as $d_1$ is finite). Thus the conclusion for $d_2$ is independent of the dimension of the ambient space. But the conclusion for the rank of $T^2$ is obviously not.