Since the matrix is nilpotent, as you note the characteristic polynomial is $x^8$, and the minimal polynomial is $x^i$ for some $i$, $1\leq i \leq 8$. It's not $x$, because $A$ is not zero, and it's not $x^2$, because $A^2$ is not zero either. So $3\leq i\leq 8$.
Let's go form there.
Because $A$ has rank $5$, the Rank-Nullity Theorem tells you the nullity is $3$; that means that the dimension of the eigenspace corresponding to $0$ is $3$; this tells you, by the basic theory of the Jordan Canonical Form, the number of Jordan blocks associated to zero. So the Jordan canonnical form of $A$ has exactly 3 blocks, all corresponding to zero.
The key observation to make is that if $J$ is $k\times k$ Jordan block associated to $0$, then it has rank $k-1$, and $J^s$ has rank $k-s$ ($1\leq s\leq k$); just notice that each time the $1$s "move up" one row until they disappear.
So: the original matrix consists of three blocks; say the sizes are $n_1$, $n_2$, and $n_3$, with $n_1\leq n_2\leq n_3$. We know the largest block has size at least $3$ (the degree of the minimal polynomial is the size of the largest block). And we know $n_1+n_2+n_3=8$.
We cannot have both $n_1$ and $n_2$ equal to $1$; if we did, the matrix would be a $6\times 6$ block and the rest would be zeros; after squaring, the matrix would have rank $4$ (by the observation above), which is not the case. So $n_2\geq 2$. We cannot have $n_1=1$ and $n_2=2$, because then $n_3=5$, and after squaring we would get a matrix of rank $3$, which is not the case.
In fact, if the blocks are of size $1+n_2+n_3$, with $n_3\geq n_2\geq 2$, then the rank of the square would be $n_2+n_3-4 = 3$, which is not the case. So $2\leq n_1$.
That will give the right rank, since then we will have that the rank of $A^2$ is $(n_1-2)+(n_2-2)+(n_3-2) = n_1+n_2+n_3-6 = 8-6 = 2$.
So we must have $2\leq n_1\leq n_2\leq n_3$, $n_1+n_2+n_3=8$; any of these combinations will be a possible Jordan canonical form.
After finding all possibilities, verify that in each case you get a different value for $\mathrm{rank}(A^3)$, so that will show the last clause of the problem is true.
They are correct. What the characteristic polynomial $$x^2(x-1)(x+1)$$tells you is that there are at least $3$ cages, of which the cages for eigenvalues $1$ and $-1$ have size $1$. This only leaves the eigenvalue $0$, for which you have $2$ options:
- There are $2$ linearly independent eigenvectors for the eigenvalue $0$. In that case, your matrix has $4$ eigenvectors and can be diagonalized.
- There is only $1$ linearly independent eigenvector for $0$. This means that the cage for $0$ is $2\times 2$ and the Jordan form is as your second matrix suggests.
Of course, all this is done up to permutation, meaning that you can change the order of the eigenvalues on the diagonal and still get a Jordan form.
Best Answer
As you mentioned correctly- The characteristic polynomial of $N$ is $\chi_N(x)=x^5$, we know that not by computing, but by the nilpotent property and the dimension of $N$. To understand the characteristic polynomial of $N+I$ we need to go back to the definition: To any matrix $M$ it's characteristic polynomial is given by $\det(xI-M)$. Let's examine what happens when we add the identity matrix:
$$\det(xI-(N+I))=\det((x-1)I+N)$$
This is exactly the definition of the characteristic polynomial but instead of being a polynomial over of the variable $x$, it's shifted by $1$. It's a composition of the original characteristic polynomial with $x-1$, so if we had $\chi_N(x)=x^5$ then $\chi_{N+I}(x)=(x-1)^5$, and thus 1 is an eigenvalue of $N+I$.
Knowing all this we get these options of the jordan canonical form (up to rearranging etc.) (also, I'm using lower triangular matrices, I know some people use upper, these are equivalent):
$$ \left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {0} & {1} & {0} & {0} & {0} \\ {0} & {0} & {1} & {0} & {0} \\ {0} & {0} & {0} & {1} & {0} \\ {0} & {0} & {0} & {0} & {1}\end{array}\right],\left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {1} & {1} & {0} & {0} \\ {0} & {0} & {1} & {1} & {0} \\ {0} & {0} & {0} & {0} & {1}\end{array}\right],\left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {1} & {1} & {0} & {0} \\ {0} & {0} & {0} & {1} & {0} \\ {0} & {0} & {0} & {0} & {1}\end{array}\right],\left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {1} & {1} & {0} & {0} \\ {0} & {0} & {0} & {1} & {0} \\ {0} & {0} & {0} & {1} & {1}\end{array}\right] $$ $$ \left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {0} & {1} & {0} & {0} \\ {0} & {0} & {1} & {1} & {0} \\ {0} & {0} & {0} & {0} & {1}\end{array}\right],\left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {0} & {1} & {0} & {0} \\ {0} & {0} & {0} & {1} & {0} \\ {0} & {0} & {0} & {0} & {1}\end{array}\right],\left[\begin{array}{ccccc}{1} & {0} & {0} & {0} & {0} \\ {1} & {1} & {0} & {0} & {0} \\ {0} & {1} & {1} & {0} & {0} \\ {0} & {0} & {1} & {1} & {0} \\ {0} & {0} & {0} & {1} & {1}\end{array}\right] $$