I know how to begin the procedure but I don't know how to finish it. Let's start with an example (sorry for it being so unwieldy).
Let
$$A =\begin{pmatrix}
177& 548& 271& -548& -356\\ 19& 63& 14& -79& -23\\ 8& 24& 17& -20& -20\\ 42& 132& 55& -141& -76\\ 56& 176& 80& -184& -105\end{pmatrix}$$Find the Jordan canonical form of A and the change of basis matrix.
STEP 1. Find the characteristic polynomial.
You can do this step using the following command
[V, lam] = eig(A)
which produces for the variable lam the eigenvalues with repeats, allowing one to easily deduce the following for the characteristic polynomial.
$\chi_A(\lambda) = (\lambda – 3)^4(\lambda + 1)$.
STEP 2. Find the geometric multiplicity for $\lambda = 3$.
To do this we must find the nullity, which is equal to $5 – \text{rank}(A – 3I)$ by the rank-nullity theorem. Commanding 5 - rank(A - 3 * eye(5)) gives $2$.
STEP 3. Find the geometric multiplicity for $\lambda = -1$.
To do this we must find the nullity, which is equal to $5 – \text{rank}(A + I)$ by the rank-nullity theorem. Commanding 5 - rank(A + eye(5)) gives $1$.
STEP 4. Make a table and try to guess the Jordan form.
$$
\begin{array}{c|c|c}
\lambda & \operatorname{am}_C(\lambda) & \operatorname{gm}_C(\lambda) \\ \hline
3 & 4 & 2 \\
-1 & 1 & 1
\end{array}
$$
However in this case we cannot guess, because are two Jordan blocks and their dimensions sum to 4. There are two ways to split 4 into two integers and we don't know which one is it.
How do I proceed? If you could, I would greatly appreciate it if you use the step structure I had been using throughout this post. Ideally with Octave commands.
Best Answer
If you calculate $$\operatorname{rank}(A-\lambda I)-\operatorname{rank}((A-\lambda I)^2),$$ you'll get the number of blocks corresponding to $\lambda$ which have size at least $2\times2$. (This works in general, $\operatorname{rank}((A-\lambda I)^{k-1})-\operatorname{rank}((A-\lambda I)^2)^k)$ gives you the number of the blocks corresponding to $\lambda$ with the size $\ge k$.)
In this case you get $\operatorname{rank}(A-3I)-\operatorname{rank}((A-3I)^2)=3-1=2$, and you see that there are two blocks for the eigenvalue $3$ that are of the size $2\times2$ or bigger. Now you have only one possibility for the Jordan form (up to a permutation of the diagonal blocks): $$J= \begin{pmatrix} -1 & 0 & 0 & 0 & 0 \\ 0 & 3 & 1 & 0 & 0 \\ 0 & 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 3 & 1 \\ 0 & 0 & 0 & 0 & 3 \\ \end{pmatrix} $$
To add also a brief explanation why - at least for this particular case - notice that for the two alternatives you're considering you get
\begin{array}{cc} J= \begin{pmatrix} -1 & 0 & 0 & 0 & 0 \\ 0 & 3 & 1 & 0 & 0 \\ 0 & 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 3 & 1 \\ 0 & 0 & 0 & 0 & 3 \\ \end{pmatrix} & J= \begin{pmatrix} -1 & 0 & 0 & 0 & 0 \\ 0 & 3 & 1 & 0 & 0 \\ 0 & 0 & 3 & 1 & 0 \\ 0 & 0 & 0 & 3 & 0 \\ 0 & 0 & 0 & 0 & 3 \\ \end{pmatrix} \\ J-3I= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix} & J-3I= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix}\\ (J-3I)^2= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix} & (J-3I)^2= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix} \\ (J-3I)^3= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix} & (J-3I)^3= \begin{pmatrix} -4 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ \end{pmatrix} \end{array}
If matrices $A$ and $J$ are similar, then also matrices $(A-\lambda I)^k$ and $(J-\lambda)^k$ are similar, so they have the same rank.
In these two cases you get \begin{array}{|c|c|c|} \hline \operatorname{rank}(A-3I)=\operatorname{rank}(J-3I) & 3 & 3 \\\hline \operatorname{rank}((A-3I)^2)=\operatorname{rank}((J-3I)^2) & 1 & 2 \\\hline \operatorname{rank}((A-3I)^3)=\operatorname{rank}((J-3I)^3) & 1 & 1 \\\hline \end{array} So the ranks of these powers can help you to distinguish between these two cases.
And you can also notice the general pattern - if you look at a block of size at least $k$ corresponding to $\lambda $, then the power $(J-\lambda I)^k$ has one more zero row in this block, compared to the previous power $(J-\lambda I)^{k-1}$. So the rank decreases by one for every such block.
You can find an explanation of the relationship between the number of blocks of the given size and ranks or these powers also here: Jordan form, number of blocks..