To fix the $2\times 2$ block in the lower right, do
$$
\begin{bmatrix}x & 0\\0 & x^2-5x-7\end{bmatrix}\to
\begin{bmatrix}x & x\\0 & x^2-5x-7\end{bmatrix}\to
\begin{bmatrix}x & x\\-x^2 & -5x-7\end{bmatrix}\to
\begin{bmatrix}x & x\\-x^2+5x & -7\end{bmatrix}\to
\begin{bmatrix}-\frac{1}{7}x^3+\frac{5}{7}x^2+x & 0\\-x^2+5x & -7\end{bmatrix}\to
\begin{bmatrix}-\frac{1}{7}x^3+\frac{5}{7}x^2+x & 0\\0 & -7\end{bmatrix}\to
\begin{bmatrix}x^3-5x^2-7x & 0\\0 & 1\end{bmatrix}\to
\begin{bmatrix}1 & 0\\0 & x^3-5x^2-7x \end{bmatrix}.
$$
The operations are
- add column 1 to column 2
- subtract $x$ times row 1 from row 2
- add $5$ times row 1 to row 2
- add $\frac{1}{7}x$ times row 2 to row 1
- add $-\frac{1}{7}(x^2-5x)$ times column 2 to column 1
- multiply row 1 by $-7$ and row 2 by $-\frac{1}{7}$
- swap rows 1 and 2 and columns 1 and 2.
In Steps 2, 3, and 4 we are using the Euclidean algorithm to place the GCD of the elements in column 2 in the lower right corner.
You want to find row/column operation for which your initial matrix looks like this
\begin{equation*} \begin{pmatrix}a_1 & 0 \\0&a_2 \\0&0 \end{pmatrix}\end{equation*}
and $a_1\mid a_2$. According to Wikipedia this numbers are unique up to multiplication by a unit, and they can be found by using the formula $a_i =\frac{d_i(A)}{d_{i-1}(A)}$, where $d_i(A)$ is the greatest common divisor of all $i\times i$ minor of you matrix.
Moreover according to
this thread you are only allowed to
- interchange two rows or two columns,
- multiply a row or column by $\pm1$ (which are the invertible elements in $\mathbf{Z}$),
- add an integer multiple of row to another row (or an integer multiple of a column to another column).
Let us compute $a_1$. As you can see the greatest common divisor of all $1\times 1$ minor of your matrix is $2$, so we will have $a_1 =2$ once the algorithm is over.
To find $a_2$ you need to compute all the $2\times 2$ minor of your matrix, and you should get $d_2(A)= GCD(-24,108,96)=12$, so $a_2 = \frac{12}{2}=6$.
Now we should find operations that respects 1.,2.,3. such that the final matrix is
$$\begin{pmatrix}2&0\\0&6\\0&0\end{pmatrix}$$
This is done in the following way (I can try it by yourself, and you should get the same solution also with other operations, because $a_1,a_2$ are unique up to multiplication by unit):
\begin{align*}
\begin{pmatrix} 6&-6 \\-6&-12\\4&-8\end{pmatrix} &\overset{Ir-IIIr; \, IIr + IIIr}{\leadsto} \begin{pmatrix} 2&2\\-2&-20\\4&-8\end{pmatrix} \\
\overset{IIc-Ic}{\leadsto}\begin{pmatrix}2&0\\-2&-18\\4&-12\end{pmatrix} &\overset{IIr+Ir;\, IIIr-2Ir}{\leadsto} \begin{pmatrix}2&0\\0&-18\\0&-12\end{pmatrix} \\
\overset{IIr-IIIr}{\leadsto}\begin{pmatrix} 2&0\\0&-6\\0&-12\end{pmatrix}&\overset{IIIr-2IIr}{\leadsto}\begin{pmatrix}2&0\\0&-6\\0&0\end{pmatrix}
\end{align*}
Now you can mulitply by $-1$ the second row.
Best Answer
To expand my comment...Add column 2 to column 1. Subtract row 2 from row 1. Now you have a scalar in the (1,1) position -- rescale to 1.
$$\begin{pmatrix} x-3 & 0 & 0 & 0 \\ 0 & x+1 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim \begin{pmatrix} x-3 & 0 & 0 & 0 \\ x+1 & x+1 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim $$ $$\begin{pmatrix} -4 & -x-1 & 0 & 0 \\ x+1 & x+1 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim \begin{pmatrix} 1 & (1/4)(x+1) & 0 & 0 \\ x+1 & x+1 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim $$
Now add $(-1/4)(x+1)$ times column 1 to column 2 (to clear everything beside 1).
$$\begin{pmatrix} 1 & 0 & 0 & 0 \\ x+1 & x+1-(1/4)(x+1)^2 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim $$
Add $-(x+1)$ times row 1 to row 2 (to clear everything below 1) & simplify the (2,2)-entry. Then rescale row 2 (so the polynomial is monic).
$$\begin{pmatrix} 1 & 0 & 0 & 0 \\ 0 & -(1/4)(x+1)(x-3) & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim \begin{pmatrix} 1 & 0 & 0 & 0 \\ 0 & (x+1)(x-3) & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & x+1\end{pmatrix} \sim $$
Finally swap columns 2 and 4 and then rows 2 and 4 to switch the positions of $(x+1)(x-3)$ and $x+1$. We are left with the Smith normal form.
$$\begin{pmatrix} 1 & 0 & 0 & 0 \\ 0 & x+1 & 0 & 0 \\ 0 & 0 & x+1 & 0 \\ 0 & 0 & 0 & (x+1)(x-3)\end{pmatrix} $$