You want to find row/column operation for which your initial matrix looks like this
\begin{equation*} \begin{pmatrix}a_1 & 0 \\0&a_2 \\0&0 \end{pmatrix}\end{equation*}
and $a_1\mid a_2$. According to Wikipedia this numbers are unique up to multiplication by a unit, and they can be found by using the formula $a_i =\frac{d_i(A)}{d_{i-1}(A)}$, where $d_i(A)$ is the greatest common divisor of all $i\times i$ minor of you matrix.
Moreover according to
this thread you are only allowed to
- interchange two rows or two columns,
- multiply a row or column by $\pm1$ (which are the invertible elements in $\mathbf{Z}$),
- add an integer multiple of row to another row (or an integer multiple of a column to another column).
Let us compute $a_1$. As you can see the greatest common divisor of all $1\times 1$ minor of your matrix is $2$, so we will have $a_1 =2$ once the algorithm is over.
To find $a_2$ you need to compute all the $2\times 2$ minor of your matrix, and you should get $d_2(A)= GCD(-24,108,96)=12$, so $a_2 = \frac{12}{2}=6$.
Now we should find operations that respects 1.,2.,3. such that the final matrix is
$$\begin{pmatrix}2&0\\0&6\\0&0\end{pmatrix}$$
This is done in the following way (I can try it by yourself, and you should get the same solution also with other operations, because $a_1,a_2$ are unique up to multiplication by unit):
\begin{align*}
\begin{pmatrix} 6&-6 \\-6&-12\\4&-8\end{pmatrix} &\overset{Ir-IIIr; \, IIr + IIIr}{\leadsto} \begin{pmatrix} 2&2\\-2&-20\\4&-8\end{pmatrix} \\
\overset{IIc-Ic}{\leadsto}\begin{pmatrix}2&0\\-2&-18\\4&-12\end{pmatrix} &\overset{IIr+Ir;\, IIIr-2Ir}{\leadsto} \begin{pmatrix}2&0\\0&-18\\0&-12\end{pmatrix} \\
\overset{IIr-IIIr}{\leadsto}\begin{pmatrix} 2&0\\0&-6\\0&-12\end{pmatrix}&\overset{IIIr-2IIr}{\leadsto}\begin{pmatrix}2&0\\0&-6\\0&0\end{pmatrix}
\end{align*}
Now you can mulitply by $-1$ the second row.
Best Answer
I note that the question refers to "doing the same operations to both columns and rows," so maybe this is what's intended.
Starting with $$\pmatrix{0&1\cr1&0\cr}$$ add the 2nd row to the 1st, then add the 2nd column to the 1st; you get $$\pmatrix{1&1\cr1&0\cr}{\rm\ then\ }\pmatrix{2&1\cr1&0\cr}$$ Now subtract half the 1st row from the 2nd, followed by subtracting half the 1st column from the second; $$\pmatrix{2&1\cr0&-1/2\cr}{\rm\ then\ }\pmatrix{2&0\cr0&-1/2\cr}$$ and there's your diagonal matrix.
Now, adding $a$ times the 2nd row to the 1st is the same as multiplying on the left by $$\pmatrix{1&a\cr0&1\cr}$$ and adding $a$ times the 2nd column to the 1st is the same as multiplying on the right by the transpose, $$\pmatrix{1&0\cr a&1\cr}$$ So we really are getting $A=P^tBP$ with $B$ diagonal, and you can walk through the steps to see what $P$ is.