Let $G = GL_2(\mathbb Z)$ and $X = \mathbb Z^2$ with elements of $X$ written as column vectors.
(a) Show that $\phi: G\to \text{Perm}(X)$ given by $$\phi(g)\left(\begin{matrix} m \\ n\end{matrix}\right) = g\left(\begin{matrix} m \\ n\end{matrix}\right)$$ is a group action.
(b) Show that the orbits of this action are the same as the equivalence classes under the relation $(m,n)\sim (a,b)$ if $\gcd(m,n) = \gcd(a,b)$.
(c) Let $(m,n)\in\mathbb Z^2$. Show that there exists $g\in G$ such that $$g\ \text{stab}_G\left(\begin{matrix} m \\ n\end{matrix}\right)g^{-1} = \left\{\left(\begin{matrix} 1 & a \\ 0 & b\end{matrix}\right) : a\in\mathbb Z, b=\pm 1\right\}$$
Before anything, it's important to note that for all $g\in G$, $\det g = \pm 1$.
I was able to do (a). $\left(\begin{matrix} 1 & 0 \\ 0& 1\end{matrix}\right) \in G$ is the identity element $e$ for this group action, and $g\cdot (h\cdot x) = (gh)\cdot x$ for all $x\in X$ follows from associativity of matrix multiplication.
I tried (b). Firstly, $\sim$ is indeed an equivalence relation – this is easy to check. Now, I considered arbitrary $x\in X$ and looked at $G\cdot x = \{g\cdot x: g\in G\}$, the orbit of $x\in X$. I presume if I start working with arbitrary matrix entries at this point, things will get messy, so there must hopefully be a nicer way. Also, I noted that if $(m,n)\sim (a,b)$ then there exist integers $a_1,b_1,m_1,n_1$ such that $mm_1 + nn_1 = aa_1 + bb_1$. What do I do next? Don't have much clue about (c) either.
I'd appreciate any hints or solutions that would help since I have been working on this for over a week now without much progress.
P.S. $\text{Perm}(X)$ denotes the set (actually a group under composition itself) of all bijections $X\to X$.
Best Answer
Hint for part (b): let $d$ be the greatest common divisor of $m$ and $n$, which is also the greatest common divisor of $a$ and $b$. Try showing there is a $g \in \operatorname{GL}_2(\mathbb Z)$ such that
$$g \begin{pmatrix} m \\ n \end{pmatrix} = \begin{pmatrix} d \\ 0 \end{pmatrix}. \tag{$\ast$}$$
By the same reasoning, there will exist another matrix $h \in \operatorname{GL}_2(\mathbb Z)$ such that
$$h \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} d \\ 0 \end{pmatrix}.$$
It follows that $h^{-1}g$ will send $\begin{pmatrix} m \\ n \end{pmatrix}$ to $\begin{pmatrix} a \\ b \end{pmatrix} $, so they will be in the same orbit.
How do you know that such a matrix $g$ exists? You know that $d$ is a divisor of $m$ and $n$, so $\frac{m}{d}$ and $\frac{n}{d}$ are integers. You also know that there exist integers $x$ and $y$ such that $xm + ny = d$. Now you can take
$$g = \begin{pmatrix} x & y \\ ? & ? \end{pmatrix}$$
so that the top row of $\ast$ comes out correctly. What should go in the blanks '?' so that the determinant of $g$ is $1$ and the bottom row of $\ast$ comes out correctly?
The above reasoning shows that if $(a,b)$ and $(m,n)$ have the same GCD $d$, then they are in the same orbit, since they are both in the orbit of $(d,0)$. You still need to show conversely that if $g \in \operatorname{GL}_2(\mathbb Z)$, and $g.(m,n) = (a,b)$, then $(m,n)$ and $(a,b)$ have the same GCD.
Hint for part (c): in general, if $G$ is a group acting on a set $X$, then for $g \in G$ and $x \in X$, we can consider the stabilizers $H_x$ and $H_{g.x}$ of the elements $x$ and $g.x$ of $X$, respectively. It's quite easy to see that these subgroups of $G$ are related by conjugation: $gH_x g^{-1} = H_{g.x}$.
So the question is, what element of $\mathbb Z^2$ is
$$\{ \begin{pmatrix}1 & a \\ 0 & b \end{pmatrix} a \in \mathbb Z, b = \pm 1 \}$$ the stabilizer of?