$\text{SL}_2(\mathbb Z)$ acts on upper plane $\mathbb H= \{z \in \mathbb{C} | \Im(z) > 0 \}$ via Mobius transformation. $$
\text{ For } \gamma =\begin{bmatrix}
a &b \\c&d \end{bmatrix} \in\text{SL}_2(\mathbb Z), \ \gamma z =\begin{bmatrix}
a &b \\c&d \end{bmatrix}\cdot z = \frac{az +b}{cz+d} $$
Stabilizer of $z$ means set $\{\gamma \in \text{SL}_2(\Bbb Z), \gamma z=z\}$.
I want to know what kind of points have non-trivial stabilizer and the number of orbits.
My effort:
For $z \in \Bbb H$, suppose $z = x + i y$.
$\text{For } \gamma =\begin{bmatrix} a &b \\c&d \end{bmatrix}, \gamma z =\frac{az+b}{cz+d}=z\iff az+b=cz^2+dz$
$$\iff ax+ayi+b = cx^2-cy^2+2cxyi + dx +dyi$$
$$\iff ay=2cxy+dy\ \&\ ax+b=cx^2-cy^2+dx$$
$$\iff a=2cx+d \ \&\ ax+b=cx^2-cy^2+dx$$
$$\implies b=-c(x^2+y^2),\ \gamma =\begin{bmatrix} 2cx+d &-c(x^2+y^2) \\c&d \end{bmatrix}.$$
$$\gamma \in \text{SL}_2(\Bbb Z),\ (2cx+d)\times d-(-c(x^2+y^2))\times c=1 $$
$$\implies (cx+d)^2+(cy)^2=1.$$
Then how to proceed? Thanks in advance.
I haven't learnt modular form yet, and I don't know if these help:
Good description of orbits of upper half plane under $SL_2 (Z)$
Orbit of complex unit $i$ under moebius tranformation in $SL_2(\mathbb{Z})$
Edit:
GTM$105$, Serge Lang, SL$_2(\mathbb R)$ might help.
Comment:
It's an exercise of section about group action on set, and before this section the book just introduces definition and basic property of group, so this problem is a bit more difficult than I thought.
Best Answer
The group $\mathrm{PSL}_2(\mathbf{Z})$ is a free product $C_2\ast C_3$, and hence its nontrivial torsion elements have order 2 or 3, and has 1 conjugacy class of cyclic subgroup of order 2 (a representative is $z\mapsto -1/z$ with fixed point $i$), 1 conjugacy class of cyclic subgroup of order 3 (a representative being generated by $z\mapsto -1/(1+z)$, fixing $j$, the unique root of $1+z+z^2$ in the upper half-plane). The elements of infinite order have no fixed point on $\mathbf{H}$.
So the points with nontrivial stabilizer are those in the orbit of $i$ and the orbit of $j$. They are not in the same orbit since otherwise the stabilizer should contain an element of order 6.