Suppose I have a finitely generated subgroup of $G:={\rm PSL}(2,\mathbb{Z})$ defined by $H=\langle A,B,C\rangle$ where
$$A = \begin{bmatrix}
0 & -1 \\
1 & 0
\end{bmatrix},\quad
B = \begin{bmatrix}
-1 & 0 \\
3 & -1
\end{bmatrix},\quad
C = \begin{bmatrix}
-5 & 4 \\
6 & -5
\end{bmatrix}.$$
I have reason to believe this subgroup has finite index in $G$, and I'm trying to calculate it. Following a suggestion the top answer of this MO post, I first tried calculating the intersection of $H$ with $P\Gamma(2)$, where $P\Gamma(2)$ are those matrices which are congruent to the identity matrix when we reduce elements mod $2$. After some work I was able to show that $A^2=-I, B^2, C, (AB)^3, (BA)^3\in H\cap P\Gamma(2)$. I think(?) these and their inverses are the smallest length words with this property. Does that mean $H\cap P\Gamma(2) = \langle B^2, C, (AB)^3, (BA)^3\rangle$? I'm not sure what else to do to find the intersection.
After that, the answer seems to suggest rewriting the generators of the intersection in terms of the standard generators of $P\Gamma(2)$, say
$$x = \begin{bmatrix}
1 & 2 \\
0 & 1
\end{bmatrix}, \quad
y = \begin{bmatrix}
1 & 0\\
-2 & 1
\end{bmatrix}.$$
It was indicated in the MO post that GAP could somehow rewrite the intersection in terms of these generators, and this would somehow be useful information. There was also a comment suggesting the Nielson-Schreier Index Formula could be used. Can someone spell out this argument in greater detail for me? Won't we just fine that the intersection is rank 4 and has index 3 in $P\Gamma(2)$? How does one find the index of $H$ in the entirety of ${\rm PSL}(2,\mathbb{Z})$ from there? I feel like I am missing something.
One last note: I am aware that one can use Fuchsian group methods to calculate the index as well, but I am more interested in understanding the method proposed in the MO thread (or some other algebraic method, if anyone has something better).
Best Answer
Here is a similar spproach, which I think is less complicated. There are various known presentations of ${\rm GL}(2,{\mathbb Z})$ and ${\rm SL}(2,{\mathbb Z})$. I tried using
$${\rm GL}(2,{\mathbb Z}) = \langle p,s,u \mid p^2,s^2,(sp)^4,(upsp)^2,(ups)^3, (us)^2 \rangle,$$
where $$p = \left( \begin{array}{cc}0&1\\1&0\end{array}\right),\ s = \left( \begin{array}{cc}-1&0\\0&1\end{array}\right),\ s = \left( \begin{array}{cc}1&1\\0&1\end{array}\right).$$
Now we can try and write your matrices $A,B,C$ as words in the generators $p,s,u$. I did this in Magma, but you could just as easily do in GAP. I just did it by trial and error rather than using any rewriting functions - it was not too hard. It's basically just using $u$ and $u^{sp}$ to perform row reduction.
I got $$A =sp,\ B = (u^{sp})^3(sp)^2,\ C =u^{-1} (u^{sp})^{-6} u^{-1}.$$
Now we can just use standard coset enumeration to find the index of the subgroup $\langle A,B,C \rangle$ in $G$. The index is $18$, so the index in ${\rm SL}(2,{\mathbb Z})$ is 9.