In the group $PSL(2,\mathbb{Z})$ (which acts on the upper half plane of $\mathbb{C}$), suppose $S$ is the inversion and $T$ is the translation by $1$, i.e.
$$S=
\left( {\begin{array}{cc}
0 & 1 \\
-1 & 0 \\
\end{array} } \right),
T=
\left( {\begin{array}{cc}
1 & 1 \\
0 & 1 \\
\end{array} } \right).
$$
From books on modular forms, the group $PSL(2,\mathbb{Z})$ is generated by $S$ and $T$. Let $G$ by the subgroup generated by $S$ and $T^2$,
$$G=\langle S,T^2\rangle$$
Is $G$ a normal subgroup of $PSL(2,\mathbb{Z})$? Is $[PSL(2,\mathbb{Z}):G]$ finite? If so, are there studies on the modular forms with congruence group $G$?
Similarly one could ask the questions when $G=\langle S,T^k\rangle, k\in \mathbb{Z}+$!
Best Answer
It's of infinite index and generates $\mathrm{PSL}_2(\mathbf{Z})$ as normal subgroup (hence is not normal).
Let me first check that it generates $\mathrm{PSL}_2(\mathbf{Z})$ as normal subgroup. Write $U=TS$. (All equalities are meant in $\mathrm{PSL}_2(\mathbf{Z})$.) So $U^3=1$. Modding out $\langle S,U\rangle$ by the relators $S,(US^{-1})^2$ is equivalent to mod out by $S,U^2$, and hence to mod out by $S,U$ since $U^3=1$. Hence the quotient is trivial, which is precisely the given assertion.
Next, let me check that it has infinite index. I use that $\mathrm{PSL}_2(\mathbf{Z})$ is the free product of $\langle S\rangle$ and $\langle U\rangle$.
Every free product $A\ast B$ can be written as $B^{\ast A}\rtimes A$, where the free factors in the kernel are the $aBa^{-1}$ when $a$ ranges over $A$. Here we thus write $\mathrm{PSL}_2(\mathbf{Z})= (\langle U\rangle\ast \langle SUS^{-1}\rangle)\rtimes \langle S\rangle$. Write $V=SUS^{-1}$: then $T^2=(US^{-1})^2=US^{-1}US=UV$ and $SUS^{-1}=VU$. Thus, in the free subgroup $F$ of index two $\langle U,V\rangle$, the intersection with $G$ is generated by $UV,VU$.
Let us check that it has infinite index. Indeed, inside $F$, modding out by $UV,VU$ yields an infinite cyclic group. So the normal subgroup of $F$ generated by $G\cap F$ has infinite index, and hence $G\cap F$ has infinite index in $F$, which in turn implies that $G$ has infinite index in $\mathrm{PSL}_2(\mathbf{Z})$.
Maybe you can check what this strategy provides when replacing $T^2$ with $T^k$.