I would like to know the subgroup generated by the following type of matrices.
$$A=\left(\begin{matrix}1&a\\ 0&1\end{matrix}\right)$$
and
$$B=\left(\begin{matrix}1&0\\ b&1\end{matrix}\right)$$
for integer $a,b\in\mathbb Z$.
Is there a way we can parameterize the full set of matrices $G\in\mathcal G=\langle A,B\rangle$?
The group operation is the matrix product
Best Answer
$\newcommand{\Z}{\mathbf{Z}}$Here's a full proof that the subgroup $\Gamma$ generated by the matrix $u=\begin{pmatrix}1 & 1\\0 & 1\end{pmatrix}$ and its transpose $v$ is all of $\operatorname{SL}_2(\Z)$. Given a matrix $m=\begin{pmatrix}a&b\\c&d\end{pmatrix}$, we have $$mu=\begin{pmatrix}a&b+a\\c&d+c\end{pmatrix},mu^{-1}=\begin{pmatrix}a&b-a\\c&d-c\end{pmatrix},mv=\begin{pmatrix}a+b&b\\c+d&d\end{pmatrix},mv^{-1}=\begin{pmatrix}a-b&b\\c-d&d\end{pmatrix}.$$ Write $N(m)=|a|+|b|$. We see that as soon as $|ab|>0$, one of these four matrices $m'$ satisfies $N(m')<N(m)$.
Hence, choosing $m$ that minimizes $N$ within $m\in m\Gamma$, we see that $N(m)=0$. Since entries in the first row are coprime (because determinant is $\pm 1$), the first row of $m$ is $(\pm 1,0)$ or $(0,\pm 1)$. Using the elementary operations above (which remain in $m\Gamma$), we can change $$(-1,0)\to (-1,1)\to (0,1),\quad (0,1)\to (1,1)\to (1,0),\quad (0,-1)\to (1,-1)\to (1,0).$$ Hence $m\Gamma$ contains a matrix with first row $(1,0)$, so, the determinant is $1$, this matrix has the form $\begin{pmatrix}1&0\\c&1\end{pmatrix}$, namely equals $v^c$. Hence $v^c\in m\Gamma$, and since $v\in\Gamma$ this ensures $m\in\Gamma$.