I thought it would be useful to work through the Reidemeister-Schreier rewriting process, as in my comment to the question. So here it is!
So, we know that your subgroup has index $4$, because of what Derek Holt has said. However, this is not too difficult to do by hand - you just end up with your (right) cosets being,
$H$, $Hx$, $Hxt$ and $Hxtx$.
(I've un-capitalised the matrices and have called the subgroup we want to find the presentation for $H$). Because your group has finite index it will have a finite presentation.
Note: finding the cosets is the annoying bit. If your subgroup is normal, however, then it is dead easy!
So basically you now want to do what Ryan Budney said. That is, draw your four cosets in a square ($0$-cells) and draw the lines where each generator sends each coset ($1$-cells). Index the generators by the cosets they leave from (so, for example, $x_{Hxtx}$ takes $Hxtx$ to $Hxt$). You should have $4$ lines labelled by an $x_i$, $4$ by a $t_i$ and $4$ by an $r_i$. (I apologise for not drawing a picture - I have no idea how to on here!)
Then add in your $2$-cells. Basically, start at each of the $4$ cosets and trace out where your original defining relations take each coset. Write down these paths (they should all be loops).
Your new group is the fundamental group of this $2$-complex. So, your generators are the $0$-cells, relations the $2$-cells, and then take a collapsing tree.
Writing $x_0$ for $x_{H}$, $x_1$ for $x_{Hx}$, $x_2$ for $x_{Hxt}$ and $x_3$ for $x_{Hxtx}$ I got the following $2$-cells,
$x_0x_1$, $x_2x_3$, $t_0^2$, $t_1t_2$, $t_3^2$, $x_0t_1x_2t_3x_3t_2x_1t_0$, $x_1t_0x_0t_1x_2t_3x_3t_2$, $x_2t_3x_3t_2x_1t_0x_0t_1$, $x_3t_2x_1t_0x_0t_1x_2t_3$, $(r_0t_0)^3$, $(r_1t_2)^3$, $(r_2t_1)^3$, $(r_3t_3)^3$, $r_0t_0r_1t_2x_1$, $r_2t_1x_2r_3t_3x_3$, $r_0x_0t_1r_2x_1t_0$, $r_1x_2t_3r_3x_3t_2$, $r_2x_1t_0r_0x_0t_1$, $r_3x_3t_2r_1x_2t_3$
and $x_0$, $t_1$, $x_2$ looks like a nice candidate for a collapsing tree.
Thus, we have the presentation,
$\langle x_1, x_3, r_0, r_1, r_2, r_3, t_0, t_2, t_3;$
$x_1, x_3, t_0^2, t_2, t_3^2, t_3x_3t_2x_1t_0, x_1t_0t_3x_3t_2, t_3x_3t_2x_1t_0, x_3t_2x_1t_0t_3, (r_0t_0)^3, (r_1t_2)^3, r_2^3, (r_3t_3)^3,$
$r_0t_0r_1t_2x_1, r_2r_3t_3x_3, r_0r_2x_1t_0, r_1t_3r_3x_3t_2, r_2x_1t_0r_0, r_3x_3t_2r_1t_3\rangle$
Noting that $x_1$, $x_3$ and $t_2$ are all trivial, and eliminating the appropriate elements (which takes quite a few steps) we get
$\langle r_0, r_3; (r_3r_0r_3)^2, (r_0r_3)^3\rangle$
which, applying the appropriate Tietze transformations, becomes
$\langle s, t; t^2, (rt)^3\rangle$
which is (thankfully - you have no idea how long this post took!) the same as Derek Holt's answer.
For the rest of this post, let
$$ A=\begin{pmatrix} 1 & 0\\ 2 & 1\end{pmatrix}, B=\begin{pmatrix} 1 & 2\\ 0 & 1\end{pmatrix}, C=\begin{pmatrix} -1 & 0\\ 0 & -1\end{pmatrix}, D=\begin{pmatrix} -1 & 0\\ 0 & 1\end{pmatrix}.$$
Note that all of $A$, $B$, $C$ and $D$ live in $\Gamma(2)$.
First, let's consider the case of $G=SL(2,\mathbb{Z}$), which is the case I think you wanted (so in your notation, we have the requirement that $ad-bc=1$). Note that in this case, $D\notin G$.
Proposition: $A$, $B$, and $C$ generate $\Gamma(2)$.
Proof: Define a mapping from $f:\ \Gamma(2)\rightarrow \mathbb{Z}^+$ by the formula
$$ f:\ \begin{pmatrix} a & b\\ c & d\end{pmatrix}\mapsto |a|+|c|.$$
Let $\mathfrak{H}$ be the subgroup of $\Gamma(2)$ generated by $A$, $B$, and $C$, and let $X$ be an arbitrary element of $\Gamma(2)$. We will be done if we can show $X\in \mathfrak{H}$.
To this end, pick an element $Y\in \mathfrak{H}X$ [the right coset of $\mathfrak{H}$ containing $X$] for which $f(Y)$ is minimal.
Now letting $Y=\begin{pmatrix} a & b\\ c & d\end{pmatrix}$, consider the following cases:
- $c=0$.
We know $ad-bc=1$, and so in this case $a=d=\pm 1$. But then $Y$ (or $YC$) must be a power of $B$, since
$$ B^n=\begin{pmatrix} 1 & 2n\\ 0 & 1\end{pmatrix}.$$
This means $Y\in\mathfrak{H}\cap\mathfrak{H}X$, so that $\mathfrak{H}=\mathfrak{H}X$, or $X\in \mathfrak{H}$.
- $c\neq0$, and $|a| > |c|$.
Then there exists an $n\in\mathbb{Z}$ such that $-|c| < a+2nc < |c|$ [strict inequality because $a$ is odd and $c$ is even], and then
$$ B^nY=\begin{pmatrix} 1 & 2n\\ 0 & 1\end{pmatrix}\begin{pmatrix} a & b\\ c & d\end{pmatrix}=\begin{pmatrix} a+2nc & b+2nd\\ c & d\end{pmatrix},$$
so that $f(B^nY)=|a+2nc| + |c| < |c| + |c| < |a| + |c|$, contradicting the choice of $Y$. In other words, this case does not happen.
- $c\neq 0$, and $|a| < |c|$. Then a similar argument to the one above, using $A$ instead of $B$, leads also to a contradiction. The proof is now complete.
Thus we see that if $Z=\langle C\rangle$, then $\Gamma(2)/Z$ is generated by $A$ and $B$. But the group generated by $A$ and $B$ is free, by the Ping-Pong Lemma.
I think this is enough for the question, but if you actually meant $G=GL(2,\mathbb{Z})$, one can still say a lot. In this case, $\Gamma(2)/Z$ is not free, but has $F_2$ as a subgroup of index 2. There are only a handful of groups which possess $F_2$ as a subgroup of index 2, and in this case one gets the isomorphism $\Gamma(2)/Z\cong F_2\rtimes C_2$, where $F_2=\langle A, B\rangle$ and $C_2=\langle D\rangle$, with $D$ acting via $A^D=A^{-1}, B^D=B^{-1}$.
Best Answer
The name of the group is $\text{GL}_2(\mathbf{Z})$.
Indeed, note that $$ \left( \begin{smallmatrix} 1&0\\ 1&1 \end{smallmatrix} \right)= \left( \begin{smallmatrix} 0&1\\ 1&0 \end{smallmatrix} \right) \left( \begin{smallmatrix} 1&1\\ 0&1 \end{smallmatrix} \right) \left( \begin{smallmatrix} 0&1\\ 1&0 \end{smallmatrix} \right),$$ and thus your group contains $\text{SL}_2(\mathbf{Z})$ (see, e.g., Corollary 2.6 in http://www.math.uconn.edu/~kconrad/blurbs/grouptheory/SL%282,Z%29.pdf). On the other hand, your group is not contained in $\text{SL}_2(\mathbf{Z})$, and since $[\text{GL}_2(\mathbf{Z}):\text{SL}_2(\mathbf{Z})]=2$ your group must be $\text{GL}_2(\mathbf{Z})$.