Formally, if $F$ is the free group with basis $X = \{r, s, t\}$ and $N$ is the normal subgroup generated by $R = \{r^2 s^{-3}, s^3 t^{-3}, t^{3} (rst)^{-1}\}$, and $G = F/N$, I want to show that the coset of $rst$ has order $2$ in $G$. There are two parts to this: showing $rstrst = 1$ and showing $rst \neq 1$.
There must be some way to play with and combine the relations in just the right way to get $rstrst = 1$, but it seems difficult to get inverses to even appear in the right places and amounts to cancel all the $r, s, t$ of positive exponent. So far I know $r^2 = s^3 = t^3 = rst = str = trs$, $r = st$, $s^2 = tr$, $t^2 = rs$, and so I try at random some manipulations like $rstrst = t^2 s^2 r = (s^{-1}r)^2 (rt^{-1})^2 (t^2 s^{-1}) = \cdots$ or similar, which has just resulted in wandering about aimlessly.
On the other hand, to show $rst \neq 1$, I have previously shown that $G/\langle rst \rangle \cong A_4$, so I could exhibit a property that is different between $G$ and $A_4$ to say that $rst \neq 1$, but such a property might already rely on $rst$ being not $1$ to demonstrate, as the groups share all other defining relations. Proving $rst \neq 1$ directly by showing $rst \notin N$ in $F$ seems way too messy, since the elements of $N$ are words on the conjugates of $R$.
Best Answer
Cool problem, I enjoyed it. I found a completeley elementary proof by hand. Since $r = st$, I will work with the relations $$ s^3 = t^3 = stst. $$ From this, we may deduce that $$ s^2 = tst, \quad \text{or} \quad t^{-1}s^2 = st. \quad (1) $$ Similarly, we have $$ t^2 = sts, \quad \text{or} \quad t^2s^{-1} = st, \quad \text{or} \quad t^2s^{-1}t^{-2} = st^{-1}. \quad (2) $$ By applying $(1)$ three times, we may write that
\begin{align} (stst)^2 &= s^3t^3 \\ &= s^2(st)t^2 \\ &= s^2t^{-1}s(st)t \\ &= s(st^{-1})st^{-1}s(st) \\ &= s\big(st^{-1})^2st^{-1}s^2 \\ &= s\big(st^{-1}\big)^3s^2. \end{align} One may now notice that we are in the position to apply $(2)$, to obtain \begin{align} (stst)^2 &= s\big(t^2s^{-1}t^{-2}\big)^3s^2 \\ &= s\big(t^2s^{-3}t^{-2}\big)s^2 \\ &= st^2t^{-3}t^{-2}s^2 \\ &= st^{-3}s^2 \\ &= ss^{-3}s^2 \\ &= 1, \end{align} proving the first part completely by hand.
Next we show that $stst \neq 1$. We do this by considering the group homomorphism $$ f : G \to \text{SL}_2(\mathbb{F}_3) $$ mapping $$ s \mapsto \begin{pmatrix} -1 & 1 \\ 0 & -1 \end{pmatrix}, \quad \text{and} \quad t \mapsto \begin{pmatrix} -1 & 0 \\ 1 & -1 \end{pmatrix}. $$ To check that this is a group homomorphism, by the universal property of the free group subject to relations, it suffices to verify that $$ f(s)^3 = f(t)^3 = f(s)f(t)f(s)f(t). $$ I will leave it to you to check that all of these expressions evaluate to $-I$, where $I$ denotes the $2 \times 2$ identity matrix over $\mathbb{F}_3$. Thus, we have now found a group homomorphism $f : G \to \text{SL}_2(\mathbb{F}_3)$ under which the element $stst$ has non-trivial image, namely $-I$. Hence the element $stst$ cannot be trivial itself in $G$. Combining the two arguments above, we conclude that $stst$ must be of order precisely $2$.
Finally, I would like to remark that this argument circumvents having to show that $f$ is in fact an isomorphism; I wouldn't immediately see how to do that by hand but fortunately the above suffices for the problem in question.