I have the following group presentation:
$$G=\langle a,b:a^5=b^3=(ab)^2=1\rangle.$$
I've used MAPLE to determine that $|G|=60$. I'm now attempting to do this using the Coxeter-Todd algorithm by hand. Using the algorithm, I've found $12$ cosets and that $\langle x,y\rangle$ with
$$x=(2,3,4,5,6)(7,10,12,13,8)~~\text{and}~~y=(1,2,3)(4,6,7)(5,8,10)(12,13,14)$$ generates a group of order $60$ satisfying the relations given in the presentation (also checked with MAPLE). Note that I didn't simplify after finding that $7=11$ and $8=9$. I.e. this could be relabeled with $10=9,12=10,13=11,14=12$.
I'm stuck in justifying that my result does in fact give $|G|$. My thoughts are to use Lagrange's theorem to put an upper bound on $|G|$ and disjoint cycle enumeration to put a lower bound on $|G|$. My issue is that I don't actually know the orders of elements in $G$. If I were sure that the order of $a$ is $5$, then I could let $H=\langle a\rangle$ so that $|H|=5$ and then $|G|=[G:H]|H|\leq12\cdot|H|=60$. Similarly, naively assuming the orders of $a$, $b$, and $ab$ to be $5$, $3$, and $2$ respectively would suggest that a lower bound on $|G|$ is $2\cdot3\cdot5=30<60$, so I'm stuck here as well.
Best Answer
Let $H$ be the subgroup of $G$ generated by $a$. Then either $|H|=1$ or $|H|=5$. The Coxeter-Todd algorithm gives $12$ (right) cosets, so by application of Lagrange's theorem, $|G|=[G:H]|H|\leq12\cdot5=60$ is an upper bound for $|G|$. Similarly, using the permutations listed above, since $|\langle x\rangle|=5$, $G$ has an image of order $5\cdot12=60$, so $60$ is a lower bound for $|G|$. The desired result follows.