In the answer I will use the standard way to number the relators in a presentation. Then, if $G$ is a group of deficiency $\ge 1$ (i.e., admits a presentation with $n$ generators and $k$ relators, where $n>k$) then $G$ is infinite and, moreover, admits an epimorphism to the infinite cyclic group. To prove this, consider the rational vector space $V=Hom(G, {\mathbb Q})$ (where we regard ${\mathbb Q}$ as the additive group of the field ${\mathbb Q}$):
This vector space $V$ is given by imposing $k$ linear equations on
$Hom(F_n, {\mathbb Q})={\mathbb Q}^n$, since every homomorphism to ${\mathbb Q}$ is determined by its values on generators of $G$, while the only restrictions on the images of generators are that each relator maps to zero: Every such condition is one linear equation.
Hence, $dim Hom(G, {\mathbb Q}) \ge n-k\ge 1$. It therefore, follows that there exists a nonzero homomorphism $h: G \to {\mathbb Q}$. The image of this homomorphism is an infinite torsion free finitely generated subgroup (as ${\mathbb Q}$ contains no nonzero finite subgroups), i.e. ${\mathbb Z}^r$ for some $r\ge 1$. Since it is a subgroup of ${\mathbb Q}$, $r=1$. Thus, $G$ admits an epimorphism $h: G\to {\mathbb Z}$, and, hence, is an infinite group. In particular, $G$ contains an element of infinite order (any element $g\in G$ such that $h(g)\ne 0$).
In fact, one can prove more, namely that abelianization of $G$ has rank $\ge n-k$, but we do not need this.
A much more interesting result is due to Baumslag and Pride: They proved that every group of deficiency $\ge 2$ has a finite index subgroup which admits an epimorphism to a nonabelan free group. Such a group is called large. See also http://arxiv.org/pdf/1007.1489.pdf and references there.
Ok. I figured this out myself. Any element generated is of the form $n_1$ $a$'s followed by $n_2$ $b$'s followed by $n_3$ $a$'s followed by $n_4$ $b$'s and so on. Using the relation $a^2=1=b^2$, we can simplify all the elements to a sequence of alternating $a$'s and $b$'s, i.e., the elements are of the form
$$abab\cdots ab \text{ or }baba\cdots ba$$
But we have $(ab)^3 = 1$. Hence, all the elements apart from the identity are
$$\{a,ab,aba,abab,ababa,b,ba,bab,baba,babab,bababa\}$$
But we have
- $bababa = a^2bababa = a(ab)^3a = a^2 = 1$.
- $babab = bababa^2 = (ba)^3a = a$
- $ababa = b(ab)^3 = b$
- $(ab)^2 = b^2(ab)^2a^2 = b(ba)^3a = ba$
- $(ba)^2 = a^2(ba)^2b^2 = a(ab)^3b = ab$
- $aba = bab$
Hence, the only distinct elements are are $1, a,b,ab,ba,aba$. So $6$ elements and we can now do a one to one matching with elements of $S_3$.
As an aside, the users on this website are too rude. I am a new user and here to clarify questions on algebra. Shouting at a new user is is not the right way to welcome her!
Best Answer
Since $G$ is generated by $a,b$, the general element is of the form $$\tag1a^{n_1}b^{m_1}a^{n_2}b^{m_2}\cdots a^{n_k}b^{m_k}$$ with $n_i, m_i\in\mathbb Z$. However, $a^5=1$ implies that we can restrict to $0\le n_i\le 4$ and $b^4=1$ implies that we can restrict to $0\le m_i\le 3$. The last relation (rewritten as $ab=ba^3$ by our wish to have $0\le n_i\le 4$) shows that we can "pull" all $b$s to the left of all $a$s, but also the other way: $$\tag 2a^2b=ba^6=ba$$ allows us to pull $a$s to the left instead.
Now among all representations of an element $g\in G$ we may pick one that minimizes $k$ in $(1)$. Use the relation $(2)$ to arrive at a contradiction against the minimality of $k$ if $k>1$.
So far we have shown that $|G|\le 20$. You could in principle write down a multiplication table for these twenty elements $a^nb^m$. Alternatively, exhibit a well-known group of order $20$ that has generators obeying the given relations. To the latter end, we may want to rewrite $ab=ba^{-1}$ as $b^{-1}ab=a^{-1}$. That is: The cyclic group generated by $b$ (which we expect to be isomorphic to $\mathbb Z/4\mathbb Z$) acts on the cyclic group generated by $a$ (which we expect to be isomorphic to $\mathbb Z/4\mathbb Z$) by conjugation. This makes $\langle a\rangle$ a normal subgroup of $G$. Moreover, the action by the generator $b$ is "take the inverse" (indeed an automorphism of $\mathbb Z/5\mathbb Z$ of order dividing $4$). Thus we identify $G$ as semidirect product $\mathbb Z/5\mathbb Z\rtimes\mathbb Z/4\mathbb Z$.