Problem: Find the order of the group $G = \langle a,b \mid a^5 = b^4 = 1, aba^{-1}b = 1 \rangle.$
I have no idea how you're supposed to approach this. The only thing I can think of it to just try a bunch of combinations and see where that goes, but there must be some type of trick here. I can't think of a familiar group that this is a presentation of, this doesn't degenerate to the trivial group because $a^5 = 1$ and $b^4 = 1$ so there are at least $8$ elements in the group from considering the cyclic subgroups generated by $a$ and $b$ respectively. What does the commutation relation $aba^{-1}b$ get us, $a = b^{-1}ab^{-1}$? Am I supposed to just play around until I find something useful?
What am I missing? Any hint and guidance is greatly appreciated.
Best Answer
Your group is cyclic of order $10$.
As freakish correctly notes, from $a^5=1$ you can conclude that either $a$ has order $5$, or else is trivial (the order divides $5$). Likewise, from $b^4=1$ you can conclude that $b$ has order $4$, or $2$, or $1$. But you cannot conclude that the orders are exactly $4$ and $5$.
General remarks:
How can one approach these problems? Unfortunately, we know there is no systematic way to approach them that is guaranteed to work: this is a consequence of the negative solution to the word problem. (See a discussion related to that here).
Two main tools to try to figure out the size of such a group are Von Dyck's Theorem, which can be used to give lower bounds to the order; and finding a normal form, which can be used to give upper bounds to the order.
Von Dyck's Theorem says: if you have a group given by a presentation $G=\langle X\mid R\rangle$, where $X$ is your set of generators and $R$ is your set of relations among the elements of $X$, and you can find a group $H$ and a set map $f\colon X\to H$ (think of identifying certain elements of $H$ that will play the role of the elements of $X$), if the images $f(x)$ of $x\in X$ satisfy the relations $R$ in $H$, then there exists a group (unique) homomorphism $\mathcal{F}\colon G\to H$ such that $\mathcal{F}(x)=f(x)$ for each $x\in X$.
In the case of your example: we can tell from the relation $a^5=1$ that $a$ has order dividing $5$. We can verify that it has order at least $5$ by finding a homomorphism $G\to H$ to a group where the image of $a$ has order $5$. If we consider the cyclic group of order $5$, $C_5=\langle x\mid x^5=1\rangle$, note that the elements $x$ and $e$ satisfy the same relations as $a$ and $b$ do in $G$: $x^5=1$, $e^4=1$, and $xex^{-1}e = 1$. That means that there is a homomorphism $\phi\colon G\to C_5$ with $\phi(a)=x$ and $\phi(b)=e$. Since $\phi(a)$ has order $5$, that means that $a$ has order a multiple of $5$. Since we already knew it has order a divisor of $5$, that tells us that $a$ indeed has order $5$.
Also, because $\phi$ is surjective, we know that $G/\ker(\phi)\cong C_5$. That means that $|G|=5|\ker\phi|$, so $G$ is either infinite or has order a multiple of $5$.
(We would not be able to play the same trick to determine that the order of $b$ is $4$, though: if we tried to look at $C_4$ with generator $x$, and said "map $a$ to $e$ and $b$ to $x$", these elements do not satisfy the same relations $a$ and $b$, since $exe^{-1}x$ does not equal $1$.)
By finding appropriate images of $G$ through Von Dyck's Theorem, we can find lower bounds for the order of $G$.
The other tool is to find a "normal form". This is an expression for elements of $G$ that is restricted in some way, so that every element has an expression of that form, though different expressions may perhaps correspond to the same element. That will provide an upper bound for $|G|$, by counting how many different expressions there are. Think about how linear transformations over $\mathbb{C}$ can be represented by matrices of a particular type (Jordan canonical form), so that we can restrict ourselves to a smaller collection when thinking about them.
In this case, relations of the form $aba^{-1}b^k=1$ are very useful: we can rewrite them as saying $ab=b^{-k}a$. They give us a way of "rewriting" any expression involving $a$s and $b$s. Since every element of $G$ is a product of powers of $a$ and $b$, imagine that you have some long complicated expression alternating $a$s and $b$s... the fact that $ab=b^{-k}a$ means that whenever you encounter a $b$ following an $a$, you can "shuffle it to the left". So if you have an expression of the form $w_1abw_2$, where $w_1$ and $w_2$ are some expressions involving $a$s, $b$s, and their inverses, then you know that you can rewrite it as $w_1b^{-k}aw_2$.
Now imagine you have your big long word, and you start moving every instance of $b$ that appears to the right of an $a$ to the left of the $a$ using this relation. By doing this repeatedly, you will eventually end up with an expression that has a bunch of $b$s followed by a bunch of $a$s, and that's it. That means that every element of $G$ can be written in the form $b^ra^s$; then using the fact that $b^4=1$ we can reduce $r$ modulo $4$, and using the fact that $a^5=1$ we can reduce $s$ modulo $5$. So we end up with the fact that every element can be written in the form $b^ra^s$ with $r=0$, $1$, $2$, or $3$; and $s=0$, $1$, $2$, $3$, or $4$. That gives a maximum of $20$ elements, since perhaps two different expressions may correspond to the same element. Sometimes you do it the other way, shuffling $b$s that appear left of $a$ to appear on the right. Whatever form you pick, you seek one that lets you restrict the number of ways that an element may be expressed in terms of the generators.
Ideally, a combination of the two methods will give you an upper bound for $|G|$ using normal forms, and a lower bound using Von Dyck's Theorem, and if you can manage to make the two agree... then you know the order of $G$. This is what happens here.
The relation $aba^{-1}b=1$, tells you that $aba^{-1}=b^{-1}$, and therefore that $ab^{-1}a^{-1} = b$. This tells you that you have $ab^{-1}=ba$, so you can take any expression involving $a$ and $b$, and "shuffle" any $b$ that appears to the left of an $a$ by replacing the $ba$ product with $ab^{-1}$. Repeating this process until all your $b$s appear to the right of any $a$ (and using $a^{-1}=a^4$ and $b^{-1}=b^3$ to replace any negative exponents) tells you that you can write any element of this group in the form $a^ib^j$ with $0\leq i\leq 4$ and $0\leq j\leq 3$. But in fact, we can say even more.
Since $aba^{-1}=b^{-1}$ and $ab^{-1}a^{-1} = b$, we have: $$\begin{align*} aba^{-1} &= b^{-1}\\ a^2ba^{-2} &= ab^{-1}a^{-1} = b\\ a^3ba^{-3} &= aba^{-1} = b^{-1}\\ a^4ba^{-4} &= ab^{-1}a^{-1} = b\\ a^5ba^{-5} &= aba^{-1}=b^{-1}. \end{align*}$$
But $a^5=1$, so $a^5ba^{-5} = b$. Thus, we conclude that $b=b^{-1}$, and in fact $b^2=1$ and the third relation reduces to $aba^{-1}=b$, or $ab=ba$. Thus, $a$ and $b$ commute, with $a$ of order dividing $5$, and $b$ of order dividing $2$
That means that the group actually has order at most $10$, as every element can be written as $a^ib^j$ with $0\leq i\leq 4$, $0\leq j\leq 1$ (instead of $0\leq j\leq 3$).
To prove that it has order exactly $10$, note that the cyclic group of order $10$, $C_{10} = \langle x\mid x^{10}=1\rangle$ has elements that satisfy the relations, namely $x^2$ and $x^5$. The map $a\mapsto x^2$ and $b\mapsto x^5$ induces a morphism $G\to C_{10}$, and since $C_{10}=\langle x^2,x^5\rangle$, the map is surjective. Thus, $G$ has $C_{10}$ as a quotient, so $|G|$ is at least $10$.
Since we already knew that $|G|$ as at most $10$, we conclude that $|G|=10$, and therefore the map we just described must be an isomorphism. Thus, your group $G$ is cyclic of order $10$.