Suppose $G$ is a non-cyclic group generated by two elements, both of order $2.$ Prove that $G$ has a cyclic subgroup of index $[G:C]=2.$ If such a group is of finite order $2k,$ prove it is isomorphic to a dihedral group $D_k.$
My attempt:
Let $G=\langle x,y\rangle$ with $x^2=y^2=1.$ I first tried to write down $G$ in the most convenient way. I looked at all possible strings of $x's$ and $y's$ that appear in the group: $x,y,(xy)^n,(xy)^nx,(yx)^n,(yx)^ny,$ for $n\in\Bbb N.$ Now, $x^2=y^2=1$ gives us $(xy)^{-1}=yx$ so we may as well write $(yx)^n=(xy)^{-n}$ and $(yx)^ny=(xy)^{-n}yxx=(xy)^{-n-1}x$ and instead write the elements as $e_G,(xy)^n,(xy)^nx,n\in\Bbb Z.$
Now, let $C:=\langle xy\rangle.$ That is a cyclic group and, if $g\in G,$ and $c\in C,$ $gc$ will either be in the form of $(xy)^n$ or $(xy)^nx$ for some $n\in\Bbb Z$ as $$\begin{aligned}(xy)^n(xy)^m&=(xy)^{n+m},n,m\in\Bbb Z\\(xy)^nx(xy)^m&=(xy)^n(xy)y(xy)^mxx\\&=(xy)^{n+1}(yx)^{m+1}x\\&=(xy)^{n-m}x.\end{aligned}$$
So, $[G:C]=2.$
For the second part, I intended to use the following result proven in my script:
(1) For $n\ge 3,$ dihedral group $D_n$ is a non – abelian group of order $2n$ generated by elements (rotation around the center) $a$ and (reflection wrt the line passing through the center of the polygon $P_n$) $b.$ Those elements satisfy the following conditions: $$\begin{aligned}a^n&=1=b^2,\\a^k&\ne 1,\forall k\in\{1,\ldots,n-1\}\\a^{-1}b&=ba.\end{aligned}$$
(2) Every group $\mathcal D$ generated by some elements $a$ and $b$ that satisfy the same conditions as do $a$ and $b$ above is isomorphic to the $n$ – th dihedral group, that is $\mathcal D\cong D_n.$
Since $C:=\langle xy\rangle$ and $|G|=2k<\infty,$ by Lagrange, $$|xy|=|\langle xy\rangle|=|C|=\frac{|G|}{[G:C]}=\frac{2k}2=k.(*)$$ I tried the approach from this answer: since $x^2=1,$ $y$ can be written as $y=x(xy)$ and therefore $\langle x,y\rangle\le\langle x,xy\rangle,$ hence $\langle x,y\rangle =\langle x,xy\rangle.$ Let $a:=xy$ and $b:=x.$ Then $$a^{-1}b=(xy)^{-1}x=yx^2=y=x^2y=x(xy)=ba.$$ Therefore, due to $(*),$ all three conditions are met if $k\ge 2$ and, therefore, $G\cong D_k.$
My question is if there are any fallacies or wrong conclusions and if there is a more elegant way of solving the problem.
Best Answer
Your argument looks fine to me. Here is another way of organising what you have done. I prove slightly more than the question asks you to, in order to try and make clear where various properties are necessary. (You can decide if it is more elegant or not, regardless, seeing another way of presenting an argument is useful because you may like some aspects of each presentation better than other!)
Let $G_1 = \langle x,y\mid x^2=y^2 = e\rangle$ and $G_2 = \langle a,b\mid b^2 = baba=(ba)^2=e\rangle$.
a) We have $G_1 \cong G_2$.
Indeed if $\phi_g\colon \{x,y\} \to G_2$ is given by $\phi(x)=ba, \phi(y)=b$ then $\phi_g(x)^2=(ba)^2 =baba= e$ and $\phi_g(y)^2=b^2=e$ so the image of $\phi_g$ satisfies the relations of $G_1$. Thus $\phi_g$ extends uniquely to a surjective homomorphism $\phi\colon G_1 \to G_2$ (where surjectivity follows because its image contains the generators of $G_2$).
Now $a=b.(ba) = \phi(y)\phi(x)=\phi(yx)$, thus if $\phi$ is an isomorphism, its inverse must satisfy $a\mapsto yx, b\mapsto y$. But if $\psi_g \colon \{a,b\}\to G_1$ is given by $\psi_g(a)=yx$ and $\psi_g(b)=y$ then $$ \psi_g(b)^2=y^2=e, \quad (\psi_g(b)\psi_g(a))^2 = (y.yx)^2 = (y^2.x)^2 = x^2 = e, $$ so that the image of $\psi_g$ satisfies the relations of $G_2$ and so $\psi_g$ extends to a homomorphism $\psi\colon G_2 \to G_1$. Then since $\psi\circ \phi = \psi_g \circ \phi_g = \text{id}$ is the identity on the generators of $G_1$ and $\phi\circ \psi = \phi_g\circ \psi_g$ is the identity on the generators of $G_2$, we must have $\psi=\phi^{-1}$ and $G_1$ and $G_2$ are isomorphic.
b) The cyclic subgroup $\langle a \rangle$ has index $2$ in $G_2$.
The relations in $G_2$ can be rewritten as $b^2=e$ and $bab^{-1} = bab = a^{-1}$, that is $ba=a^{-1}b$. It follows by induction that $a^kb^l = b^la^{(-1)^l.k}$, and using this we see that any word $w=a^{k_1}b^{l_1}\ldots,a^{k_r}b^{l_r}$ (where $k_i,l_j \in \mathbb Z$) is equal to a word of the form $b^la^k$ by using this relation, and then $b^2=e$ shows that every word in $G_2$ is equal to one of $\{a^k,ba^k: i \in \mathbb Z\}$. Thus $\langle a \rangle$ is an index $2$ subgroup of $G_2$, and $\langle\psi(a)\rangle =\langle yx \rangle$ is an index $2$ subgroup of $G_1$.
c) If $|G_1|=|G_2|=2k$, then they are isomorphic to $D_{2k}$.
Since $\langle a \rangle$ is index $2$ in $G_2$, Lagrange's theorem shows that $\langle a \rangle$ has order $k$, i.e., $a$ has order $k$. But this says exactly that $a^k=e$ and $a^l\neq e$ for all $l \in \{1,2,\ldots,k-1\}$, hence by the result you quote from your course, $G_2$ (and hence $G_1$) is isomorphic to $D_{2k}$.
Note that the result for your course can be shown using the same ideas as above: $b)$ shows that if $a$ has order $k$, then $|G_2| \leq 2k$ since every element of $G_2$ can be written in the form $ba^l$ for some $l \in \mathbb Z$. On the other hand, if $r$ denotes the rotation by $2\pi/k$ in $D_{2k}$ and $s$ denotes a reflection in an axis of symmetry of a regular $k$-gon, then $r^k=s^2=e$ and $srs=r^{-1}$ since $s$ switches the orientation of $\mathbb R^2$, hence if $\theta_g(a)=r, \theta_g(b)= s$, its image in $D_{2k}$ satisfies the relations of $G_2$, and so it defines a homomorphism $\theta\colon G_2 \to D_{2k}$. Since the image of $\theta$ contains $\langle r \rangle$ and $s$, it has at least $k+1$ elements and it is a subgroup of $D_{2k}$, hence Lagrange's theorem implies the image of $\theta$ must be all of $D_{2k}$, and hence $|G_2| \geq 2k$. Since we already know $|G_2| \leq 2k$ it follows $|G_2| = 2k$ and hence $\theta$ is bijective and so $G_2 \cong D_{2k}$.
Update
In b) I asserted that you could show by induction that $ab=ba^{-1}$ implies $a^kb^l = b^la^{(-1)^l k}$ for any $k,l \in\mathbb Z$, which was quite terse. You can use induction -- first on $k$ with $l=1$ to establish the cases $k\geq 0$ and $l=1$, and then deduce the result for $k<0$. Then you use induction to show the claim for $l geq 1$ and then finally deduce the case $l<0$.
A slightly different way to show the claim is as follows: note first that I can rewrite $ab = ba^{-1}$ as $b^{-1}ab = a^{-1}$. Now the map $\phi(x) = b^{-1}xb$, the map of conjugating by $b^{-1}$, is an automorphism of $G_2$, that is $\phi(g_1)\phi(g_2) = b^{-1}g_1b.b^{-1}g_2b = b^{-1}g_1g_2 b = \phi(g_1g_2)$ and it is invertible because its inverse is clearly $\phi^{-1}(g)= bgb^{-1}$. It follows that $\phi(a^k)= \phi(a)^k = (a^{-1})^k =a^{-k}$, so that $b^{-1}a^k b = a^{-k}$, that is, $a^k b = ba^{-k}$. This gives our claim for $k \in \mathbb Z$ and $l=1$. Finally, noting (by an easy induction say) that $b^{-l}a^kb^l = \phi^l(a^k)$ hence it follows that $b^{-l}a^k b^l = a^{(-1)^l k}$ or $a^k b^l = b^l a^{(-1)^l k}$ as required.