When D&F say "The image of $g$" in such contexts, they mean its image under the canonical homomorphism that takes $g$ to the coset $gH$. (Here the subgroup $H$ is $\left\langle r^{2}\right\rangle $.)
The next sentence is about the orders of the cosets $r\left\langle r^{2}\right\rangle ,s\left\langle r^{2}\right\rangle $ in the quotient group $D_{2n}/\left\langle r^{2}\right\rangle$, which are indeed $\leq 2$. (Not the orders of $r,s$ in the dihedral group.)
Now, by the important defining relation of the dihedral group $srs=r^{-1}$, one can directly verify that these generators commute. Since they are also of order $\leq 2$ , this implies that $D_{2n}/\left\langle r^{2}\right\rangle$ is abelian. Now, using Proposition 7(4) on page 169, ('the commutator subgroup is the largest abelian quotient'), we conclude $D_{2n}' \leq \left\langle r^{2}\right\rangle$. Therefore, $\left\langle r^{2}\right\rangle =D_{2n}'$
.
It should be obvious that the rotations form a normal subgroup of index 2 (the fact that it is of index 2 is sufficient to prove normality). If we call the rotation subgroup $R$, and the reflection coset $F$, we have:
$RR = R$
$RF = F$
$FR = F$
$FF = R$.
Subgroups containing only rotations are cyclic, due to the fact that $R$ is cyclic. We thus get exactly one subgroup of order $d$ contained in $R$, for each divisor $d$ of $n$.You should prove any (and all) of these subgroups are normal.
Subgroups containing only reflections and the identity must have order a power of 2. Since a reflection times a reflection is a rotation, with $(r^ks)(r^ms) = r^{k-m}$, it should be clear that any such subgroup is in fact of order 2 (we must have $k = m)$. These subgroups are typically NOT normal, but there is an exception for $n = 2$ ($D_2 = V$ is abelian).
Which brings us to "mixed subgroups", containing at least one rotation, and one reflection. These are going to be of the form $\langle r^k,s\rangle$ where $k|n$, that is, isomorphic to $D_m$, where $m = \dfrac{n}{k}$ . You can think of these as symmetries of an $m$-gon, which are also symmetries of an $n$-gon, since $n$ is a multiple of $m$ (the axes of symmetry of an $n$-gon include all the axes of symmetry of the $m$-gon, plus more).
These "mixed subgroups" aren't, in general, normal, but in some special cases they are: for example if $n$ is even and $k = \dfrac{n}{2}$, or $k = 2$.
Another case worth mentioning is when $n$ is an odd prime; in this case, any subgroup containing more than one reflection, or a reflection and a (non-trivial) rotation, is the entire group, which limits the possibilities for subgroups.
For a more complete analysis, see: https://in.answers.yahoo.com/question/index?qid=20091014113730AA7KJDt
Best Answer
If two elements do not commute, their commutator is not the identity. For instance, we have $$aba^{-1}b^{-1} = aba^3b = abba = a^2$$ You can calculate the commutator of any other two elements as well, but they will all give you either $a^2$ or $e$. So the commutator subgroup of $D_8$ (which, for the record, I prefer to write as $D_4$) is $\{e, a^2\}\cong C_2$.
Note that in some groups, the set of commutators is not actually a subgroup, because the product of two commutators is not necessarily itself a commutator. So the commutator subgroup is the subgroup generated by the set of commutators. That's not relevant in this case, thoguh.