For the subgroup $N$ of $S_3$, $N = \{(1),(123),(132)\}$, I calculate that $(13)N = \{(13),(123),(23)\}$ and $N(13) = \{(13),(23),(12)\}$. Shouldn't this show that $N$ is not a normal subgroup, as opposed to what's printed here?
Normal subgroup of $S_3$
abstract-algebragroup-theorynormal-subgroupsproof-verification
Related Solutions
I assume you haven't seen Lagrange's theorem, as that makes this question much easier. As a result, we can hash through this the old-fashioned way.
Since subgroups are closed under inverses, and the $3$-cycles, $(123),(321)$ are inverses of one another, any subgroup of order $5$ must have both of them and the identity (because otherwise it is missing more than one element, and cannot have size $5$), which means if the subgroup exists, then it is just equal to $S_3\setminus\{x\}$ for some $x\in\{(12),(23),(13)\}$. Let us call this candidate set "$H$".
First note that $(123)=(12)(23)$
Case 1: $(12\not\in H$. Then as $(123),(23)\in H$ and $(123)(23)=(12)$ we have a contradiction.
Case 2: $(23)\not\in H$. Then $(12),(123)\in H$ and $(12)(123)=(23)$, a contradiction.
Case 3: $(13)\not\in H$. Then note $(12),(23)\in H$ and $(12)(23)(12)=(13)$ a contradiction, hence there is no way to have a subset of order $5$ which is also a group.
In the case you have seen Lagrange's theorem and just cannot see how to apply it, we note that subgroups have the property that if $H\le G$ then the order of every element divides the order of the group. But then we know that the identity, $e$ is the only element of order $1$, and we can see directly from examining each element that the orders of all elements of $G$ are one of $1,2$ or $3$. As a result, if we denote by $|x|$ the order of an element $x$, then when $|x|\ne 1$ we must have $|x|\in\{2,3\}$, and neither of these divide $5$, hence there can be no subgroup of that size since non-identity elements of such a subgroup would have to have order dividing $5$, and neither of $2,3$ divide $5$.
A priori $a$ and $b$ need not be distinct, but if $a = b$ then $[a, b] = [a, a] = e$, and so such choices contribute nothing to the subgroup. Applying the definition shows that $[a, b] = [b, a]^{-1}$, so commutator in general depends on the ordering.
Hint For this particular group, note that regardless of the parity of $a, b$, the commutator $[a, b]$ has even parity. So, $S_3'$ is generated by some subset of $A_3$ and hence $S_3' \leq A_3$. At this point, it becomes useful to compute one or more commutators manually.
Additional hint Computing the commutator of the transpositions $(12)$ and $(13)$ gives that $S_3'$ contains $$[(12), (13)] = (12)(13)(12)^{-1}(13)^{-1} = (12)(13)(12)(13) = (123) .$$ This $3$-cycle has order $3$ and so generates $A_3$, so $S_3' \geq A_3$ and thus $S_3' = A_3$.
Best Answer
Your computation of $(13)N$ is wrong. The $(123)$ should be $(12)$. One way to check this is that every permutation in $N$ is even, so the coset should consist only of odd permutations.