This is a nice exercise...Hints
1) $\,\operatorname{Aut}(N)\,$ is abelian
2) Every inner automorphism of $\,G\,$ is, when restricted to $\,N\,$ , is an element of $\,\operatorname{Aut}(N)\,$
3) $\,\forall x,y\in G\,\,,\,[x,y]^{-1}=[y,x]\,$
Try now to do something with this and, if after thinking it over for a while you're still stuck, write back below as a comment.
Added on request: As noted, $\,\operatorname{Aut}(N)\,$ is abelian and if $\,\phi_g\,$ denotes the inner automorphism determined by $\,g\,$ , then $\,\forall\,g\in G\,\,\,,\,\,\text{then}\;\; \left.\phi_g\right|_N\,\in\operatorname{Aut}(N)$ . We show now that any basic commutator $\,[x,y]\in H\,$ centralizes any element $\,n\in N\,$ :
$$[x,y]n[x,y]^{-1}=[x,y]n[y,x]=x^{-1}y^{-1}xyny^{-1}x^{-1}yx=\left(\phi_{x^{-1}}\phi_{y^{-1}}\phi_x\phi_y\right)(n)=$$
$$\stackrel{\text{Aut}(N)\,\,\text{is abelian!}}=\left(\phi_{x^{-1}}\phi_x\phi_{y^{-1}}\phi_y\right)(n)=Id_N\circ Id_N(n)=n$$
and since the above is true for any generator of $\,H=G'=[G,G]\,$ then it is true for the whole group.
Second solution: Perhaps easier: for any subgroup $\,K\leq G\,$ , the map $$f:N_G(K)\to\operatorname{Aut}(K)\,\,,\,\,f(k):=\phi_k=\,\text{conjugation by}\,\,k$$
is a group homomorphism (with $\,\phi_k(x):=kxk^{-1}\,$), whose kernel is precisely $\,C_G(K)\,$ , and from here
$$N_G(K)/C_G(K)\cong T\leq\operatorname{Aut}(K)$$
In our case, we have $\,N\triangleleft G\Longleftrightarrow N_G(N)=G\,$ , so that we get $\,G/C_G(N)\cong T\leq\operatorname{Aut}(N)\,$ .
But $\,\operatorname{Aut}(N)\,$ is abelian, so that
$$G/C_G(N)\,\,\,\text{is abelian}\,\,\Longleftrightarrow G'\leq C_G(N)\;\;\;\;\;\;\square$$
I assume you haven't seen Lagrange's theorem, as that makes this question much easier. As a result, we can hash through this the old-fashioned way.
Since subgroups are closed under inverses, and the $3$-cycles, $(123),(321)$ are inverses of one another, any subgroup of order $5$ must have both of them and the identity (because otherwise it is missing more than one element, and cannot have size $5$), which means if the subgroup exists, then it is just equal to $S_3\setminus\{x\}$ for some $x\in\{(12),(23),(13)\}$. Let us call this candidate set "$H$".
First note that $(123)=(12)(23)$
Case 1: $(12\not\in H$. Then as $(123),(23)\in H$ and $(123)(23)=(12)$ we have a contradiction.
Case 2: $(23)\not\in H$. Then $(12),(123)\in H$ and $(12)(123)=(23)$, a contradiction.
Case 3: $(13)\not\in H$. Then note $(12),(23)\in H$ and $(12)(23)(12)=(13)$ a contradiction, hence there is no way to have a subset of order $5$ which is also a group.
In the case you have seen Lagrange's theorem and just cannot see how to apply it, we note that subgroups have the property that if $H\le G$ then the order of every element divides the order of the group. But then we know that the identity, $e$ is the only element of order $1$, and we can see directly from examining each element that the orders of all elements of $G$ are one of $1,2$ or $3$. As a result, if we denote by $|x|$ the order of an element $x$, then when $|x|\ne 1$ we must have $|x|\in\{2,3\}$, and neither of these divide $5$, hence there can be no subgroup of that size since non-identity elements of such a subgroup would have to have order dividing $5$, and neither of $2,3$ divide $5$.
Best Answer
A priori $a$ and $b$ need not be distinct, but if $a = b$ then $[a, b] = [a, a] = e$, and so such choices contribute nothing to the subgroup. Applying the definition shows that $[a, b] = [b, a]^{-1}$, so commutator in general depends on the ordering.
Hint For this particular group, note that regardless of the parity of $a, b$, the commutator $[a, b]$ has even parity. So, $S_3'$ is generated by some subset of $A_3$ and hence $S_3' \leq A_3$. At this point, it becomes useful to compute one or more commutators manually.