I've thought of the following solution to this assignment: find the number of the group homomorphisms $G\to S_3$, where $|G|=8$.
Let's call $\mathbb{h}$ the sought number. Any such a homomorphism is equivalent to an action of $G$ on $X:=\{1,2,3\}$. The only allowed orbit equations are $3=1+1+1$ and $3=1+2$, because $3=3$ is ruled out by the orbit-stabilizer theorem and the fact that $3\nmid 8$. The former corresponds to the trivial homomorphism, so it counts one. The latter corresponds to two stabilizers of order $4$, hence normal (index $2$), hence equal (as they are conjugate); therefore, each subgroup of order $4$ of $G$ gives rise to one such an action, or rather to three, considered that the singleton's role ("$3=1+\dots$") can be played by any of the three elements of $X$. So, said $\mathbb{n}_4$ the number of subgroups of order $4$ of $G$, we'd have:
$$\mathbb{h}=3\mathbb{n}_4+1 \tag 1$$
Unless it is a coincidence, $(1)$ would account for the well-known four homomorphisms $C_8\to S_3$ (where $\mathbb{n}_4=1$) and the ten $D_4\to S_3$ (where $\mathbb{n}_4=3$).
Is my argument correct and then $(1)$ valid for every $G$ of order $8$?
Edit. I think that the argument and $(1)$ work. In fact, the subgroups of order $4$ of $G$ are the only proper, normal (index $2$) subgroups suitable as kernels of nontrivial homomorphisms $\phi\colon G\to S_3$ (kernels of order $2$ would lead to $|\phi(G)|=4(\nmid 6)$ and the trivial kernel to $|\phi(G)|=8(> 6)$). For each kernel of order $4$, say $H_4$, we have that the elements of $G\setminus H_4$ have order either $2$ or $4$ or $8$; therefore, they must be mapped to one same $2$-cycle of $S_3$; in fact, they cannot be mapped to any $3$-cycle (as $3\nmid 2,4,8$), nor to different $2$-cycles only (as $\phi(G)$ wouldn't be a subgroup of $S_3$). So, for each $H_4$, there are three homomorphisms, each mapping all the elements of $G\setminus H_4$ to a given $(ij)$.
Best Answer
I think your argument is basically correct.
More generally: Let $G$ and $H$ be groups.
The number of homomorphisms $\psi: G \rightarrow H$ with $\operatorname{Im} \psi \cong C_2$ is equal to $tk$, where $t$ is the number of elements of order $2$ in $H$, and $k$ is the number of non-trivial homomorphisms $G \rightarrow C_2$.
(In your case $t = 3$.)
For non-trivial homomorphisms $\psi, \psi': G \rightarrow C_2$ you have $\psi = \psi'$ if and only if $\operatorname{Ker} \psi = \operatorname{Ker} \psi'$. So $k$ is equal to the number of subgroups of index $2$ in $G$. (Using the notation in your case, $k = n_4$.)
What remains is for you to calculate the number of subgroups of index $2$ for the groups of order $8$ (of which there are five). This you will need to do case-by-case.