If there is a homomorphism $\phi: G \to H$ then $G / \ker \phi \cong \phi(H)$. If $\phi$ is not trivial then $\phi(H) \neq 1$.
In your example $G=S_3$ and $H=C_3$ so if a nontrivial homomorphism exists, then $\phi(C_3) = C_3$ because $C_3$ has no nontrivial subgroups. Therefore $|\ker \phi| = 2$. But $S_3$ has no normal subgroups of order 2, so such a homomorphism cannot exist. As the groups get bigger, arguments like this based on the order are in general easier than arguments based on generators.
In general, if $G$ is generated by $\{g_1, g_2, \dots, g_n\}$, and the set map $\phi : \{g_1, g_2, \dots, g_n\} \to H$ maps
$g_i \mapsto h_i$, then it can be extended uniquely to a homomorphism as long as all the relations satisfied by the $g_i$ are also satisfied by the corresponding $h_i$ in $H$.
In your example, you choose three generators of $G$ and in your case $H$ was cyclic of prime order. Since the generators satisfied $g_i^2=1$ but none of the nonidentity elements of $H$ can satisfy $h_i^2=1$ you knew that all the $h_i$ were 1, and so the homomorphism was trivial. As the groups get more complicated there will be more to check. In general this is a slick method only in special cases where $H$ has nice structure and the presentation of $G$ is especially easy.
One generalization that you can prove is this: If $G$ is generated by
$\{g_1, g_2, \dots, g_n\}$ and the order of $g_i$ is relatively prime to $m$ for each $i$, then there can be no nontrivial homomorphism $G \to C_{m}$.
Suppose $\;\gamma_n=1\;$ but $\;\gamma_{n-1}\neq 1\;$ (according to my definition, the correct one, and thus $\;G\;$ is of class $\;n\;$), then
$$\gamma_n:=[\gamma_{n-1},G]=1\iff \forall\,x\in\gamma_{n-1}\;\;and\;\;\forall\,g\in G\;,\;\;x^{-1}g^{-1}xg=1\iff xg=gx\implies$$
$$\implies \gamma_{n-1}\le Z(G)\implies Z(G)\neq 1$$
Best Answer
Hint $:$ Use rank nullity theorem to show that $\text {nullity} (A) = m - \text {rank} (A) \geq m-n > 0$ and observe that $ker\ (\phi)$ is the null space of $A.$