Let $\,Q_p\,$ be a Sylow p-sbgp. of $\,G\,$ . If either $\,Q_5\,,\,Q_7\,$ is normal then the product $\,P:=Q_5Q_7\,$ is a sbgp. of $\,G\,$ and thus it is normal as well, as its index is the minimal prime dividing $\,|G|\,$. In this case, and since the only existing group of order $35$ is the cyclic one, we'd get that $\,|\operatorname{Aut}(P)|=\phi (35)=24$, and thus we can define a homomorphism $\,Q_3=\langle c\rangle\to\operatorname{Aut}(P=\langle d\rangle)\,$ by $\,d^c:= c^{-1}dc\,$ . As this homom. isn't trivial (of course, we assume at least one of the Sylow sbgps. is non-abelian) we get a (non-abelian, of course) semidirect product $$ P\rtimes Q_3 $$ .
If both $\,Q_5,Q_7\,$ are not normal, a simple counting argument tells us there are $\,90\,$ elements of order $7$ and $84$ elements of order 5, which is absurd as we've only $\,105\,$ elements in the group.
Thus, the above is the only way to get a non-abelian group of the wanted order.
To get you started some rather basic (or even trivial, depends) observations, putting $\,n_p=$ number of Sylow $p-$subgroups , $\,G:=\,$ simple group of order $\,168\,$:
$$168=2^3\cdot 3\cdot7\Longrightarrow n_7=8\;,\;n_3=7,28$$
It can't be $\,n_3=4\,$ as then $\,G\,$ has a subgroup of index 4, which is impossible since this would mean $\,G\,$ is isomorphic with a subgroup of $\,S_4\,$ . For the same reason, it must be $\,n_2=7,\,21\,$...
Perhaps reading here you'll have have the whole view. Don't worry about the number of pages as the first ones are basic results listed.
Best Answer
If $n_5=6$ and $P$ is a Sylow 5-subgroup, you have $|N_G(P)|=60$. Set $H=N_G(P)$, and we find a subgroup of order $10$ in $H$. Your arguments work. In $H$, $n_5$ is either $1$ or $6$. If $n_5=1$, multiply Sylow $5$-subgroup with a cyclic of order $2$. If $n_5=6$ and $Q$ is Sylow $5$-subgroup, then $|N_H(Q)|=10$. So, $N_H(Q)\leq H\leq G$ is a subgroup of order $10$.