By using some more powerful results, it is possible to do this a lot easier.
The two main ingredients for this will be the following:
Burnside's $pq$-Theorem: If only two distinct primes divide the order of $G$, then $G$ is solvable.
Burnside's Transfer Theorem: If $P$ is a $p$-Sylow subgroup of $G$ and $P\leq Z(N_G(P))$ then $G$ has a normal $p$-complement.
So when looking for a non-abelian simple group, the first result immediately tells us that we can assume at least $3$ distinct primes divide the order of $G$.
The way we will use the second result is the following:
Let $P$ be a $p$-Sylow subgroup where $p$ is the smallest prime divisor of $|G|$. We will show that if $|P| = p$ then $P\leq Z(N_G(P))$ (and then the above result says that $G$ is not simple).
To see this, we note that $N_G(P)/C_G(P)$ is isomorphic to a subgroup of $\rm{Aut}(P)$ (this is known as the N/C-Theorem and is a nice exercise), which has order $p-1$, and since the order of $N_G(P)/C_G(P)$ divides $|G|$, this means that $N_G(P) = C_G(P)$ and hence the claim (since we had picked $p$ to be the smallest prime divisor).
The same argument actually shows that if $p$ is the smallest prime divisor of $|G|$, then either $p^3$ divides $|G|$ or $p = 2$ and $12$ divides $|G|$. The reason for this is that if the $p$-Sylow is cyclic of order $p^2$ the precise same argument carries through, since in that case all prime divisors of $|\rm{Aut}(P)|$ are $p$ or smaller.
If $P$ is not cyclic then the order of $\rm{Aut}(P)$ will be $(p^2-1)(p^2 - p) = (p+1)p(p-1)^2$ and the only case where this can have a prime divisor greater than $p$ is when $p = 2$ in which case that prime divisor is $3$, and we get the claim.
In summary we get that the order of $G$ must be divisible by at least $3$ distinct primes, and either the smallest divides the order $3$ times, or the smallest prime divisor is $2$ (actually, by Feit-Thompson, we know this must be the case, but I preferred not to also invoke that), and $3$ must also divide the order.
This immediately gives us $60$ as a lower bound on the order of $G$, and the next possible order would be $2^2\cdot 3\cdot 7 = 84$ and all further orders are greater than $100$. So we are left with ruling out the order $84$ (which you have done), and showing that the only one of order $60$ is $A_5$.
To sum up the comments:
There is a normal subgroup of order $7$ (the $7$-Sylow subgroup).
The $3$-Sylow subgroup is not normal, since otherwise $G$ would be cyclic (and therefore abelian).
Any non trivial (normal) subgroup is of order $3$ or $7$, and thus has to be a Sylow subgroup (in this particular case).
To conclude: $G$ has only one non-trivial normal subgroup.
Best Answer
In a finite abelian group $G$, there's only one $p$ Sylow subgroup for all $p$. This follows from the Sylow theorems: all $p$ Sylow subgroups are conjugate, but in an abelian group, conjugation doesn't do anything. Thus an abelian group of order $34=2 \cdot 17$ will have one $2$-Sylow subgroup and hence one non-identity element $g \in G$ satisfying $g=g^{-1}$.