In his Notes on Group Theory, 2019 edition (http://pdvpmtasgaon.edu.in/uploads/dptmaths/AnotesofGroupTheoryByMarkReeder.pdf p. 83 and ff.)
Mark Reeder gives a proof of the non-existence of simple groups of order 720.
P. 83, before the proof, he says : "In the former case, where $n_3(G) = 40$, the normalizer of a Sylow3-subgroup P acts by an involution on P with trivial fixed points, and normalizes every subgroup of P."
A little lower, in the proof of Lemma 10.26, he says :
"If $n_{3}(G) = 40$ then $N_{G}(P)$ contains an element inverting $P$, hence normalizing $Q$."
If I understand it correctly, the reasoning is as follows : if $G$ is a simple group of order 720, if the number of Sylow 3-subgroups of $G$ is 40, then the normalizer $N_{G}(P)$ of a Sylow 3-subgroup $P$ of $G$ has order 18 and is not abelian. So far, so good (the normalizer is nonabelian in view of Burnside's normal complement theorem). M. Reeder seems to find it obvious that this implies that $N_{G}(P)$ is isomorphic either to the dihedral group of order 18 or to the generalized dihedral group constructed on a noncyclic group of order 9. But a nonabelian group $H$ of order 18 can also be isomorphic to the direct product of a group of order 3 with $S_{3}$ and in this case, it is not true that every element of order 2 of $H$ normalizes every subgroup of order 3 of $H$. Thus, for me, the remark of Mark Reeder is not evident.
Mark Reeder gives the following link to a proof by Derek Holt :
http://sci.tech-archive.net/Archive/sci.math/2006-12/msg07456.html
but this link no longer works.
I can prove that $G$ has exactly 10 Sylow 3-subgroups and deduce from this that these Sylow 3-subgroups have trivial pairwise intersections, but my proof is quite long, so, reading M. Reeder, I'm afraid that something is escaping me.
Thus, my question is : can you explain the two sentences of M. Reeder that I quoted above ? Thanks in advance.
By the way, I think that the non-existence of simple groups of order 720 can be proved in the following way. Let us define a colian group as a finite froup G with the following properties :
1° G is simple;
2° the order of G is divisible by 9 and not by 27;
3° the Sylow 3-subgroups of G are in number 10;
4° the Sylow 3-subgroups of G are noncyclic;
5° the Sylow 3-subgroups of G interset pairwise trivially.
The proof given by Cole of the isomorphy of all simple groups of order 360 (or, in any case the variant of this proof given here : https://fr.wikiversity.org/wiki/Th%C3%A9orie_des_groupes/ chapter 35) can easily be extended to the following statements :
1° every simple group of order 360 is colian;
2° every colian group is isomorphic to $A_{6}$;
3° (and thus every simple group of order 360 is isomorphic to $A_{6}$.)
Then we prove that a simple group of order 720 should be colian, and thus should be isomorphic to $A_{6}$, which is absurd since $A_{6}$ has order 360.
Edit 1 (September 18, 2020). There is no problem with this part of Mark Reeder's proof. He proves (lemma 10.16) that if $P$ is an abelian Sylow subgroup of a nonabelian finite simple group $G$, then no non-identity element of $P$ is centralized by $N_{G}(P)$. Thus if $\vert P \vert = 9$, $N_{G}(P)$ cannot be the direct product of a group of order $3$ with a group isomorphic to $S_{3}$.
Edit 2 (September 22, 2020). I think that the end of the proof of lemma 10.26 in M. Reeder's exposition (p. 83-84) can be simplified.
The author assumes that $G$ is a simple group of order 720 and that $Q$ is a subgroup of order 3 of $G$ contained in several Sylow 3-subgroups of $G$ and he needs to draw a contradiction from it.
He proves that $N_{G}(Q)$ has order 72, so $Q$ has exactly 10 conjugates in $G$. Let $X$ denote the set of the conjugates of $Q$ in $G$. Thus, $X$ has cardinality 10 and, as noted by the author, $G$ acts faithfully on $X$ by conjugation. The author proves that the $Q$-orbits in $X$ have sizes 1, 3, 3, 3. Thus, if $t$ is an element of $Q \setminus \{1\}$,
(1) the permutation $M \mapsto tMt^{-1}$ of $X$ has only one fixed point.
The author also notes that, by the $N/C$ theorem, $C_{G}(Q)$ has order 36 or 72.
From here, I would say what follows. Just remember that $C_{G}(Q)$ has even order. That implies that $t$ is the square of an element of order 6. (Choose $a$ of order 2 in $C_{G}(Q)$, then $t$ is the square of $t^{-1}a$ and $t^{-1}a$ has order 6.) Thus $t = u^{2}$, with $u$ of order 6. In view of simplicity of $G$, $u$ acts on $X$ by conjugation as an even permutation of order 6 and thus $t$ acts on $X$ by conjugation as the square of an even permutation of order 6. But an even permutation of order 6 of a set with cardinality 10 has cyclic structure 6-2-1-1, 3-3-2-2 or 3-2-2-1-1-1, thus the square of such a permutation has at least 4 fixed points, which contradicts the result (1) of the author,
If I'm wrong, please say it me.
Edit 3. (October 26, 2020) There is another problem, perhaps more serious, with M. Reeder's proof. See (No simple group of order 720, again).
Edit 4. (March 26, 2023) @Derek Holt. Introducing your proof, you said : "Let me know if it would helpful to include any further details anywhere, or if you can shorten any parts of the proof."
I think your proof is correct, but I can perhaps make some remarks. Here is my first remark. (I will continue if it seems to interest you.)
1° I think that in the proof of Jordan's proposition, it is not necessary to distinguish between the cases $|\Delta \cap g(\Delta)| = 1$ and the other case. Here is a proof without this distinction. It is long, but the reason is perhaps that I try to be exhaustive.
For a finite set $\Omega$ and a subset $E$ of $\Omega$, I will note $Alt_{\Omega}(E)$ the subgroup of $Alt({\Omega})$ formed by the even permutations of $\Omega$ that fix all elements of $\Omega \setminus E$. $Alt_{\Omega}(E)$ is canonically isomorphic to $Alt(E)$. (You note it $Alt(E)$.)
If $\Omega$ is a finite set and $G$ a subgroup of $Sym(\Omega)$, I will say that a subset $E$ of $\Omega$ is richly permutated by $G$ if $Alt_{\Omega}(E)$ is contained in $G$.
The two following lemmas are easy to prove.
Lemma 1. Let $\Omega$ be a finite set, $E$ a subset of $\Omega$ and $\sigma$ an element of $Sym(\Omega)$. Then $Alt_{\Omega}(\sigma(E)) = \sigma Alt_{\Omega}(E) \sigma ^{-1}$.
Lemma 2. Let $\Omega$ be a finite set, $G$ a subgroup of $Sym(\Omega)$, $E$ a subset of $\Omega$ richly permutated by $G$. For every element $g$ of $G$, $g(E)$ is also a subset of $\Omega$ richly permutated by $G$.
Theorem (Jordan). Let $\Omega$ be a finite set and $G$ a primitive subgroup of $Sym(\Omega )$. If $G$ contains a $3$-cycle, then $G$ contains $Alt(\Omega )$ (and is thus equal to $Alt(\Omega )$ or to $Sym(\Omega )$).
Proof. By hypothesis, $G$ contains a $3$-cycle $(x_{1} \ x_{2} \ x_{3})$. Then the subset $\{x_{1}, \ x_{2}, \ x_{3})$ of $\Omega$ is richly permutated by $G$, so there is at least a subset of $\Omega$ that is richly permutated by $G$ and that has cardinality at least $3$ (for example, $\{ x_{1}, \ x_{2}, \ x_{3} \}$ is such a subset). So, among the subsets of $\Omega$ that are richly permutated by $G$ and that have cardinality at least $3$, we can choose one, say $\Delta$, that is maximal for inclusion. We will prove that $\Delta$ is the whole $\Omega$.
Assume, by contradiction, that
(hyp. 1) $\Delta$ is not the whole $\Omega$.
Then, since $| \Delta | > 1$, $\Delta$ is not a trivial block for $G$. Thus, since $G$ is primitive by hypothesis, $\Delta$ is not a block for $G$. So, there is an element $g$ of $G$ such that we can choose
$e_0 \in \Delta \cap g(\Delta )$
and $c_0 \in g(\Delta ) \setminus \Delta$.
For every distinct $a, b$ in $\Delta \setminus \{e_0 \}$, $a, b$ and $e_0$ are three distinct elements of $\Delta$, thus (since $\Delta$ is richly permutated by $G$)
(2) $(a \ b \ e_0 ) \in G$.
On the other hand, $c_0 $ and $e_0 $ are two distinct elements of $g(\Delta )$. Since $|\Delta| \geq 3$, which implies $|g(\Delta )| \geq 3$, we can choose
an element $d_0 $ of $g(\Delta )$ that is distinct from $c_0 $ and $e_0 $.
Since (Lemma 2), $g(\Delta )$ is a subset of $\Omega$ richly permutated by $G$, we have
(3) $(c_0 \ d_0 \ e_0 ) \in G$.
From (2) and (3) results
$(c_0 \ d_0 \ e_0) \ (a \ b \ e_0 ) \ (c_0 \ d_0 \ e_0 )^{-1} \in G$, i.e. (I compose permutations from right to left)
(4) $(a \ b \ c_0 ) \in G$, for every distinct $a$ and $b$ in $\Delta \setminus \{e_0 \}$.
We proved this for every distinct $a$ and $b$ in $\Delta \setminus \{e_0 \}$. Let us prove that is true for every distinct $a$ and $b$ in $\Delta $. We have to prove that (4) is still true if $a$ or $b$ is equal to $e_0 $. In other words, we have to prove the two following theses :
(thesis 5) for every $b$ distinct from $e_0$ in $\Delta $, $( e_0 \ b \ c_0 ) \in G$
(thesis 6) for every $a$ distinct from $e_0$ in $\Delta $, $( a \ e_0 \ c_0 ) \in G$.
Let $b$ be an element distinct from $e_0$ in $\Delta $. Since $|g(\Delta )| \geq 3$, we can choose an element $a_0$ distinct from $b$ in $\Delta \setminus \{ e_{0} \}$. Then, by (2),
(7) $(a_0 \ b \ e_0 ) \in G$.
On the other hand, we have, by (4),
$(a_0 \ b \ c_0 ) \in G$.
With (7), this implies
$(e_0 \ b \ a_0 ) (a_0 \ b \ c_0 ) \in G$, i.e.
(8) $( e_0 \ b \ c_0 ) \in G$,
which proves our thesis (5).
Let now $a$ be an element distinct from $e_0$ in $\Delta $. Since $|g(\Delta )| \geq 3$, we can choose an element $b_0$ distinct from $a$ in $\Delta \setminus \{ e_{0} \}$. Then by (2) et (4),
$(a \ b_0 \ e_0) \ (c_0 \ b_0 \ a) \in G$, i.e.
(9) $(a \ c_0 \ e_0) \in G$.
which proves our thesis (6).
From (4), (8) and (9), it results that for every distinct $a$ and $b$ in $\Delta $, $(a \ b \ c_0 )$ is ìn $G$. Since $\Delta $ is richly permutated by $G$, it proves that every $3$-cycle in $\Delta \cup \{ c_0 \}$ is in $G$ (more rigorously : the canonical image of every such cycle in $Sym(\Omega)$ is in $G$). Since every alternating group is generated by its $3$-cycles, $\Delta \cup \{c_0 \}$ is thus richly permutated by $G$, which contradicts the maximality of $\Delta $. This contradiction shows that our hypothesis (1) is false, so $\Delta$ is the whole $\Omega$, thus $\Omega$ is richly permutated by $G$, which means that $G$ contains $Alt(\Omega)$, which proves Jordan's proposition.
By the way, a primitive group is by definition a transitive group with no non-trivial block. If I'm not wrong, the proof of Jordan's proposition doesn't depend on transitivity, only on the non-existence of non-trivial blocks. So, Jordan's proposition can be stated : "Let $\Omega$ be a finite set and $G$ a subgroup of $Sym(\Omega )$ with no non-trivial block. If $G$ contains a $3$-cycle, then $G$ contains $Alt(\Omega )$ (and is thus equal to $Alt(\Omega )$ or to $Sym(\Omega )$)."
It is not really a strengthening of Jordan's proposition, in the sense that there is no case where the weak hypotheses are satisfied and the strong hypotheses are not, since the " strengthening" of Jordan's proposition shows that transitivity results from the weak hypotheses (since $Alt(\Omega)$ is transitive when $\Omega$ has at least $3$ elements).
Best Answer
Since the link is broken, here is my proof. Let me know if it would helpful to include any further details anywhere, or if you can shorten any parts of the proof.
This question was also discussed in this MO post.
Let $G$ be simple of order $720 = 16 \times 9 \times 5$.
By Sylow, $|{\rm Syl}_3(G)| = 1, 4, 16, 10$ or $40$. It is clearly not $1$ or $4$. By Sylow, all groups of order $45$ are abelian, so $|{\rm Syl}_3(G)|$ cannot be $16$ by BTT (Burnside's Transfer Theorem).
We need to eliminate $|{\rm Syl}_3(G)| = 40$. If $|{\rm Syl}_3(G)| = 40$, then $P \in {\rm Syl}_3(G)$ has an orbit of length $3$ in the conjugation action of $G$ on ${\rm Syl}_3(G)$, so there is a subgroup $Q$ of order $3$ (the pointwise stabilizer of this orbit in $P$) such that $N := N_G(Q)$ has more than one Sylow $3$-subgroup. So it has at least four, and we get $|N| = 36$ or $72$.
If $|N| = 36$ then $N/Q$ has order $12$ and has four Sylow $3$-subgroups, so $N/Q = A_4$, and since $A_4$ cannot act non-trivially on $Q$, $Q$ is central in $N$. Hence $N$ has a normal subgroup $T$ of order $4$, and $|N_G(T)|$ is divisible by $8$, so strictly contains $N$. Then the only possibility is $|N_G(T)| = 72$, but then $Q = O_3(N)$ is characteristic in $N$ and hence normal in $N_G(T)$, contradiction, since $N = N_G(Q)$.
So $|N| = 72$. Since $|{\rm Aut}(Q)| = 2$, $C(Q)$ has order at least $36$, and a subgroup $R$ of order $12$ in $C(Q)$ must be abelian. Consider the action of $G$ on the $10$ cosets of $N$. The subgroup $N$ must be maximal in $G$ because any larger subgroup would have too small an index. So this action is primitive. Let $Q = \langle t \rangle$.
By the result of Jordan proved below, $t$ cannot consist of a single $3$-cycle.
If $t$ consists of two $3$-cycles, then an element $u$ of order $2$ in $R$ must interchange those cycles forming a $6$-cycle $tu$. Since the $6$-cycle is self-centralizing in $S_6$, an element in $R$ outside of $\langle tu \rangle$ must fix all $6$ points of the $6$-cycle, so it must be a single transposition, which is impossible.
If $t$ consists of three $3$-cycles, then an element of order $2$ in $R$ must interchange two of these $3$-cycles and fix the other pointwise, so it consists of three $2$-cycles, and is an odd permutation, which is impossible in a simple group.
So $|{\rm Syl}_3(G)| = 10$. Let $P \in {\rm Syl}_3(G)$ and $N = N_G(P)$, so $|N| = 72$ and $G$ acts transitively by conjugation on ${\rm Syl}_3(G)$, which we denote by $\{ 1,2,\ldots,10 \}$, with $P = 1$, and $N = G_1$ the stabilizer of $1$ in $G$.
If $P$ is cyclic then it must act as a $9$-cycle on $\{ 2,\ldots,10 \}$. Since $|{\rm Aut}(P)| = 6$, there is an element of order $2$ in $N$ which centralizes $P$, and there is no way for such an element to act on $\{ 2,\ldots,10 \}$.
So $P$ is elementary abelian. If a subgroup $Q$ of $P$ of order $3$ fixes more than one point, then $N_G(Q)$ has more than one Sylow $3$-subgroup, giving a contradiction as before.
So $P$ acts fixed-point-freely on $\{ 2,\ldots,10 \}$. In fact we can assume that $P = \langle a,b \rangle$ with $$a = (2,3,4)(5,6,7)(8,9,10),\ b = (2,5,8)(3,6,9)(4,7,10).$$ The stabilizer $S = N_2$ of $2$ in $N$ has order $8$ and is a Sylow $2$-subgroup of $N$. Now $S$ is contained in $X_2$, where $X$ is the normalizer of $P$ in the symmetric group on $\{ 2,\ldots,10 \}$, and $X_2$ can be identified with ${\rm Aut}(P) = {\rm GL}(2,3)$.
Note that the element $(5,8)(6,9)(7,10)$ of $X_2$ is an odd permutation and corresponds to an element of determinant $-1$ in ${\rm GL}(2,3)$. Since ${\rm SL}(2,3)$ is the unique subgroup of index $2$ in ${\rm GL}(2,3)$, it follows that the elements of ${\rm SL}(2,3)$ correspond to the even permutations in $X_2$. So $S$ corresponds to a Sylow $2$-subgroup of ${\rm SL}(2,3)$, which is isomorphic to $Q_8$. In fact ${\rm SL}(2,3)$ has a unique Sylow $2$-subgroup, so $S$ is uniquly determined. In fact $S = \langle c,d \rangle$ with $$c = (3,5,4,8)(6,7,10,9),\ d = (3,6,4,10)(5,9,8,7).$$
Note that $G$ is $3$-transitive, with no elements fixing more than $2$ points.
Now $N_G(S)$ must have order $16$ and contain an element $e$ outside of $S$ containing the cycle $(1,2)$. Now $e$ must also normalize a subgroup of order $4$ in $S$, which we will take to be $\langle c \rangle$. (The argument in the other two cases, $\langle d \rangle$ and $\langle cd \rangle$ is similar.) By multiplying $e$ by an element of $S$, we may assume that $e$ fixes the point $3$. Since $e$ fixes at most two points, it must invert $\langle c \rangle$, and hence contains the cycle $(5,8)$.
So there are just two possibilities, for $e$: $(1,2)(5,8)(6,7)(9,10)$ and $(1,2)(5,8)(6,9)(7,10).$ For the second of these, $be$ fixes $3$ points, which is impossible, so $$e = (1,2)(5,8)(6,7)(9,10),\ {\rm and}\ G = \langle a,b,c,d,e \rangle.$$
In fact, this really is a group of order $720$, but it is the group $M_{10}$, which is not simple: the subgroup $\langle a,b,c,e \rangle$ has order $360$.
A proof of the uniqueness of the simple group of order $360$ follows similar lines to this one, and ends up proving that $G=\langle a,b,c,e \rangle$.
$\mathbf{Proposition}$ (Jordan) If a primitive permutation group $G$ on a set $\Omega$ contains a $3$-cycle then $G$ contains the alternating group ${\rm Alt}(\Omega)$.
Proof. (From Thm 3.3A of "Permutation Groups" by Dixon & Mortimer. It is true even when $\Omega$ is infinite and the proof is not much harder, but I will just prove it in the finite case.) Let $\Delta$ be a largest subset of $\Omega$ such that $G$ contains ${\rm Alt}(\Delta)$ (i.e. the alternating group on $\Delta$ fixing all other points of $\Omega$). By hypothesis we have $|\Delta| \ge 3$. We prove the proposition by showing that $\Delta=\Omega$. Suppose not.
Since $G$ is primitive, there exists $g \in G$ such that $\Delta \cap \Delta^g$ is a non-empty proper subset of $\Delta$. Then $G$ contains ${\rm Alt}(\Delta^g)$. If $\Delta \cap \Delta^g = \{e\}$ has size $1$, then we have $(a,b,e),(c,d,e) \in G$ for any $a,b \in \Delta\setminus \{e\}$, $c,d \in \Delta^g\setminus \{e\}$, and then $(a,e,c) = [(a,b,e),(c,d,e)] \in G$. If $\Delta \cap \Delta^g$ has size bigger than one then we have $(a,b,c) \in G$ with $a,b \in \Delta \cap \Delta^g$ and $c \in \Delta^g \setminus \Delta$.
So in either case, we can find $a,b \in \Delta$ and $c \in \Omega \setminus \Delta$ with $(a,b,c) \in G$. Let $H$ be the subgroup of $G$ generated by ${\rm Alt}(\Delta)$ and $(a,b,c)$. Then $H$ is transitive on $\Delta \cap \{c\}$, and so we have $H={\rm Alt}(\Delta \cap \{c\})$, contradicting the maximality of $\Delta$.