Nonabelian finite simple groups come in a few types:
- Alternating groups
- Classical groups in odd characteristic
- Exceptional groups in odd characteristic
- Groups in even characteristic (classical or exceptional)
- Sporadic groups
In cases 1,2,3 the Sylow 2-subgroups are (slightly deformed versions of) direct products of wreath products $P_2 \wr C_2 \wr C_2 \wr \ldots \wr C_2$ where $P_2$ is the Sylow $2$-subgroup of a tiny group from the family. In case 4, the groups are best understood using linear algebra. In case 5, it would be nice to know which sporadics “borrow” a Sylow 2-subgroup and which have their own unique Sylow 2-subgroup.
Alternating
The Sylow 2-subgroups of the symmetric groups are direct products of wreath products of Sylow 2-subgroups of $S_2$ -- this was known in the 19th century. The Sylow 2-subgroups of the alternating groups are index 2 subgroups.
For $n=4m+2$ and $n=4m+3$, the copies of $S_{4m}$ inside $A_n$ have odd index $2m+1$ or $(4m+3)(2m+1)$, so the Sylow 2-subgroup of $S_{4m}$ is isomorphic to the Sylow 2-subgroups of $A_{4m+2}$ and $A_{4m+3}$.
Weisner (1925) computes the order of the normalizers of the Sylow $p$-subgroups of symmetric and alternating groups (so counts them). The main result for us is that Sylow 2-subgroups are self-normalizing in simple alternating groups (except $A_5$ with normalizer $A_4$).
Weir (1955) computes the characteristic subgroups of the Sylow $p$-subgroup of the symmetric groups, but only for odd $p$. Lewis (1968) modifies this to handle $p=2$ for both symmetric and alternating groups. Dmitruk–Suščanskʹkiĭ (1981) take the approach of Kaloujnine (1945-1948), again handling $p=2$ and alternating groups.
Harada–Lang (2005) observes that the Sylow 2-subgroups of $A_{4m}$ and $A_{4m+1}$ are directly indecomposable (while those of $A_{4m+2}$ and $A_{4m+3}$ are directly indecomposable iff $m$ is a power of $2$).
- Weisner, Louis;
“On the Sylow Subgroups of the Symmetric and Alternating Groups.”
Amer. J. Math. 47 (1925), no. 2, 121–124.
MR1506549
DOI:10.2307/2370639
- Kaloujnine, Léo
“La structure des p-groupes de Sylow des groupes symétriques finis.”
Ann. Sci. École Norm. Sup. (3) 65, (1948). 239–276.
Also see: C. R. Acad. Sci. Paris
221 (1945), 222–224; ibid.
222 (1946), 1424–1425; ibid.
223 (1946), 703–705; ibid.
224 (1947), 253–255.
- Weir, A. J.
“The Sylow subgroups of the symmetric groups.”
Proc. Amer. Math. Soc. 6 (1955), 534–541.
MR72142
DOI:10.2307/2033425
- Lewis, Robert Edward.
“On the Sylow two-subgroups of the alternating groups.”
Thesis (Ph.D.)–University of Illinois at Urbana-Champaign. 1968. 48 pp.
MR2617989
- Dmitruk, Ju. V.; Suščanskʹkiĭ, V. Ī.
“Construction of Sylow 2-subgroups of alternating groups and normalizers of Sylow subgroups in symmetric and alternating groups.”
Ukrain. Mat. Zh. 33 (1981), no. 3, 304–312.
MR621637
- Harada, Koichiro; Lang, Mong Lung.
“Indecomposable Sylow 2-subgroups of simple groups.”
Acta Appl. Math. 85 (2005), no. 1-3, 161–194.
MR2128910
DOI10.1007/s10440-004-5618-0
Classical groups in odd characteristic
There is a huge difference in Sylow $p$-subgroup structure depending on whether $p$ is the characteristic of the field. In this section we assume $p$ is not the characteristic of the field.
In case $p$ is not the characteristic, then Weir (1955) showed that symmetric groups and classical groups are very similar, but again $p=2$ was left out until Carter-Fong (1964), and then more uniformly in Wong (1967). Algorithms to handle all Sylow $p$-subgroups of classical groups are described in Stather (2008).
The gist is that in GL, GO, GU, and Sp, the Sylow $p$-subgroups are direct products of wreath products of cyclic groups of order $p$ with the Sylow $p$-subgroup of the two-dimensional groups. For SL, SO or $\Omega$, SU the answers are more complicated, but only because an easy to understand part has been chopped off the top.
Exceptional groups in odd characteristic
Sylow 2-subgroups for finite groups of Lie type are similar to the classical case: there is a 2-dimensional group $P_2$ and a “top” group $X$ (which need not be $C_2 \wr C_2 \wr \ldots \wr C_2$, but that is probably the correct picture to have) such that the $X$-conjugates of $P_2$ are commute with each other, so that $X \ltimes P_2^n$ is a Sylow 2-subgroup. The $P_2$ are the Sylow 2-subgroups of the so-called “fundamental subgroups” of Aschbacher (1977), where we view groups of Lie type as built up from rank 1 groups, in this case commuting rank 1 subgroups isomorphic to SL2. These are used in Aschbacher (1980) to describe groups in which a Sylow 2-subgroup is contained in a unique maximal subgroup, and Harada–Lang (2005) describes which Sylow 2-subgroups are indecomposable. GLS I.A.4.10 covers Aschbacher's ideas as well.
- Aschbacher, Michael.
“A characterization of Chevalley groups over fields of odd order.”
Ann. of Math. (2) 106 (1977), no. 2, 353–398.
MR498828
- Aschbacher, Michael.
“A characterization of Chevalley groups over fields of odd order. II.”
Ann. of Math. (2) 106 (1977), no. 3, 399–468.
MR498829
- Aschbacher, Michael.
“On finite groups of Lie type and odd characteristic.”
J. Algebra 66 (1980), no. 2, 400–424.
MR593602
DOI:10.1016/0021-8693(80)90095-2
- Gorenstein, Daniel; Lyons, Richard; Solomon, Ronald.
The classification of the finite simple groups. Number 3. Part I. Chapter A.
Almost simple K-groups. Mathematical Surveys and Monographs, 40.3. American Mathematical Society, Providence, RI, 1998. xvi+419 pp. ISBN: 0-8218-0391-3
MR1490581
- Harada, Koichiro; Lang, Mong Lung.
“Indecomposable Sylow 2-subgroups of simple groups.”
Acta Appl. Math. 85 (2005), no. 1-3, 161–194.
MR2128910
DOI10.1007/s10440-004-5618-0
Groups in characteristic 2
Here the Sylow 2-subgroups are basically groups of upper triangular matrices and are often best understood in terms of linear algebra. Weir (1955) describes the characteristic subgroups and those normalized by important subgroups of GL. These general ideas work in all the groups of Lie type. The main description I know is Chevalley's commutator formula, as explained in Carter (1972).
XXX: Decent reference for the classical, and then the exceptional. Maybe specifically handle Suzuki and Ree.
Sporadic
I think each one is a special snowflake. XXX: Lookup coincidences in Sylow structure.
This is more delicate and difficult than most problems about Sylow subgroups, and it must be tackled by relating the 5-sylow subgroups and the 3-sylow subgroups. In general such a problem in group theory at the undergraduate level is either going to be trivially easy, or require this technique.
As you established $n_3 = 1$ is one possibility but it was noted in the comments that $n_3 = 10$ is also possible. Suppose $n_5 = 6$ so that the $5$-sylows are not normal. Then they must be their own normalizers because the number of them is equal to the index of the normalizer of any one, and they are the unique subgroups of order $5$. However the fact that only $5$ divides $30$ and not $25$ allows us to know that all of the elements of order $5$ in this group are contained in a $5$-sylow, and that these sylows have trivial intersection. But that means there are $(5-1)*6 = 24$ elements of order $5$ in this group. But this means that there cannot possibly be $10$ subgroups of order $3$ otherwise there would be $20$ elements of order $3$ and that would give $47$ elements in the group which would be absurd.
So we know that the $3$-sylows would have to be normal in this situation. Then we have that if $H_3$ is a $3$-sylow and $H_5$ is a $5$-sylow then $H_5H_3$ is a subgroup because $H_3$ is normal. But then $H_5H_3/H_3 \cong H_5/(H_5 \cap H_3)$ and counting orders we know that $H_5H_3$ has order $15$. But there is a unique group of order $15$ (there are many ways to show this, but in our particular situation the fact that $H_5H_3 \cong \mathbb{Z}_{15}$ comes down to the fact that $H_5$ is normal by a sylow argument on this subgroup) and thus this group is abelian and this contradicts that $H_5$ is its own normalizer. Thus $H_5$ is normal.
Finally we can now count again. Since $H_5$ is normal we can take any one of these $H_3$ and do the same argument as the above to realize that $H_3$ has a normalizer of order at least $15$ and thus $H_3$ must be normal because it is supposed to have normalizer or index either $1$ or $10$.
Hope this helps.
Best Answer
In general: let $S \in Syl_p(G)$ and $H \leq G$, with $N_G(S) \subseteq H$, then $N_G(H)=H$. Moreover, $[G:H]\equiv 1$ mod $p$.
Proof for the first part it suffices to show that $N_G(H) \subseteq H$: observe that $S \in Syl_p(H)$ and take a $g \in N_G(H)$. Then $S^g \subseteq H$ and hence $S^g=S^h$ for some $h \in H$. That means $gh^{-1}\in N_G(S)$, so $g \in hN_G(S) \subseteq H$.
For the second part we use the fact that in general for a $p$-subgroup $P$ of $G$, it holds that $[G:P]\equiv[N_G(P):P]$ mod $p$ (this can be shown by letting $G$ act by right multiplication on the right cosets of $P$). Further, since $N_G(S) \subseteq H$, $N_H(S)=N_G(S) \cap H = N_G(S)$, so the number of Sylow $p$-subgroups of $G$ and $H$ are equal. But $[G:S]=[G:H][H:N_G(S)][N_G(S):S]$ and taking the equation mod $p$ and using the fact that the number of Sylow $p$-subgroups $\equiv 1$ mod $p$ yields the required result.