Prove that if $G$ is a finite group in which every proper subgroup is nilpotent, then $G$ is solvable. (Hint: Show that a minimal counterexample is simple. Let $M$ and $N$ be distinct maximal subgroups chose with $|M\cap N|$ as large as possible and apply Part 2 of Theorem 3. Now apply the methods of Exercise 53 in Section 4.5.)
This is Exercise 6.1.35 in Dummit and Foote. Using the idea from the hint, I tried the following proof. But I couldn't prove that $M\cap N=1$. Does anyone know how to prove this? Thanks.
Here is what I have done so far:
We proceed by induction. If $|G|=2$, then $G$ is clearly solvable. Let $|G|\geq6$. Assume that the statement is true for all groups of order $<|G|$.
If $G$ is of prime order, then clearly $G$ is solvable. So we assume that $G$ is not of prime order. Since $G$ is finite, $G$ contains nontrivial maximal subgroups.
Claim: There exists a maximal subgroup of $G$ which is normal. Suppose not. Since conjugates of a maximal subgroup are maximal subgroups, $G$ has more than one maximal subgroups. Let $M$ and $N$ be the distinct maximal subgroups such that $|M\cap N|$ is maximal. Since $M$ and $N$ are nilpotent, $M\cap N<N_M(M\cap N)$ and $M\cap N<N_N(M\cap N)$. (Here I want to show that $M\cap N=1$ following the hint.)
Now since $G\neq\bigcup_{g\in G}gMg^{-1}$, there exists $H\leq G$ maximal such that $H$ is not a conjugate of $M$. So $G$ has at least the following number of nonidentity elements:
\begin{equation*}
\begin{split}
(|M|-1)|G:N_G(M)|+(|H|-1)|G:N_G(H)|=&(|M|-1)|G:M|+(|H|-1)|G:H|\\=&2|G|-|G:M|-|G:H|\\\geq&2|G|-\frac{1}{2}|G|-\frac{1}{2}|G|=|G|
\end{split}
\end{equation*}
which is a contradiction. Hence there exists a maximal subgroup of $G$ which is normal.
Now let $M\unlhd G$ be a maximal subgroup. Then $M$ is nilpotent and hence solvable. Now $|G/M|<|G|$. Since every subgroup of $G$ is nilpotent, by the correspondence theorem, every subgroup of $G/M$ is nilpotent. So $G/M$ is solvable. Hence $G$ is solvable.
Best Answer
Your proof is not complete and also incorrect. Here is a proof. If $G$ is not solvable, then one of the composition factors $B/A$ is simple non-Abelian. If $B$ is a proper subgroup then $B$ is nilpotent, hence $B/A$ is nilpotent, a contradiction. So $B=G$. Similarly if $A\ne 1$, then $|G/A|<|G|$, all proper subgroups of $G/A$ are nilpotent, hence $G/A$ is solvable.Since $A$ is nilpotent, $G$ is solvable.
Thus $G=G/A=B/A$ is simple non-Abelian.
The shortest way to finish is then by using J. Thompson's famous theorem about classification of all simple finite groups where all proper subgroups are solvable. Each of these groups contains non-nilpotent solvable subgroups. QED
Now a longer way suggested by D-F. $G$ contains at least two maximal subgroups $M,K$ since $G$ is simple. Assume that $L=M\cap K$ is maximal possible. Since $M,K$ are nilpotent $N_1=N_M(L)>L<N_2=N_K(L)$ (Theorem 3 in D-F). If $L$ is not 1, its normalizer in $G$ is not $G$ (again because $G$ is simple), whence that normalizer must be inside some maximal subgroup $K'$ of $G$. But then $K'\ge N_1$, hence $K'\cap M$ is bigger than $L$, a contradiction. Thus $L=1$ and the intersection of any two maximal subgroups of $G$ is trivial.
We can assume that $M$ and $K$ are not conjugate. Note that $N_G(M)=M, N_G(K)=K$ since these subgroups are maximal and not normal. Hence there are $[G:M]$ conjugates of $M$ each two intersecting trivially and there are $[G:K]$ conjugates of $K$ each two intersecting trivially. Altogether these subgroups contain $2|G|-[G:K]-[G:M]+1$ elements. Note that the indices of these maximal subgroups are not bigger than $|G|/2$ by Lagrange's theorem. So $2|G|-[G:K]-[G:M]+1\ge |G|+1$ a contradiction.