Let $G$ be a p-group $|G| = p^n$. Then G is nilpotent in the following sense:
Let $C^1(G) = [G,G] = G'$
And $C^n(G) = [G,C^{n-1}]$.
G is nilpotent iff there is a $n_0$ such that $C^n = \{e\}$.
What I proved in previous exercises is:
If H is a proper subgroup of a nilpotent group then H is a normal proper subgroup of its normalizer. i.e.
if $H \lneq G$ then $H \lneq N_G(H)$.
I use induction: If $|G| = p$, then G is isomorphic to $\mathbb{Z}/p\mathbb{Z}$ which is cyclic. therefore $C^1(G) = G' = \{e\}$.
Suppose that the following assertion is true:
if $|L| = p^i$, for $i \lt n$ then L is nilpotent.
Let G be a group of order $p^n$. If I can just prove $G' \neq G$, then we'd have $|G'| = p^i, i \lt n$ and in such a case G' would be nilpotent. So that G would be nilpotent.
Is it? If G is a p-group, then $G'\neq G$?
Best Answer
A very easy way to prove that $G'\neq G$ is to note that $G'$ is the smallest normal subgroup of $G$ such that $G/G'$ is abelian. If you can find any proper normal subgroup of $G$, $N$, such that $G/N$ is abelian, then it will follow that $G'\subseteq N$, and so $G'\neq G$.
For example: have you proven that a maximal subgroup of a $p$-group must be normal? This can be done easily by induction, using the fact that $Z(G)$ (the center) is nontrivial. With that in hand, it follows that $G'$ is always contained in a maximal subgroup of $G$, and you'd be done.
(The previous exercise you quote can't help you because it assumes your group is nilpotent, and here you are trying to prove that a group is nilpotent).
Added. Another way of proving this:
Show that if $f\colon G\to K$ is an onto group homomorphism, then $f([G,G])=[K,K]$. In particular, if $[K,K]\neq K$, then $[G,G]\neq G$. Then use the fact that if $G$ is a $p$-group then $Z(G)\neq \{1\}$ and induction to show that either $Z(G)=G$ (so $[G,G]=\{1\}$) or else $[G/Z(G),G/Z(G)]\neq G/Z(G)$ and get the result.
Note: Your notation for the lower central series is uncommon. It is more common to use $G_1=G$ and $G_{n+1}=[G_n,G](=[G,G_n])$.
Added. I wasn't paying enough attention when I answered this, and what Derek Holt points out in comments is quite true: it is not enough to prove that $[G,G]$ is nilpotent in order to prove that $G$ is nilpotent, even if $[G,G]\neq G$: there are groups where $[G,G]$ is nilpotent, but $G$ is not. For example, $S_3$ has $[G,G]\cong A_3$, which is abelian, but $S_3$ is not nilpotent.
Instead, we can proceed by induction on $n$, where $|G|=p^n$.
If $n=1$ or $n=2$, then $G$ is abelian, hence $G$ is nilpotent.
Assume that $G$ has order $p^n$. If $Z(G)=G$, then $G$ is abelian and hence is nilpotent. Otherwise, consider $\mathfrak{G}=G/Z(G)$. This is a $p$-group of order strictly less than $p^n$ (since the center of a finite $p$-group is always nontrivial), so by the induction hypothesis, $\mathfrak{G}$ is nilpotent. Now we use the lemma mentioned above, slightly strengethened (I will use $H_n$ to denote the $n$th term of the lower central series of $H$, that is, $H_n$ is what the original post calls $C^n(H)$):
Now, since $\mathfrak{G}$ is nilpotent, there exists $r$ such that $\mathfrak{G}_r$ is trivial; therefore, by the lemma, $f(G_r)$ is trivial, so $G_rZ(G) = Z(G)$. Hence $G_r\subseteq Z(G)$, so $G_{r+1}=[G,G_r]\subseteq [G,Z(G)] = \{1\}$, so $G$ is nilpotent, as claimed. $\Box$