First some background: Some time ago I learned that since any finite abelian group is a direct product of (finite) cyclic groups, the finite cyclic groups ($\mathbb{Z}_n$) are the key to understanding all finite abelian groups. This makes sense to me on a philosophical level. It makes sense that when we learn group theory we spend a lot of time on the cyclic groups, because they are the key to all finite abelian groups.
I have heard again and again that simple groups are likewise the key to understanding all finite groups. But I still fail to understand how this works.
To my question: Recently I came across the concept of a nilpotent groups. I read somewhere that nilpotent groups are "almost abelian". (I am not sure what that means.) But I did come across the result that any finite nilpotent group is the direct product of $p$-groups (groups of prime power order $p^n$.)
My big question is: is it correct to compare this result to the result about the finite abelian groups?
My second question is: How much is know about $p$-groups? I assume they are not as elementary to approach as finite cyclic groups (where basically any question has an answer).
Best Answer
The study if finite nilpotent groups does indeed come down to the study of finite $p$-groups. If you understand $A$ and $B$, and $\gcd(|A|,|B|)=1$, then it is very easy to understand $A\times B$ and all its subgroups (thanks to Goursat's Lemma, if necessary). Since a finite nilpotent group is the product of its Sylow $p$-subgroups (a finite group is nilpotent if and only if its Sylow $p$-subgroups are normal), it follows that understanding finite $p$-groups would be enough to understand finite nilpotent groups.
Unfortunately, "understanding finite $p$-groups" turns out to be not just hard, but very hard. There are a lot of them, and their structure varies quite a bit. In the 1940s Philip Hall attempted a wholesale classification along the lines of the classification for finitely generated abelian groups; he divided finite $p$-groups into "regular" and "irregular" $p$-groups. For the first, he succeeded in providing a description (but not full classification) that parallels the one for abelian groups; he promised a similar treatment of irregular $p$-groups, but that never came about: they turned out to be much too hard to work with.
It is only recently that some sort of general order/coarse-grain picture has been obtained for finite $p$-groups, thanks to the Coclass conjectures (now theorems). Even so, they only provide a general picture of $p$-groups, not a specific description of them.
As to comparisons: I would say that "finite nilpotent groups are products of $p$-groups" is comparable to "finite abelian groups are the product of their $p$-parts" (which is part, but not all, of the classification of finitely generated abelian groups when you decompose them into primary divisors), but not to the full classification into products of cyclic groups. Unfortunately, the problem is much, much, much harder than dealing with abelian groups.
You can say a lot about different types of $p$-groups, but a "classification" along the lines of the abelian groups just does not exist, and very few people believe that one is possible. It is a folk conjecture that given any finite collection of group invariants, you can find two finite $p$-groups that have identical values for the invariants, but are not isomorphic.
You can write multiple volumes of books giving results and asking thousands of questions about the structure of finite $p$-groups. For instance, this has long occupied Yakov Berkovich and Zvonimir Janko, who have written a series of books on the topic which is up to six volumes as of 2018. Here is a review of the first two volumes by Leedham-Green, which appeared in the Bulletin of the AMS. (Link is to a PDF file). The review also discusses "what can be said about the number of isomorphism types of groups of order $p^n$ for fixed $p$ and $n$?".
Why do we say nilpotent groups are "almost abelian"? Nilpotent groups can be constructed from abelian groups in a very simple way (and this will play into your other aside about simple groups).
We say a group $G$ is an extension of $A$ by $B$ if and only if there is a normal subgroup $N\triangleleft G$ such that $A\cong N$ and $G/N\cong B$. (Careful, as this situation is not symmetric, but is sometimes defined as being an extension of $B$ by $A$; see this discussion). We say the extension is central if the subgroup $N$ is contained in $Z(G)$, the center of $G$.
Abelian groups are "nilpotent groups of class $1$".
A "nilpotent group of class $2$" is a nonabelian group $G$ that is a central extension by abn abelian group; that is, an extension of an abelian group $A$ by an abelian group $B$, and such that the normal subgroup $N$ of $G$ is central. Equivalently, if $G/Z(G)$ is abelian, where $Z(G)$ is the center of $G$. (I need to say "nonabelian", since $A\times B$ is an extension of $A$ by $B$, but would be abelian if $A$ and $B$ are abelian).
A "nilpotent group of class $3$" is a group that is not nilpotent of class $2$, and is central extension of an abelian group by a nilpotent group of class $2$.
More generally, a central extension of an abelian group by a nilpotent group of class $k$ which is not itself of class $k$ is a nilpotent group of class $k+1$.
Every nilpotent group has a class, which is a well-defined invariant (isomorphic groups have the same class).
Might as well explain how simple groups come into this, as I already gave you all the ingredients.
The idea was to do the following:
The idea then would be to take an arbitrary finite group $G$. If it is simple, we understand it by step 1. If it is not simple, then it has a normal subgroup $N$ which is proper and nontrivial. Then $G$ is an extension of $N$ by $G/N$, both of which are strictly smaller than $G$; assuming we understand them because they are smaller than $G$, use step 2 to leverage that understanding into an understanding of $G$.
The Jordan-Hölder Theorem guarantees that if $G$ is a finite group, then there is a unique multiset of simple groups that are "involved" in the group $G$, in that if we repeat the process above until we get simple groups, those are the groups that will show up. Thus, presumably accomplishing both steps 1 and step 2 could be leveraged to understanding $G$.
Unfortunately, "understand all finite simple groups" turned out to be much harder than originally expected, leading to a proof that is thousands of pages long and took decades to complete, the Classification of Finite Simple Groups.
But even more problematic, understanding all possible extensions of $N$ by $K$, even given a complete understanding of both $N$ and $K$, turns out to be really hard. This is known as the Extension problem. We are very far from being able to achieve anything even remotely approaching a solution (see also discussion here), so that the classification of finite simple groups turns out to not be all that conducive towards understanding all groups. Moreover, for finite $p$-groups the only finite simple group that results is $C_p$ (cyclic of order $p$), which we understand extremeley well, and yet we already run into problems once we try to iterate the extension process six or seven times, let alone fifteeen, or a hundred and two.