I'm trying to prove the following problem:
(a) Let $G$ be a non-abelian $p$-group with an abelian subgroup of index $p$. Then the number of abelian subgroups of $G$ of index $p$ is either $1$ or $p+1$; in the latter case the center of $G$ has index $p^2$.
(b) A nilpotent group of class $3$ and order $16$ has exactly one cyclic subgroup of index $2$.
I've made some progress towards (a) and tried to use (a) to show (b). But despite thinking about this for a few days I'm missing something and I can't quite finish off my argument. Here is an outline of the main results that I have managed to show; essentially, I think I've shown that for part (a), if there is more than one than there must be at least $p+1$.
Assume that there is more than one abelian subgroup of index $p$. Let $H$ and $K$ be two of them. Then they are both maximal and hence normal in $G$, and then $G=HK$. Then I was able to show that $H \cap K \leq Z(G)$, and then it follows that $Z(G)$ has index $p^2$ (as $G$ is non-abelian).
So now I want to show that there are exactly $p+1$ abelian subgroups of index $p$. Suppose $L$ is a subgroup of $G$ of index $p$ containing $H \cap K$. Then $L \lhd G$. Since $H \cap K \lhd G$ and it has index $p^2$, then $L/(H \cap K)$ is cyclic of order $p$. From this I was able to show that $L$ is abelian.
Now consider $G/(H \cap K)$, which has order $p^2$; however it can't be cyclic since it has two subgroups of order $p$, namely $H/(H \cap K)$ and $K/(H \cap K)$. So $G/(H \cap K) \cong C_p \times C_p$, and it follows that it has $p+1$ subgroups of order $p$. This gives $p+1$ subgroups of $G$ of index $p$. By the above, each of these is abelian.
Now I need to show that there can't be more than $p+1$ such subgroups… but I'm a bit stuck here.
As for part (b), I thought I might be able to use (a) for this, if I can show that there does exist an abelian group of order 8 and it's cyclic. Somehow I need to use the fact that the group is of class 3. I try looking at the upper and lower central series but can't see a way of applying them. I would be grateful for any pointers in the right direction.
Best Answer
Note that in fact any abelian subgroup of index $p$ must contain the center: for if $H$ is maximal and abelian, and $Z(G)$ is not contained in $H$, then $G=HZ(G)$, which would make $G$ abelian. Therefore, if $H$ and $K$ are both abelian of index $p$, then $Z(G)\subseteq H\cap K$. If you have proven that $H\cap K\subseteq Z(G)$, then in fact we have that $H\cap K=Z(G)$ for any two distinct abelian subgroups of index $p$.
(And indeed: if $H\neq K$ are both abelian of index $p$, then for every $g\in G$ there exists $h\in H$, $k\in K$ with $g=hk$; given $x\in H\cap K$ we have $gx = (hk)x = x(hk) = xg$, since $x$ commutes with everything in $K$ and with everything in $H$, so $H\cap K\subseteq Z(G)$).
Conversely, if $M$ is a subgroup of $G$ that contains $Z(G)$ and is of index $p$, then it must be abelian (since $M/Z(G)$ is of order $p$). Thus, every abelian subgroup of $G$ of index $p$ corresponds to a subgroup of $G/Z(G)$, and you are done.
For part (b), show that (a) implies that if $G$ has more than one abelian subgroup of index $p$, then $G$ is of class exactly $2$: by the argument in (a), you know that $Z(G)$ has index $p^2$ in $G$, but that means that $G/Z(G)$ is abelian, which means that $[G,G]\subseteq Z(G)$.
So if $G$ is of class $3$ and order $16$, then it has at most one abelian subgroup of order $8$. The center must be of order $2$: if the center were of order $4$, then $G/Z(G)$ would be of order $4$, hence abelian, so again we have that $G$ would be of class $2$. Thus, $G/Z(G)$ is of order $8$, and cannot be abelian. Therefore, $G/Z(G)$ is one of the two nonabelian groups of order $8$. If $G/Z(G)$ were quaternion, then it would have four cyclic subgroups of order $4$: each of them pull back to a subgroup $H$ of $G$ of order $8$, containing the center; and it is not hard to show that they are all abelian, contradicting the fact that $G$ has at most one abelian subgroup of order $8$. So $G/Z(G)$ must be dihedral. Verify that you get a cyclic subgroup of order $8$ in $G$ in this case.
Added. So, let us suppose that $G$ is of class $3$ and order $16$, and has a unique abelian subgroup of index $2$. We know $Z(G)$ is of order $2$, and that $G/Z(G)$ is dihedral of order $8$. Let $z\in G$ generate $Z(G)$, and let $x\in G$ map to a generator of the cyclic group of order $4$ in $G/Z(G)$. We know that $x$ has order either $4$ or $8$ in $G$. We aim to show that it has order $8$.
If $x$ has order $8$, we are done. Otherwise, $|x|=4$. Let $y\in G$ be such that $y$ maps to the generators of $G/Z(G)$ of order $2$, so that $yx = x^3y$ or $yx=x^3yz$. In the former case, we have $yx^2 = x^6y = x^2y$, so $x^2$ commutes with $x$, $y$, and $z$; in the latter case we would have $yx^2 = x^3yxz = x^6yz^2 = x^2y$, so again $x^2$ commutes with $x$, $y$, and $z$. Since $x$, $y$, and $z$ generate $G$, it follows that $x^2\in Z(G)$. But then the image of $x$ in $G/Z(G)$ would have order $2$, which is a contradiction. Therefore, $x$ cannot have order $4$, and so must have order $8$, as desired.