The question states:
Let $G$ be an Abelian group with subgroup $H < G$. Let $S=\{x~|~x\in G$ and $x^2 \in H\}$. Show that $S$ is a subgroup of $G$.
My proof is different than what is in the book, I proceeded as follows:
Let the identity be $e$.
$e^2 = e$, therefore $e^2 \in H$, and hence $e \in S$. Therefore $S$ is non-empty.
By definition, $S\subseteq G$.
Let $a$, $b$ $\in S$. Therefore, $a^2, b^2 \in H$ and $a^2, b^2 \in G$.
$\therefore ab^{-1} \in S \iff {(ab^{-1})}^2 \in H$,
$\therefore ab^{-1} \in S \iff a^2(b^2)^{-1} \in H$.
As $H$ is a group, each element has a unique inverse, so $a^2, b^2 \in H \implies b^2 \in H$.
Also, as $H$ is a group, it is closed, and $ \therefore a^2(b^2)^{-1} \in H$. Hence, $ab^{-1} \in S$.
So, S is a non-empty subset of a group G, and $ab^{-1} \in S$ for all $a, b \in S$.
Thus, by the subgroup test, $S$ is a subgroup of $G$.
I never used the fact that G is Abelian though… which is why I am worried my solution may be incorrect.
Best Answer
To reiterate what was said in the comments by @ArturoMagidin, you used the fact that $G$ is abelian in this step:
Thusly:
$$\begin{align} (ab^{-1})^2&=(ab^{-1})(ab^{-1}) &\\ &=a(b^{-1}a)b^{-1}& \\ &=a(ab^{-1})b^{-1} & \text{ (since } G\text{ is abelian),}\\ &=a^2b^{-2}& \\ &=a^2(b^2)^{-1}. & \end{align}$$
Your proof is correct.
The test you use is the "one-step subgroup test", despite there being two steps (because the nonemptiness is ignored for some reason).
Well done!