Could you give me a simple example of $G$ abelian with ${\rm Aut}(G)$ non-abelian?
Otherwise how could I prove that $G$ abelian implies ${\rm Aut}(G)$ abelian. (I don't really think that's true)
An abelian group $G$ with ${\rm Aut}(G)$ non-abelian
abelian-groupsautomorphism-groupgroup-theory
Related Solutions
If $G$ is abelian and some $g\in G$ has $g^2\not = 1$, then $\operatorname{Aut}(G)$ contains the element $x \to x^{-1}$ of order $2\!\!\not|\,p^5$. The remaining case $G = \mathbb{Z}_2^{\oplus n}$ and $\operatorname{Aut}(G) = GL_n(\mathbb{Z}_2)$ can be handled directly. (That's assuming $G$ is finite, given the tags and the proof of the theorem.)
As the poster observed, we just need to solve the problem for (finite) groups $G$ with $G/Z(G) \cong S_3$ or $V = C_2 \times C_2$.
Recall that a central automorphism $\alpha$ of a group $G$ is one that induces the identity on both $Z(G)$ and $G/Z(G)$. The central automorphisms form a (normal) subgroup of ${\rm Aut}(G)$ and, for all $g \in G$, we have $\alpha(g) = g \phi(g)$, where $\phi:G \to Z(G)$ is a homomorphism with $Z(G) \le \ker \phi$. Conversely, any such $\phi$ determines a central automorphism of $G$, so the group of central automorphisms is isomorphic to ${\rm Hom}(G/Z(G),Z(G))$.
$\textit{Case 1}$. $G/Z(G) \cong S_3$. If $|Z(G)|$ is even, then ${\rm Hom}(G/Z(G),Z(G))$ is nontrivial, so there are nontrivial central automorphisms. The inner automorphisms of $G$ are not central, so we have $|{\rm Aut}(G)| \ge 12$ and we are done.
So $|Z(G)|$ is odd, and $G$ has a normal abelian subgroup $N$ of odd order with $|G/N| = 2$ and $|N/Z(G)| = 3$. If $N$ is not a $3$-group and $P \in {\rm Syl}_p(N)$ with $p \ne 3$, then $P$ is a direct factor of $G$ and there are non-inner automorphisms of $G$ acting on $P$ and again we are done.
So $N$ is a $3$-group. Let $x,y$ be inverse images in $G$ of elements of generators of $G/Z(G)$, where we can choose $x$ to have order $2$ and $y \in N$. Then there are only three distinct commutators in $G$, the identity, $[x,y]$ and $[x,y^2]$, so $|[G,G]| = 3$. Since $[x,y] \in N \setminus Z(G)$, we have $[G,G] \cap Z(G) = 1$ and hence $N = [G,G] \times Z(G)$, and $G = \langle x,y \rangle \times Z(G)$. Again there are non-inner automorphisms of $G$ acting on $Z(G)$, and we are done.
$\textit{Case 2.}$ $G/Z(G) \cong C_2 \times C_2$. Let $x,y$ be inverse images in $G$ of generators of $G/Z(G)$. The only distinct commutators in $G$ are $1$ and $[x,y]$, so $|[G,G]|=2$, and $[G,G] \le Z(G)$. Hence $G$ is nilpotent, and we reduce as in Case 1 to the case when $G$ is a $2$-group.
If $Z(G)$ is non cyclic, then $|{\rm Hom}(G/Z(G),Z(G))| \ge 16$ and we are done, so $Z(G) = \langle z \rangle $ is cyclic.
If $z$ has order $2$ so $|G|=8$, then $G \cong D_8$ or $Q_8$, and $|{\rm Aut}(G)| = 8$ or $24$, and again we are done.
So $z$ has order $2^k$ with $k>1$. Note that $x^2$ and $y^2$ are powers of $z$. If either of them are even powers of $z$, then we can replace them by $xz^i$ and/or $yz^j$ for suitable $i,j$ to get elements of order $2$.
If we can do this for both $x$ and $y$, then $G$ is a central product of $D_8$ with $\langle z \rangle$, and there is a non-inner automorphism fixing $x$ and $y$ and mapping $z$ to $z^{-1}$, and we are done.
Otherwise, we have for example $x^2=z$. If $y^2$ is also an odd power of $z$, then $(xy)^2$ is an even power, so we can assume that $y$ has order $2$, and now $G$ is the group with the presentation $$\langle x,y \mid x^{2^{k+1}} = y^2=1, y^{-1}xy = y^{2^k+1} \rangle,$$ and $G$ has a non-inner automorphism with $x\mapsto x^{-1}$ and $y \mapsto y$.
Best Answer
Take $G=\mathbb{Z}^2$ its automorphism group $Gl(2,\mathbb{Z})$ is not commutative.