Let $G$ be a finite group. Under which conditions on $G$ is the automorphism group $\text{Aut }G$ cyclic? More precisely, does "$G$ is abelian" or "$G$ is cyclic" imply "$\text{Aut }G$ is cyclic"?
Group Theory – When is the Automorphism Group Aut(G) Cyclic?
abelian-groupsautomorphism-groupcyclic-groupsfinite-groupsgroup-theory
Related Solutions
If the automorphism group is cyclic, then the Inner automorphism group is cyclic. But the inner automorphism group is isomorphic to $G/Z(G)$, and if $G/Z(G)$ is cyclic, then it is trivial. Therefore, $G=Z(G)$ so $G$ is abelian. (The argument does not require $G$ to be finite, by the by.)
For the latter:
Prop. If $H\subseteq Z(G)$ and $G/H$ is cyclic, then $G$ is abelian.
Suppose $G/H$ is cyclic, with $H\subseteq Z(G)$. Let $g\in G$ be such that $gH$ generates $G/H$. Then every $x\in G$ can be written as $x=g^kh$ for some integer $k$ and some $h\in H$. Given $x,y\in G$, we have $x=g^kh$ and $y=g^{\ell}h'$, so $$\begin{align*} xy &= (g^kh)(g^{\ell}h')\\ &= g^kg^{\ell}hh' &\quad&\text{since }h\in Z(G)\\ &= g^{\ell}g^kh'h\\ &= g^{\ell}h'g^kh &&\text{since }h'\in Z(G)\\ &= (g^{\ell}h')(g^kh)\\ &= yx, \end{align*}$$ hence $G$ is abelian. QED
For more on what groups can occur as central quotients, see this previous question.
Added. Since you mention you did not know that $\mathrm{Inn}(G)\cong G/Z(G)$, let's do that too:
Define a map $G\to \mathrm{Aut}(G)$ by mapping $g\mapsto \varphi_g$, where $\varphi_g$ is "conjugation by $g$". That is, for all $x\in G$, $\varphi_g(x) = gxg^{-1}$.
This map is a group homomorphism: if $g,h\in G$, then we want to show that $\varphi_{gh} = \varphi_g\circ\varphi_h$. To that end, let $x\in G$ be any element, and we show that $\varphi_{gh}(x) = \varphi_g(\varphi_h(x))$. $$\varphi_{gh}(x) = (gh)x(gh)^{-1} = ghxh^{-1}g^{-1}= g(hxh^{-1})g^{-1} = \varphi_g(hxh^{-1}) = \varphi_g(\varphi_h(x)).$$
Therefore, the map $g\mapsto\varphi_g$ is a homomorphism from $G$ onto $\mathrm{Inn}(G)$. By the Isomorphism Theorem, $\mathrm{Inn}(G)$ is isomorphic to $G/N$, where $N$ is the kernel of this homomorphism.
What is $N$? $g\in N$ if and only if $\varphi_g$ is the identity element of $\mathrm{Aut}(G)$, which is the identity; that is, if and only if $\varphi_g(x)=x$ for all $x\in G$. But $\varphi_g(x)=x$ if and only if $gxg^{-1}=x$, if and only if $gx = xg$. So $\varphi_g(x) = x$ if and only if $g$ commutes with $x$. Thus, $\varphi_g(x)=x$ for all $x$ if and only if $g$ commutes with all $x$, if and only if $g\in Z(G)$. Thus, $N=Z(G)$, so $\mathrm{Inn}(G)\cong G/Z(G)$, as claimed.
$\newcommand{\Hom}[0]{\mathrm{Hom}}$$\newcommand{\Aut}[0]{\mathrm{Aut}}$$\newcommand{\Inn}[0]{\mathrm{Inn}}$As per your other question, the groups of this paper provide concrete examples of finite $p$-groups $G$ of nilpotence class $2$ such that $\Aut(G)$ is isomorphic to the additive group $\Hom(G/G', Z(G))$, and thus $\Aut(G)$ is abelian.
Note that if $G$ is a group such that $\Aut(G)$ is abelian, then so is $G/Z(G)$, as it is isomorphic to the subgroup $\Inn(G) \le \Aut(G)$ of the inner automorphisms of $G$. Thus $G$ is nilpotent, of nilpotence class at most $2$.
Best Answer
In fact, $\operatorname{Aut}(G)$ is cyclic if and only if $G$ is cyclic of order $n$ where $n$ is either $4$ or of the form $2^mp^k$ where $m$ is $0$ or $1$ and $p$ is an odd prime.
That $\operatorname{Aut}(G)$ is cyclic in these cases follows since it is will then be isomorphic to the multiplicative group of the integers mod $n$, which is cyclic in precisely the above mentioned cases.
If on the other hand $\operatorname{Aut}(G)$ is cyclic, then clearly $G$ is abelian, as $G/Z(G)$ is otherwise a non-cyclic subgroup.
But if $G = H\times K$ then $\operatorname{Aut}(H)\times \operatorname{Aut}(K)$ is naturally a subgroup of $\operatorname{Aut}(G)$ so if this is to be cyclic, then the orders of these must be coprime, and hence the order of $G$ is indeed as described above (or possible a larger power of $2$). The same argument also shows that the $p$-Sylow must be cyclic, so the only case left is that of some larger power of $2$. But now once again the same argument shows that $G$ would have to be a direct product of some number of copies of $C_2$, and we know that these groups do not even have abelian automorphism group.