If the automorphism group is cyclic, then the Inner automorphism group is cyclic. But the inner automorphism group is isomorphic to $G/Z(G)$, and if $G/Z(G)$ is cyclic, then it is trivial. Therefore, $G=Z(G)$ so $G$ is abelian. (The argument does not require $G$ to be finite, by the by.)
For the latter:
Prop. If $H\subseteq Z(G)$ and $G/H$ is cyclic, then $G$ is abelian.
Suppose $G/H$ is cyclic, with $H\subseteq Z(G)$. Let $g\in G$ be such that $gH$ generates $G/H$. Then every $x\in G$ can be written as $x=g^kh$ for some integer $k$ and some $h\in H$. Given $x,y\in G$, we have $x=g^kh$ and $y=g^{\ell}h'$, so
$$\begin{align*}
xy &= (g^kh)(g^{\ell}h')\\
&= g^kg^{\ell}hh' &\quad&\text{since }h\in Z(G)\\
&= g^{\ell}g^kh'h\\
&= g^{\ell}h'g^kh &&\text{since }h'\in Z(G)\\
&= (g^{\ell}h')(g^kh)\\
&= yx,
\end{align*}$$
hence $G$ is abelian. QED
For more on what groups can occur as central quotients, see this previous question.
Added. Since you mention you did not know that $\mathrm{Inn}(G)\cong G/Z(G)$, let's do that too:
Define a map $G\to \mathrm{Aut}(G)$ by mapping $g\mapsto \varphi_g$, where $\varphi_g$ is "conjugation by $g$". That is, for all $x\in G$,
$\varphi_g(x) = gxg^{-1}$.
This map is a group homomorphism: if $g,h\in G$, then we want to show that $\varphi_{gh} = \varphi_g\circ\varphi_h$. To that end, let $x\in G$ be any element, and we show that $\varphi_{gh}(x) = \varphi_g(\varphi_h(x))$.
$$\varphi_{gh}(x) = (gh)x(gh)^{-1} = ghxh^{-1}g^{-1}= g(hxh^{-1})g^{-1} = \varphi_g(hxh^{-1}) = \varphi_g(\varphi_h(x)).$$
Therefore, the map $g\mapsto\varphi_g$ is a homomorphism from $G$ onto $\mathrm{Inn}(G)$. By the Isomorphism Theorem, $\mathrm{Inn}(G)$ is isomorphic to $G/N$, where $N$ is the kernel of this homomorphism.
What is $N$? $g\in N$ if and only if $\varphi_g$ is the identity element of $\mathrm{Aut}(G)$, which is the identity; that is, if and only if $\varphi_g(x)=x$ for all $x\in G$. But $\varphi_g(x)=x$ if and only if $gxg^{-1}=x$, if and only if $gx = xg$. So $\varphi_g(x) = x$ if and only if $g$ commutes with $x$. Thus, $\varphi_g(x)=x$ for all $x$ if and only if $g$ commutes with all $x$, if and only if $g\in Z(G)$. Thus, $N=Z(G)$, so $\mathrm{Inn}(G)\cong G/Z(G)$, as claimed.
$\DeclareMathOperator{\Aut}{Aut}$
We'll show the slightly stronger result that $|\Aut(G)| \not= 2n+1$ for any $n \ge 1$ EDIT: Provided $G$ is abelian: see the comments.
If $\Aut(G)$ is trivial, then there is nothing to show. Suppose, then, that $G$ has a nontrivial automorphism group. We will show that $|\Aut(G)|$ is even by explicitly exhibiting an automorphism $\phi$ of $G$ having order $2$ for any group $G$. It will then follow by Lagrange's theorem that $\Aut(G)$ has even order.
If $G$ has an element of order at least $3$, then $\phi:x \to x^{-1}$ is the desired automorphism of even order. It only remains to consider the case where $G$ contains only elements of order less than or equal to $2$. In that case, we must have that $G$ is a vector space over $\mathbb{Z}_2$ (see Group where every element is order 2 for a proof of this result). If $G$ is one dimensional as a vector space, then $\Aut(G) = \Aut(\mathbb{Z}_2)$ is trivial, so there is nothing more to show. For $G$ having dimension greater than $1$, we have that $\Aut(G) \cong \operatorname{GL}(G)$, so if we pick an ordered basis for $G$, then the automorphism which interchanges the first and second basis vectors (and fixes all others) is an order $2$ automorphism. (This argument, which Slade gave in the comments, applies even to infinite $G$, while my previous argument tacitly assumed $G$ was finite.)
Best Answer
The contrapositive is much clearer:
The key facts are
$\text{Inn}(G) \cong G/Z(G)$
All groups of order less than $4$ are cyclic
If $G/Z(G)$ is cyclic then $G$ is abelian