I'm trying to solve the problem, but it doesn't work. Please help me to solve it.
Problem: prove that $\bigl| \operatorname{Aut} \, (G) \bigr| \ge 8$ if $G$ is non-abelian and not isomorphic with $S_3$.
The beginning of the solution I have the following: given that a non-abelian group has a non-cyclic factor at the center, and therefore there cannot be a simple order, it is sufficient to prove the statement in the case when the factor at the center is isomorphic to $V_4$ or $S_3$. In the first case, it all comes down to a situation where the group is a 2-group. In the second case, using Hall's theorem on Hall subgroups, everything boils down to a situation where the group has order $2^a 3^b$. How to investigate these groups is not very clear.
Best Answer
As the poster observed, we just need to solve the problem for (finite) groups $G$ with $G/Z(G) \cong S_3$ or $V = C_2 \times C_2$.
Recall that a central automorphism $\alpha$ of a group $G$ is one that induces the identity on both $Z(G)$ and $G/Z(G)$. The central automorphisms form a (normal) subgroup of ${\rm Aut}(G)$ and, for all $g \in G$, we have $\alpha(g) = g \phi(g)$, where $\phi:G \to Z(G)$ is a homomorphism with $Z(G) \le \ker \phi$. Conversely, any such $\phi$ determines a central automorphism of $G$, so the group of central automorphisms is isomorphic to ${\rm Hom}(G/Z(G),Z(G))$.
$\textit{Case 1}$. $G/Z(G) \cong S_3$. If $|Z(G)|$ is even, then ${\rm Hom}(G/Z(G),Z(G))$ is nontrivial, so there are nontrivial central automorphisms. The inner automorphisms of $G$ are not central, so we have $|{\rm Aut}(G)| \ge 12$ and we are done.
So $|Z(G)|$ is odd, and $G$ has a normal abelian subgroup $N$ of odd order with $|G/N| = 2$ and $|N/Z(G)| = 3$. If $N$ is not a $3$-group and $P \in {\rm Syl}_p(N)$ with $p \ne 3$, then $P$ is a direct factor of $G$ and there are non-inner automorphisms of $G$ acting on $P$ and again we are done.
So $N$ is a $3$-group. Let $x,y$ be inverse images in $G$ of elements of generators of $G/Z(G)$, where we can choose $x$ to have order $2$ and $y \in N$. Then there are only three distinct commutators in $G$, the identity, $[x,y]$ and $[x,y^2]$, so $|[G,G]| = 3$. Since $[x,y] \in N \setminus Z(G)$, we have $[G,G] \cap Z(G) = 1$ and hence $N = [G,G] \times Z(G)$, and $G = \langle x,y \rangle \times Z(G)$. Again there are non-inner automorphisms of $G$ acting on $Z(G)$, and we are done.
$\textit{Case 2.}$ $G/Z(G) \cong C_2 \times C_2$. Let $x,y$ be inverse images in $G$ of generators of $G/Z(G)$. The only distinct commutators in $G$ are $1$ and $[x,y]$, so $|[G,G]|=2$, and $[G,G] \le Z(G)$. Hence $G$ is nilpotent, and we reduce as in Case 1 to the case when $G$ is a $2$-group.
If $Z(G)$ is non cyclic, then $|{\rm Hom}(G/Z(G),Z(G))| \ge 16$ and we are done, so $Z(G) = \langle z \rangle $ is cyclic.
If $z$ has order $2$ so $|G|=8$, then $G \cong D_8$ or $Q_8$, and $|{\rm Aut}(G)| = 8$ or $24$, and again we are done.
So $z$ has order $2^k$ with $k>1$. Note that $x^2$ and $y^2$ are powers of $z$. If either of them are even powers of $z$, then we can replace them by $xz^i$ and/or $yz^j$ for suitable $i,j$ to get elements of order $2$.
If we can do this for both $x$ and $y$, then $G$ is a central product of $D_8$ with $\langle z \rangle$, and there is a non-inner automorphism fixing $x$ and $y$ and mapping $z$ to $z^{-1}$, and we are done.
Otherwise, we have for example $x^2=z$. If $y^2$ is also an odd power of $z$, then $(xy)^2$ is an even power, so we can assume that $y$ has order $2$, and now $G$ is the group with the presentation $$\langle x,y \mid x^{2^{k+1}} = y^2=1, y^{-1}xy = y^{2^k+1} \rangle,$$ and $G$ has a non-inner automorphism with $x\mapsto x^{-1}$ and $y \mapsto y$.