For direct product it seems easy.
Let $G=N\times K$ and assume that both $N$ and $K$ are characteristic in $G$. It is easy to show $Aut(G) \cong Aut(N)\times Aut(K)$ since both $N$ and $K$ are characteristic in G. Since $G/N \cong K$ then $Aut(G/N) \cong Aut(K)$. Thus there is a natural epimorphism $\phi:Aut(G) \to Aut(G/N)$ with $ker(\phi) \cong Aut(K)$.
Now you can ask when are they both characteristic in $G$? Actually, one simple condition provide this: Let $N$ and $K$ be finite groups with relatively prime orders, and set $G=N\times K$. Then both $N$ and $K$ are characteristic in $G$.
And you offer an example $G=C_2\times C_3$ and $N=C_2$ then set $K=C_3$ since order of N and K are relatively prime, the result is immediate from above construction.
Edit, 9/25/20: The suggestion I made at the end works.
Proposition: Let $G$ be a group of order $n$ (which may be infinite). Then $\text{Inn}(G)$ is precisely the kernel of the action of $\text{Aut}(G)$ acting on the set $\text{Hom}_{\text{HGrp}}(F_n, G)$ of (simultaneous) conjugacy classes of $n$-tuples of elements of $G$.
Proof. Suppose $\varphi \in \text{Aut}(G)$ acts trivially. Consider its action on the $n$-tuple given by every element of $G$. Fixing this $n$-tuple means fixing it up to conjugacy, which means there is some $g \in G$ such that $\varphi(h) = ghg^{-1}$ for all $h \in G$, which says precisely that $\varphi \in \text{Inn}(G)$. On the other hand, every element of $\text{Inn}(G)$ clearly acts trivially. $\Box$
Of course we can do much better than considering every element of $G$; it suffices to consider a generating set. But this construction is at least "canonical."
Here's an approach that maybe will seem like it doesn't tell you anything new but I'll extract something slightly more concrete out of it, which generalizes the suggestion to look at conjugacy classes. $\text{Out}(G)$ occurs naturally as the automorphism group of $G$ in a category we might call the homotopy category of groups $\text{HGrp}$. This category can be defined concretely as follows:
- objects are groups $G$, and
- morphisms $f : G \to H$ are conjugacy classes of homomorphisms, where two homomorphisms $f_1, f_2 : G \to H$ are identified (homotopic) iff there exists $h \in H$ such that $h f_1 = f_2 h$.
For example:
- $\text{Hom}_{\text{HGrp}}(\mathbb{Z}, G)$ is the set of conjugacy classes of $G$
- $\text{Hom}_{\text{HGrp}}(G, S_n)$ is the set of isomorphism classes of actions of $G$ on a set of size $n$
- $\text{Hom}_{\text{HGrp}}(G, GL_n(\mathbb{F}_q))$ is the set of isomorphism classes of actions of $G$ on $\mathbb{F}_q^n$
and so forth.
Now we can prove the more general fact that composition in this category is well-defined (that is, that the homotopy class of a composition of morphisms only depends on the homotopy class of each morphism), which implies in particular that the automorphism group $\text{Aut}_{\text{HGrp}}(G)$ of $G$ in this category is really a group, and of course this group is $\text{Out}(G)$.
So far this is just a slight extension and repackaging of the proof via conjugating by an inner automorphism, but the point is that this construction tells you what conjugating by an inner automorphism means. The homotopy category of groups has a second description, as follows:
- objects are Eilenberg-MacLane spaces $K(G, 1) \cong BG$, and
- morphisms $f : BG \to BH$ are homotopy classes of homotopy equivalences.
We get the ordinary category of groups if we instead insist that Eilenberg-MacLane spaces have basepoints and our morphisms and homotopies preserve basepoints. So the passing to conjugacy classes has to do with the extra freedom we get from throwing out basepoints. Here the incarnation of conjugacy classes $\text{Hom}(\mathbb{Z}, G)$ is the set of free homotopy classes of loops $S^1 \to BG$.
Anyway, all this suggests the following generalization of looking at conjugacy classes: we can look at the entire representable functor
$$\text{Hom}_{\text{HGrp}}(-, G) : \text{HGrp}^{op} \to \text{Set}.$$
By the Yoneda lemma, the automorphism group of this functor is precisely $\text{Aut}_{\text{HGrp}}(G) \cong \text{Out}(G)$. What this says is that an outer automorphism of $G$ is the same thing as a choice, for each group $H$, of an automorphism (of sets) of $\text{Hom}_{\text{HGrp}}(H, G)$, which is natural in $H$. We can furthermore hope that it's possible to restrict attention to a smaller collection of groups $H$; for example (and I haven't thought about this at all) maybe it's possible to restrict to the free groups $H = F_n$, which means looking at $\text{Hom}_{\text{HGrp}}(F_n, G)$, the set of conjugacy classes of $n$ elements of $G$ (under simultaneous conjugacy).
Best Answer
Here are two very well known infinite families of examples:
The cyclic group of order p (p an odd prime) is not the automorphism group of any group (such a group is abelian since its inner automorphism group is cyclic, but an abelian group either has inversion as an automorphism of order 2, or is an elementary abelian 2-group, and an elementary abelian 2-group either has trivial automorphism group or has a coordinate swap as an order 2 automorphism). However, the cyclic group of order p is a characteristic Sylow p-subgroup of AGL(1,p), which is the automorphism group of the dihedral group of order 2p and of itself. AGL(1,p) is the normalizer of a Sylow p-subgroup of the symmetric group on p-points.
The alternating group of degree n (n ≥ 9) is a characteristic, index 2 subgroup of its automorphism group, but is not itself the automorphism group of any group by Robinson (1982, MR678545).
At least as far as I understand it, automorphism groups of groups tend to be big and "full", and so it should not be surprising that many of their subgroups are not themselves automorphism groups of groups since they are "missing" something. For instance, a simple group cannot be an automorphism group unless it is complete; M12 is incomplete. Odd order cyclic groups do not work, since one is missing inversion (and indeed, the rest of AGL1).