I came across this problem stated in the title having no clue what to do, and got stuck even in the finite case. Here's my attempt:
I first proved that the group $G$, if it exists, cannot be finite abelian. If it were, we could factor $G$ into cyclic groups:
$$G=C_{2^{e_1}}\times \cdots \times C_{2^{e_k}} \times C_{p_1^{s_1}} \times \cdots \times C_{p_m^{s_m}}$$
where all $p_i$'s are odd primes (not necessarily distinct), $k\ge 0, m\ge 0 $, and all $e_i$'s and $s_i$'s are positive integers. Since $\operatorname{Aut}(G_1) \times \cdots \times \operatorname{Aut}(G_n)$ is a subgroup of $\operatorname{Aut}(G_1 \times \cdots \times G_n)$, we have
$$\left. \left(\prod _{i=1}^k2^{e_i-1}\cdot \prod_{i=1}^m p_i^{s_i-1}(p_i-1)\right) \right| \left|\operatorname{Aut}(G)\right|$$
Note that $2|(p_i-1)$ for all $i$, but $v_2(\left|\operatorname{Aut}(G)\right|)=v_2(10)=1$, so either $k=1, m=0$ or $k=0, m=1$. A simple argument shows that neither case satisfies the problem.
It's known that if $G$ is nonabelian, then $G/Z(G)\cong \operatorname{Inn}(G)$ cannot be cyclic. Assuming $G$ is finite, we get $\operatorname{Inn}(G)$ is a non-cyclic normal subgroup of $\operatorname{Aut}(G)\cong D_5$. But a non-cyclic normal subgroup of $D_5$ must be $D_5$ itself, so $\operatorname{Inn}(G)=\operatorname{Aut}(G)$, and every automorphism of $G$ is inner. Then I failed to construct an outer automorphism of $G$ to obtain a contradiction (thought an outer automorphism of $D_5$ would do the work).
I've searched groups of order less than $255$, and none of them has automorphism group $D_5$, so I believe the answer is no for finite groups. Could anyone please give a hint on how to proceed? Other approaches are also welcomed.
(Of course it is best to solve the general problem, without the finiteness hypothesis!)
Best Answer
There is no such group.
First, as already laid out in the comments, $G$ cannot be abelian. For completeness sake, here is the argument again: If $G$ is abelian and not an $\mathbb{F}_2$-vector space, the inversion map is central of order $2$, which cannot happen. On the other hand, if $G$ is an $\mathbb{F}_2$-vector space, it either has too few automorphisms (if $\mathrm{dim}\leq 2$) or too many (otherwise).
As a consequence, we conclude that $\mathrm{Inn}(G)=G/Z(G)$ must already be all of $D_5$ (since it cannot be cyclic). Following a comment from Derek Holt, we can then already conclude that $Z:=Z(G)$ cannot have any elements of order $2$: Indeed, since $G/Z$ has a subgroup of index $2$, this would give rise to a nontrivial homomorphism $\phi:G/Z \to Z$, which in turn induces the (central) automorphism $$\psi:G\to G, g\mapsto g\phi(gZ).$$ Now, it is known (or in any case easy to check) that $\psi$ must centralize all inner automorphisms. In our case, this is impossible, since $D_5$ has a trivial center.
Now, I claim that we may therefore express $G$ as a semidirect product $C_5\rtimes H$ for some abelian group $H$ with at most one element of order $2$. To see this, we start by picking any element $x$ that generates the $C_5$ in the quotient $G/Z$. Likewise, let $s$ be any element that maps to a reflection in $D_5$. These two elements must then satisfy $sxs^{-1}=zx^{-1}$ for some central $z\in Z$. Now, the element $y=x^2z^{-1}$ also generates the $C_5$ in $G/Z$ but now satisfies $sys^{-1}=y^{-1}$. As a consequence, the subgroup $\langle y\rangle$ is normal in $G$. Moreover, since $y^5$ is central, we must have $$y^5=s y^5 s^{-1}=y^{-5},$$ so that $y^{10}=1$. Since $Z$ has no $2$-torsion, we must have $y^5=1$ already so that $y$ generates a normal $C_5$ in $G$. Taking $H=\langle Z,s\rangle$, we find our decomposition $G=C_5\rtimes H$ (as the two subgroups clearly have trivial intersection). Since the center $Z$ is of index $2$ in $H$, the assertion about elements of order $2$ also follows immediately. Notationally, we can therefore write elements of $G$ conveniently as pairs $(n,h)$ with $n\in \mathbb{Z}/5\mathbb{Z}$ and $h\in H$, where two pairs multiply as $$(n,h)\cdot(n',h')=(n+\sigma(h)n',hh').$$ Here, $\sigma:H\to \{\pm 1\}$ is the unique nontrivial homomorphism with kernel $Z$ (and we continue to write $H$ multiplicatively).
Now, it is not hard to check that the map $$\psi:G\to G, (n,h)\mapsto (-n,h^{-1})$$ defines an automorphism of $G$ (note that this is not the inversion map, which does not give an automorphism!).
Is this automorphism inner? Well, using the multiplication formula above, we can immediately see that every inner automorphism must induce the identity on $G/C_5$. Therefore, the map I just gave can only be inner if every element of $H$ has order $2$. But, as we already observed above, $H$ has at most one such element! Hence, the only remaining possibility is that $H=C_2$, which leads to $G=D_5$. For this single remaining option, we can then calculate that $\mathrm{Aut}(G)=C_5\rtimes C_4$ to (finally!) squash any remaining possibility for such a group to exist.