Does there exist a finite group $G$ and a normal subgroup $H$ of $G$ such that $\lvert\operatorname{Aut}{(H)}\rvert>\lvert\operatorname{Aut}{(G)}\rvert$?
Group Theory – Existence of Normal Subgroup with Larger Automorphism Group
finite-groupsgroup-theory
Related Solutions
Even without the classification of finite simple groups, quite reasonable bounds are known, for example in work of P.M. Neumann. If the group $G$ can be generated by $r$ but no fewer elements, then no automorphism of $G$ can fix the $r$ given generators, so there are at most $\prod_{j=1}^{r} (|G|-j)$ different automorphisms of $G,$ since the $r$ generators must have distinct images, none of which is the identity. As P.M. Neumann has observed, $G$ can always by generated by ${\rm log}_{2}(|G|)$ or fewer elements, so we have $r \leq \lfloor {\rm log}_{2}(|G|) \rfloor .$ For $|G| >4,$ this always gives a strictly better bound for the size of ${\rm Aut}(G)$ than $(|G|-1)!.$ For large $|G|,$ it is much better. Using the classification of finite simple groups, much better bounds are known.
Later edit: Perhaps I could outline Neumann's argument, since it is quite elementary, and I do not remember a reference: Let $\{x_1, x_2, \ldots, x_r \}$ be a minimal generating set for $G$ and let $G_i = \langle x_1, x_2, \ldots, x_i \rangle $ for $i >0,$ $G_{0} = \{ e \}.$ Then for $1 \leq i \leq r,$ we have $|G_i| > |G_{i-1}|$ by minimality of the generating set. Furthermore, $|G_i|$ is divisible by $|G_{i-1}|$ by Lagrange's theorem, so $|G_i| \geq 2|G_{i-1}|.$ Hence $|G| = |G_r| \geq 2^r.$
Hint: Using the in-and-out principle (aka "inclusion and exclusion"), you can compute the number of injective homomorphisms from a finite group $H$ to a finite group $G$ from the values of $\lvert\operatorname{Hom}(H/N,G)\rvert$ for every normal subgroup $N$ of $H$. Therefore, from your assumption that $\lvert\operatorname{Hom}(H,G_1)\rvert=\lvert\operatorname{Hom}(H,G_2)\rvert$ for every finite group $H$, it follows that every finite $H$ has the same number of injective homomorphisms to $G_1$ as to $G_2$, and in particular, that there is at least one injective homomorphism from $G_1$ to $G_2$ and vice versa, whence it easily follows that $G_1\cong G_2$. This argument applies to lots of other finite structures besides groups. I believe that it's due to László Lovász and that he discovered it as a high school student.
P.S. This has come up several times on Math Stack Exchange and Math Overflow, for example in this answer, which cites L. Lovász, Operations with structures, Acta Math. Acad. Sci. Hungar. 18 (1967) 321-328.
P.P.S. I will try to present the argument using the in-and-out principle. (I still think this argument must be due to Lovász, even it it's not the one he gives in the work cited above. Unless the argument is wrong, in which case it must be due to me.)
Let $H$ and $G$ be finite groups. Let $H\setminus\{1_H\}=\{h_1,\dots,h_n\}$.
For $i\in[n]=\{1,\dots,n\}$, let $S_i=\{\varphi\in\text{Hom}(H,G):\varphi(h_i)=1_G\}$.
For $I\subseteq[n]$, let $S_I=\bigcap_{i\in I}S_i=\{\varphi\in\text{Hom}(H,G):\forall i\in I\ \varphi(h_i)=1_G\}$ if $I\ne\emptyset$, and $S_\emptyset=\text{Hom}(H,G)$.
Then $|S_I|=|\text{Hom}(H/N_I,G)|$ where $N_I$ is the normal subgroup of $H$ generated by $\{h_i:i\in I\}$. By the in-and-out principle,$$|\text{Mono}(H,G)|=|\text{Hom}(H,G)\setminus\bigcup_{i\in[n]}S_i|=\sum_{I\subseteq[n]}(-1)^{|I|}|S_I|=\sum_{I\subseteq[n]}(-1)^{|I|}|\text{Hom}(H/N_I,G)|.$$Therefore, if $H,G_1,G_2$ are finite groups, and if $|\text{Hom}(H/N,G_1)|=|\text{Hom}(H/N,G_2)|$ for every normal subgroup $N$ of $H$, then $|\text{Mono}(H,G_1)|=|\text{Mono}(H,G_2)|$.
P.P.P.S. This is not true for infinite groups. If $G_1$ and $G_2$ are nonisomorphic groups such that each is isomorphic to a subgroup of the other, then $|\text{Hom}(H,G_1)|=|\text{Hom}(H,G_2)|$ for every (finite or infinite) group $H$, although $G_1\not\cong G_2$.
Best Answer
A version of Robin Chapman's answer that might be easier to verify:
The additive group $H$ of a field of order $2^{n}$ has two nice types of automorphism: multiplication by a scalar, and the Frobenius automorphism. The semi-direct product $G$ is called $AΓL(1,2^{n})$. It has order $(2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n}$, and for $n ≥ 2$ it is its own automorphism group.
It has the additive group $H$ of the field as a normal subgroup. However, as just an additive group, $H$ has automorphism group all n×n matrices over the field with 2 elements, which has size $(2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) ≥ 2^{(n−1)^{2}}$.
For large $n$, $|Aut(G)| = (2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n} ≤ 2^{(n−1)^{2}} ≤ (2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) = |Aut(H)|$
In particular:
For $n = 4$, $G = AΓL(1,16)$, $Aut(G) = G$ has order $960$, and $Aut(H) = GL(4,2)$ has order $20160$.
For $n = 10$, $G = AΓL(1,1024)$, $Aut(G) = G$ has order $10475520$, and $Aut(H) = GL(10,2)$ has order $366440137299948128422802227200$.
In other words, $Aut(H)$ can be enormously bigger than $Aut(G)$.
This is reasonably important in finite group theory:
$G$ is a very rigid group with lots of structure. Because it contains all the "automorphisms" of the field, the group itself determines the field. An automorphism of the group will have to be an automorphism of the field, and we've already listed them all. Sometimes this expressed by saying the group G determines the geometry of the affine line on which it acts.
$H$ is a very floppy group to which you can do nearly anything. Without the maps encoding scalar multiplication, $H$ no longer remembers the field that defined it. It is just a vector space, and so instead of the (very few) field automorphisms, you are now allowed to use any vector space automorphism. $H$ has lost its structure.
I think $H$ is the canonical example of a horribly structureless group. Groups like $GL$ and $AΓL$ are pretty standard examples of groups with very clear structure. The symmetric group is very similar. Except for a few early cases, a symmetric group is its own automorphism group because it already contains within itself the set of points on which it is acting.