Here is part of Exercise 5.5.19 in Dummit & Foote's Abstract Algebra:
Let $H$ be a group of order $n$, let $K=\operatorname{Aut}(H)$ and $G=\operatorname{Hol}(H)=H\rtimes K$ (where $\varphi$ is the identity homomorphism). Let $G$ act by left multiplication on the left cosets of $K$ in $G$ and let $\pi$ be the associated permutation representation $\pi:G\to S_n$.
(a) Prove the elements of $H$ are coset representatives for the left cosets of $K$ in $G$ and with this choice of coset representatives $\pi$ restricted to $H$ is the regular representation of $H$.
(b) Prove $\pi(G)$ is the normalizer in $S_n$ of $\pi(H)$.
Deduce that under the regular representation of any finite group $H$ of order $n$, the normalizer in $S_n$ of the image of $H$ is isomorphic to $\operatorname{Hol}(H)$. [Show $|G|=|N_{S_n}(\pi(H))|$.]
I could easily show (a), but even before attempting to prove part (b), I was puzzled: Does it imply that $\pi$ is injective? The restriction of $\pi$ to $H$ is injective by part (a), but I don't think $\pi$ is injective in general.
For example, $D_8\simeq\operatorname{Aut}(D_8)$, so if $H=D_8$, then $K$ is a normal subgroup of $G$ (being of index 2), so $ker\pi=K\neq1$.
Am I wrong somewhere in my deduction?
Best Answer
In the case of $H=D_8$, it's not true $K$ is index $2$; actually $K$ is index $|H|$ in $G=H\rtimes K$. And $K$ is never normal in $G$, unless it's trivial of course.
The best way to think about $G$ is as an "affine group" of $H$. Indeed, if $H=\mathbb{Z}_p^n$ then $G$ is literally the affine group of $H$ as a vector space. In general, we can think of $G$ as the subset of $S_H$ comprised of "affine functions" of the form $x\mapsto \alpha(x)b$ where $\alpha\in\mathrm{Aut}(H)$ and $b\in H$.
Conjugating $K$ by $H$ yields functions of the form $\alpha(xb^{-1})b=\alpha(x)\alpha(b)^{-1}b$; to be an automorphism of $H$ (element of $K$) it must preserve $e\in H$ as a function, which requires $\alpha(b)=b$ (which, conversely, is sufficient), and this is only true for all $b$ if $\alpha$ is the identity automorphism. On the other hand, conjugating $H$ by $K$ yields functions $\alpha(\alpha^{-1}(x)b)=x\alpha(b)$, which are still elements of $H$, so $H$ is normal in $G$.
Note $H$ is a transversal for $G/K$, and indeed $G$ acts on $H$ representing $G/K$ matches $G$ acting on $H$ by affine functions in the way I described above. You want to show $N_{S_H}(H)=G$ here.