Let $G$ be a group, show that $\text{Aut}(G)$ acts on the set $\mathscr C$ of the conjugacy classes of $G$ with the action given by $\varphi(\mathscr C_x)=\mathscr C_{\varphi(x)}$ with $\varphi\in\text{Aut}(G)$ and $\mathscr C_x$ the class of $x$.
Also we have to show that $\mathscr F(G)=\{\varphi\in\text{Aut}(G)\mid \varphi(\mathscr C_x)=\mathscr C_x\}$ is normal in $\text{Aut}(G)$ and that $\mathscr F(G)$ contains the inner automorphisms given by conjugation.
__
This was an assignment for my group theory course. I've done this course, rep theory and rings and modules.
The problem I'm having is that I don't know how to tackle this exercise with the things we've been taught on the group theory course, because we skimmed over the automorphisms (and their actions) part, and focused more on cyclic and Sylow groups and that kind of stuff.
I've tried seeing that the difference of two elements in the same coclass is sent into that class itself and to expand the product of the action, but can't go further than that
Best Answer
As the action is on a set of classes (necessarily dealt with by means of representatives), we have first to prove that the map is well defined, namely that $x'\in \mathscr C_x \Rightarrow \varphi\cdot\mathscr C_{x'}=\varphi\cdot\mathscr C_{x}$. This in indeed the case, because:
\begin{alignat}{1} \mathscr C_{\varphi(x')} &= \{g\varphi(x')g^{-1}, g\in G\} \\ &= \{g\varphi(g'xg'^{-1})g^{-1}, g\in G\} \\ &= \{g\varphi(g')\varphi(x)\varphi(g')^{-1}g^{-1}, g\in G\} \\ &= \{(g\varphi(g'))\varphi(x)(g\varphi(g'))^{-1}, g\in G\} \\ &= \{g''\varphi(x)g''^{-1}, g''\in G\} \\ &= \mathscr C_{\varphi(x)} \\ \end{alignat}
where the last but one equality follows from $g\mapsto g\varphi(g')$ being onto $G$, for every $g'\in G$. Next, $\varphi\cdot\mathscr C_x\in \mathscr C$ by definition. Furthermore, $Id_G\cdot \mathscr C_x=\mathscr C_{Id_G(x)}=\mathscr C_x$. Finally, $(\varphi\psi)\cdot\mathscr C_x=\mathscr C_{(\varphi\psi)(x)}=\mathscr C_{\varphi(\psi(x))}=\varphi\cdot(\psi\cdot\mathscr C_x)$. So, this is indeed a group action and $\mathscr F(G)=\operatorname{Stab}(\mathscr C_x)$ is a subgroup of $\operatorname{Aut}(G)$. Then:
\begin{alignat}{1} \psi\operatorname{Stab}(\mathscr C_x)\psi^{-1} &= \{\psi\varphi\psi^{-1}\mid \varphi\cdot\mathscr C_x=\mathscr C_x\} \\ &= \{\rho\mid (\psi^{-1}\rho\psi)\cdot\mathscr C_x=\mathscr C_x\} \\ &= \{\rho\mid \rho\cdot (\psi\cdot\mathscr C_x)=\psi\cdot\mathscr C_x\} \\ &= \operatorname{Stab}(\psi\cdot\mathscr C_x) \end{alignat}
so $\mathscr F(G)$ is not normal in $\operatorname{Aut}(G)$. Rather, here we get that the stabilizers of the points of one same orbit are all conjugate to each other. Last, if $\varphi\in\operatorname{Inn}(G)$, then $\exists g\in G$ such that $\varphi\cdot\mathscr C_x=\mathscr C_{\varphi(x)}=\mathscr C_{gxg^{-1}}=\mathscr C_x$, whence $\varphi\in\mathscr F(G)$. Therefore, $\operatorname{Inn}(G)\le\mathscr F(G)$. (Note that this holds for every conjugacy class considered in $\mathscr F(G)$, and hence $\operatorname{Inn}(G)\le\operatorname{ker}\phi$, where $\phi$ is the homomorphism $\operatorname{Aut}(G)\to S_\mathscr C$ equivalent to your action.)
Comment. If the action is defined by $\varphi\cdot\mathscr C_x:=\mathscr C_{\varphi(x)}$, then the proof of the good definition focuses on the invariance w.r.t. representative's choice, being the membership of $\varphi\cdot\mathscr C_x$ to $\mathscr C$ trivial (by definition); this is the above approach. Conversely, if the action is defined by $\varphi\cdot\mathscr C_x:=\varphi(\mathscr C_x)$, then the proof of the good definition focuses on the membership of $\varphi\cdot\mathscr C_x$ to $\mathscr C$, being the invariance w.r.t. to representative's choice trivial (as $x'\in\mathscr C_x\Rightarrow \mathscr C_{x'}=\mathscr C_x$). The effort in both approach is the same, and equal to proving that $\varphi(\mathscr C_x)=\mathscr C_{\varphi(x)}$.