Let $G$ be a group and let $\mathrm{Aut}(G)$ be the set of all automorphisms of $G$. Prove that $\mathrm{Aut}(G)$ is a group under the operation of composition for functions.
I am not too sure if I am proving $\mathrm{Aut}(G)$ itself is a group. Would this mean then that I would have to make sure it fulfills all the group requirements? If so I am not too sure how to go about it. And to prove closure, I would think I need to pick say $f,g \in \mathrm{Aut}(G)$ then use the fact that these are automorphisms. From here, I am not too sure how to proceed.
Best Answer
Let $G$ be a group. By definition $$\text{Aut}(G)=\left\{f:G\rightarrow G\mid f \mbox{ is an isomorphism of }G\right\}.$$ Given two automorphisms $f,g\in \text{Aut}(G)$, we can consider the composition $g\circ f$. We claim that $\text{Aut}(G),\circ$ is a group. We have to check all axioms.
First of all we need to show that $g\circ f$ is again an automorphism, i.e. a homomorphism that is bijective. Now since $g$ and $f$ are bijective, $g\circ f$ is bijective. Moreover,
$$\begin{align} (g\circ f)(ab)&=g(f(ab))\\ &=g(f(a)f(b))\\ &=g(f(a))g(f(b))\\ &=(g\circ f)(a)(g\circ f)(b), \end{align}$$
for all $a,b\in G$. Hence $g\circ f$ is a group homomorphism.
Secondly we need to show that $\circ$ is associative, i.e. $(h\circ g)\circ f=h\circ (g\circ f)$. Just evaluate both morphisms at $a\in G$ and see that both expressions coincide due to the associativity of $G$.
Thirdly we need to check that there is a neutral element for $\circ$. Clearly $Id_G:G\rightarrow G:a\mapsto a$ is an automorphism. Since $f\circ Id_G=Id_G\circ f$ for all $f\in \text{Aut}(G)$, $Id_G$ is the neutral element.
Last but not least, we have to check that each $f\in \text{Aut}(G)$ has an inverse for $\circ$. Consider the inverse function $f^{-1}$. Clearly $f^{-1}\circ f=Id_G=f\circ f^{-1}$. So it remains to show that $f^{-1}$ is a group morphism. Now it's a very good exercise to prove this last statement.
EDIT: Let's prove the last statement. Suppose that $f:G\rightarrow G$ is a group isomorphism. We need to show that $f^{-1}$ is a group morphism. Let $a,b\in G$. By definition there exist a unique $x,y\in G$ such that $f(x)=a$ and $f(y)=b$. Hence
$$\begin{align} f^{-1}(ab)&=f^{-1}(f(x)f(y))\\ &=f^{-1}(f(xy))\\ &=xy. \end{align}$$ Similarly
$$\begin{align} f^{-1}(a)f^{-1}(b)&=f^{-1}(f(x))f^{-1}(f(y))\\ &=xy. \end{align}$$
Hence $f^{-1}(ab)=f^{-1}(a)f^{-1}(b).$