Best Answer: Let Aut(G) be the set of all automorphisms φ: G --> G. In order to show that this is a group under the operation of composition, we must verify:
(1) Is the set is closed under composition? Yes! If you are given isomorphisms φ, ψ: G --> G, then it is not too tough to show that ψ∘φ and φ∘ψ are isomorphisms. I can expand on this in more detail if you like, but you have probably seen a proof before that a composition of bijective functions is bijective. If a and b are elements of the group, ψ∘φ(ab) = ψ(φ(ab)) = ψ(φ(a)φ(b)), because φ is an isomorphism. Since ψ is also an isomorphism, ψ(φ(a)φ(b)) = ψ∘φ(a)ψ∘φ(b), so the composition ψ∘φ preserves products. Thus, ψ∘φ is an isomorphism if ψ and φ are.
(2) Is the set associative? Yes! All you need to do is show that, for any three isomorphisms φ, ψ and ξ, φ∘(ψ∘ξ) = (φ∘ψ)∘ξ. To do that, just show that for each x in G, φ∘(ψ∘ξ)(x) = (φ∘ψ)∘ξ(x) = φ(ψ(ξ(x))). It's just pushing around definitions.
(3) Does the set contain an identity element? Yes! Let the identity automorphism e: G --> G be the map e(x) = x. Clearly, e∘φ = φ∘e = φ.
(4) Does each element of the set have an inverse under ∘? Yes! Since each isomorphism φ: G --> G is bijective, there is a well-defined inverse map φ^(-1): G --> G. You may have already seen a proof that the inverse of an isomorphism is an isomorphism. If not, it isn't too difficult to prove: I'll leave it to you, but I can expand on it if you need me to. Further, the composition φ^(-1) ∘ φ = φ ∘ φ^(-1) = e.
Since Aut(G) satisfies all the group axioms, it forms a group under ∘, as needed.
In the beginning, we must show that mapping $p_a: G \to G: p_a(x) = a^{-1}xa$ exists. Then we will show, that it is injective and surjective. If so, then it is isomorphism and automorphism.
Because $(G,\cdot)$ is a group, for $a \in G $ and $x \in G$: $a^{-1}xa \in G$. Because of that, we can safely define $p_a: G \to G: p_a(x) = a^{-1}xa$, knowing that we begin and end in $(G,\cdot)$.
Let's assume that $p_a$ isn't injective. Then $\exists x_1,x_2 \in G, x_1 \ne x_2, $ such as $p_a(x_1) = p_a(x_2)$. Then $a^{-1}x_1a = a^{-1}x_2a$. $(G,\cdot)$ is a group, so we can "multiply" this expression by $a$ from the left side and by $a^{-1}$ from the right side: $aa^{-1}x_1aa^{-1} = aa^{-1}x_2aa^{-1} \Rightarrow x_1=x_2$, but $x_1 \ne x_2$. Therefore $p_a$ must be injective.
Let's assume that $p_a$ isn't surjective. Then $\exists y \in G: \forall x \in G:f(x) \ne y$. Then $a^{-1}xa \ne y$. We can transform it into $\exists y \in G: \forall x \in G: a^{-1}ya \ne x $. But that cannot be, because that means that result of $a^{-1}ya$ isn't in $(G,\cdot)$. Therefore $p_a$ must be surjective.
Now, for the second part, we should prove two things: that $Inn(G)$ is a subgroup of $Aut(G)$, and that it is a normal subgroup.
We must show that neutral element of $Aut(G)$ is in $Inn(G)$. We don't know how he looks like, but it isn't hard to guess that $id$ is a neutral element of $Aut(G)$. $id(x) = x$, so we must find $p_a$ with same properties. It isn't hard, too: $p_1(x) = 1 \cdot x \cdot 1 = x$. So $id = p_1, p_1 \in Inn(G)$.
Then, we must show that $\forall p_a,p_b \in Inn(G)$, $p_b \circ p_a \in Inn(G)$. $(p_b \circ p_a)(x) = p_b(p_a(x)) = p_b(a^{-1}xa) = b^{-1}a^{-1}xab = (ab)^{-1}x(ab) = p_{ab}(x) \in Inn(g)$.
Third part is existence of inverse element to $p_a$ in $Inn(G)$. This element would be $p_{a^{-1}}$: $(p_{a^{-1}} \circ p_a)(x) = (aa^{-1})^{-1}x(aa^{-1}) = 1 \cdot x \cdot 1 = x$.
Finally, we must show that $Inn(G)$ is a normal subgroup: for $f,f^{-1} \in Aut(G), p_a \in Inn(G):$ $(f^{-1} \circ p_a \circ f)(x) = f^{-1}(p_a(f(x))) = f^{-1}(a^{-1} \cdot f(x) \cdot a) = f^{-1}(a^{-1}) \cdot f(f^{-1}(x)) \cdot f^{-1}(a) = f^{-1}(a^{-1})\cdot x\cdot (f^{-1}(a^{-1}))^{-1}$. $f(x)$ is in $G$, so are $f^{-1}(a^{-1})$ and $(f^{-1}(a^{-1}))^{-1}$, and $f^{-1}(a^{-1})\cdot x\cdot (f^{-1}(a^{-1}))^{-1}$ defines another $p_b \in Inn(G)$. The proof is complete.
Best Answer
That is not the statement you want to be proving. Recall that a function $p:G\rightarrow G$ is a homomorphism when $$p(xy)=p(x)p(y)\text{ for all }x,y\in G.$$ You want to prove that, given two homomorphisms $f:G\rightarrow G$ and $g:G\rightarrow G$, then their composition $(f\circ g):G\rightarrow G$ is also a homomorphism. The composition $f\circ g$ is defined by $$(f\circ g)(x)=f(g(x))\text{ for all }x\in G.$$ So, do you see what the right statement you want to prove is? Also, remember that in your argument, somewhere you should use the hypotheses: namely, that $f$ and $g$ are themselves homomorphisms.