Like Qiaochu says, the key here is that the direct product of finitely many abelian groups functions as both the (categorical) product and the (categorical) coproduct in the category of abelian groups.
And before your eyes glaze over, what that means is that:
A homomorphism from an abelian group $A$ into a (finite) direct product $G_1\times G_2\times\cdots\times G_n$ of abelian groups is equivalent to a family $f_1,\ldots,f_n$ of homomorphisms $f_i\colon A\to G_i$; (in fact, this holds for arbitrary groups, and arbitrarily many direct factors, not just finitely many); and
A homomorphism from a finite direct product $G_1\times G_2\times\cdots\times G_n$ of abelian groups into an abelian group $B$ is equivalent to a family $g_1,\ldots,g_n$ of homomorphisms $g_i\colon G_i\to B$ (here we do need both finiteness and abelianness).
The first equivalence is easy: given a map $f\colon A\to G_1\times\cdots\times G_n$, the maps $f_i$ are just the compositions of $f$ with the canonical projections $\pi_i\colon G_1\times\cdots\times G_n\to G_i$; going the other way, given a family $f_1,\ldots,f_n$ of maps, you get the map $A\to G_1\times\cdots\times G_n$ by $f(a) = (f_1(a),f_2(a),\ldots,f_n(a))$.
For the second equivalence, given a homomorphism $g\colon G_1\times\cdots\times G_n\to B$, we define the maps $g_i\colon G_i\to B$ by restricting $g$ to the subgroup $\{0\}\times\cdots \times \{0\}\times B_i\times\{0\}\times\cdots\times\{0\}$. Conversely, given a family of homomorphisms $g_1,\ldots,g_n$, we construct the map $g$ by $g(x_1,\ldots,x_n) = g_1(x_1)+g_2(x_2)+\cdots+g_n(x_n)$; here, both the fact that the product has only finitely many factors and that the groups are abelian is important.
Now let $A=B=G_1\times\cdots\times G_n$. Then a homomorphism from $A$ to $B$ is equivalent, by 1, to a family of homomorphisms $\Phi_j\colon A\to G_j$. And by 2, each $\Phi_j$ is equivalent to a family of homomorphisms $\phi_{ij}\colon G_i\to G_j$. Thus, each homomorphism from $A$ to $B$ is equivalent to a family $\{\phi_{ij}\mid 1\leq i,j\leq n\}$, with $\phi_{ij}\colon G_i\to G_j$.
Now suppose you have two homomorphisms, $\Phi,\Psi\colon A\to B$, and you want to compose them. If $\Phi$ corresponds to $\{\phi_{ij}\}$ and $\Psi$ corresponds to $\{\psi_{ij}\}$, what does the composition correspond to in terms of maps $G_i\to G_j$?
If you trace the correspondence carefully, you should find that the induced map from $G_i$ to $G_j$ is precisely
$$\psi_{i1}\circ\phi_{1j} + \psi_{i2}\circ\phi_{2j}+\cdots+\psi_{in}\circ\phi_{nj},$$
so that if you arrange the families $\{\phi_{ij}\}$ and $\{\psi_{ij}\}$ into matrices, composition corresponds to matrix multiplication in the usual way (though because composition is not commutative, you have to be mindful of the order of the products.
Once you have that endomorphisms can be "coded" as matrices with composition corresponding to matrix multiplication, the fact that automorphisms correspond to invertible matrices follows immediately. However, actually writing down a formula is complicated, because these matrices have entries that don't commute with one another; even in simple cases, like trying to do something like $C_{p^{\alpha}}\times C_{p^{\beta}}$ with $\alpha\gt\beta$, writing down the inverse of an automorphism in terms of its entries turns into a computation with congruences that is difficult to write down as a formula. But never fret, you aren't asked for an explicit formula.
Best Answer: Let Aut(G) be the set of all automorphisms φ: G --> G. In order to show that this is a group under the operation of composition, we must verify:
(1) Is the set is closed under composition? Yes! If you are given isomorphisms φ, ψ: G --> G, then it is not too tough to show that ψ∘φ and φ∘ψ are isomorphisms. I can expand on this in more detail if you like, but you have probably seen a proof before that a composition of bijective functions is bijective. If a and b are elements of the group, ψ∘φ(ab) = ψ(φ(ab)) = ψ(φ(a)φ(b)), because φ is an isomorphism. Since ψ is also an isomorphism, ψ(φ(a)φ(b)) = ψ∘φ(a)ψ∘φ(b), so the composition ψ∘φ preserves products. Thus, ψ∘φ is an isomorphism if ψ and φ are.
(2) Is the set associative? Yes! All you need to do is show that, for any three isomorphisms φ, ψ and ξ, φ∘(ψ∘ξ) = (φ∘ψ)∘ξ. To do that, just show that for each x in G, φ∘(ψ∘ξ)(x) = (φ∘ψ)∘ξ(x) = φ(ψ(ξ(x))). It's just pushing around definitions.
(3) Does the set contain an identity element? Yes! Let the identity automorphism e: G --> G be the map e(x) = x. Clearly, e∘φ = φ∘e = φ.
(4) Does each element of the set have an inverse under ∘? Yes! Since each isomorphism φ: G --> G is bijective, there is a well-defined inverse map φ^(-1): G --> G. You may have already seen a proof that the inverse of an isomorphism is an isomorphism. If not, it isn't too difficult to prove: I'll leave it to you, but I can expand on it if you need me to. Further, the composition φ^(-1) ∘ φ = φ ∘ φ^(-1) = e.
Since Aut(G) satisfies all the group axioms, it forms a group under ∘, as needed.
Best Answer
Yes, and in fact every group is the automorphism group of some commutative monoid; see Hagen von Eitzen's answer here. His construction proceeds by showing first that every group is the automorphism group of some graph, then constructing a commutative monoid out of the graph with the same automorphism group; it's quite a nice construction.
The first sentence of the abstract of this paper claims that every group is the automorphism group of a lattice, which is more or less an idempotent commutative monoid (depending on the precise definitions).