It is not that associativity is required for groups... That is quite backwards: the truth is actually that groups are associative.
Your question seems to come from the idea that people decided how to define groups and then began to study them and find them interesting. In reality, it happened the other way around: people had studied groups way before actually someone gave a definition. When a definition was agreed upon, people looked at the groups they had at hand and saw that they happened to be associative (and that that was a useful piece of information about them when working with them) so that got included in the definition.
If I may say so, it is this which is important to understand. The way we teach abstract algebra nowdays somewhat obscures this fact, but this is how essentially everything comes to be.
Best Answer: Let Aut(G) be the set of all automorphisms φ: G --> G. In order to show that this is a group under the operation of composition, we must verify:
(1) Is the set is closed under composition? Yes! If you are given isomorphisms φ, ψ: G --> G, then it is not too tough to show that ψ∘φ and φ∘ψ are isomorphisms. I can expand on this in more detail if you like, but you have probably seen a proof before that a composition of bijective functions is bijective. If a and b are elements of the group, ψ∘φ(ab) = ψ(φ(ab)) = ψ(φ(a)φ(b)), because φ is an isomorphism. Since ψ is also an isomorphism, ψ(φ(a)φ(b)) = ψ∘φ(a)ψ∘φ(b), so the composition ψ∘φ preserves products. Thus, ψ∘φ is an isomorphism if ψ and φ are.
(2) Is the set associative? Yes! All you need to do is show that, for any three isomorphisms φ, ψ and ξ, φ∘(ψ∘ξ) = (φ∘ψ)∘ξ. To do that, just show that for each x in G, φ∘(ψ∘ξ)(x) = (φ∘ψ)∘ξ(x) = φ(ψ(ξ(x))). It's just pushing around definitions.
(3) Does the set contain an identity element? Yes! Let the identity automorphism e: G --> G be the map e(x) = x. Clearly, e∘φ = φ∘e = φ.
(4) Does each element of the set have an inverse under ∘? Yes! Since each isomorphism φ: G --> G is bijective, there is a well-defined inverse map φ^(-1): G --> G. You may have already seen a proof that the inverse of an isomorphism is an isomorphism. If not, it isn't too difficult to prove: I'll leave it to you, but I can expand on it if you need me to. Further, the composition φ^(-1) ∘ φ = φ ∘ φ^(-1) = e.
Since Aut(G) satisfies all the group axioms, it forms a group under ∘, as needed.
Best Answer
Things may be clarified if you adjust your parentheses a bit. Suppose $x$ is an element of the domain of $h$. On one hand, we have
$$((f \circ g) \circ h)(x) = (f \circ g)(h(x))= f(g(h(x))),$$
and on the other hand,
$$(f \circ (g \circ h))(x) = f((g \circ h)(x)) = f(g(h(x))).$$