If $G$ has $|G|$ many elements, then the set
$$\{a, a^2, a^3, \cdots, a^{|G|}\}$$
either has a repetition or exhaust the group. If it exhaust the group, one of these elements is $1$. If there is a repetition, so $a^{r} = a^{s}$ for $r>s$, then $a^{r-s} = 1$. In any event, at least one of these elements must equal $1$, say $a^n = 1$, and choose $n$ to be the smallest such exponent. If we can show that $n$ divides $|G|$, we are done. But $n$ is the order of the subgroup
$$\{a, a^2, \cdots , a^n(=1)\}$$
And by Lagrange's theorem, the order of a subgroup must divide the order of the group.
To rephrase here a bit: If $a$ is a generator of $G$, then raising it to the power of the order of the group guarantees that it will cycle through all the elements and return to the identity. If $a$ is not a generator of $G$ then Lagrange's theorem guarantees that the order of the subgroup generated by $a$ divides $|G|$ and therefore if $a^{|a|}=1$ then $(a^{|a|})^{u}=1$ where $|G|=u|a|$.
In response to your questions about definitions: A group is an abstract object. We do not know anything about the sorts of objects inside the group. All we know about the group is that it satisfies certain axioms and has a (binary) group operation $\ast$. Given a pair of elements $a,b \in G$, we write $a \ast b$ or $ab$ to denote the group operation acting on the pair of elements. The exponent notation means: $a^{2} = aa, a^{7} = aaaaaaa$.
A function $f$ is injective iff $f(a)=f(b)$ implies $a=b$, for every pair of elements $a$ and $b$. Now suppose $(f\circ g)(a)=(f\circ g)(b)$. By definition of $\circ$, you have $(f\circ g)(a)=(f\circ g)(b)$ implies $f(g(a))=f(g(b))$ which implies (use injectivity of $f$) that $g(a)=g(b)$ which implies (now use injectivity of $g$) that $a=b$. Since composition is associative for an arbitrary function, it is associative also for the subset of functions given by injective ones. For the identity, note that the identity function is trivially injective.
For the inverse, you need also surjectivity (an injection of a set in itself is not in general also surjective). You want to show that $f^{-1}$ is also one-to -one. So suppose $f^{-1}(a)=f^{-1}(b)$. Apply $f$ to both sides and deduce $a=b$. Here you use injectivity to guarantee that $f^{-1}$ is defined.
Best Answer
For the general question of $(\{a,b\},*)$ being a group, the answer is no: consider the operation $\max$ on the set $\{0,1\}$. It induces a monoid structure - associative and $0$ is the neutral element - but $1$ has no inverse.
Moreover, any set $S$ with a constant map $c:S\times S\to S$ is certainly a semigroup.
There is a last semigroup operation: $$a*b=b$$ This is neither abelian nor a monoid, and it completes, together with $(\Bbb Z/2\Bbb Z,+)$, $\max$, and the constant, the list of the possible isomorphism classes of semigroups of two elements.