As someone who has an undergraduate education in mathematics, but didn't take it any further, I have often wondered something.
Of course mathematicians like to generalize ideas. i.e. it is often better to define and write proofs for a wider scope of objects than for a specific type of object. A kind of "paradox" if you will – the more general your ideas, often the deeper the proofs (quoting a professor).
Anyway I used to often wonder about group theory especially the idea of it being a set with a list of axioms and a binary function
$a\cdot b = c$.
But has anybody done research on tertiary (or is that trinary/ternary) groups? As in, the same definition of a group, but with a $\cdot (a,b,c) = d$ function.
Is there such a discipline? Perhaps it reduces to standard group theory or triviality and is provably of no interest. But since many results in Finite Groups are very difficult, notably the classification of simple groups, has anybody studied a way to generalize a group in such a way that a classification theorem becomes simpler?
As a trite example: Algebra was pretty tricky before the study of imaginary numbers.
Or to be even more trite: the Riemann $\zeta$ function wasn't doing much before it was extended to the whole complex plane.
EDIT:
Just to expand what I mean.
In standard groups there is an operation
$\cdot :G\times G\to G$
I am asking about the case with an operation
$\cdot :G\times G\times G\to G$
Best Answer
One problem with the idea is that the most obvious generalization to ternary operations really adds nothing new:
The proof is easy. Define $\otimes:G\times G\to G$ by $a\otimes b=[ab1_G]$. Then
$$\begin{align*} (a\otimes b)\otimes c&=[(a\otimes b)c1_G]=\big[[ab1_G]c1_G\big]=\big[ab[1_Gc1_G]\big]=[abc]\\ &=\big[[abc]1_G1_G\big]=\big[a[bc1_G]1_G\big]=[a(b\otimes c)1_G]=a\otimes(b\otimes c)\;, \end{align*}$$
and $a\otimes 1_G=[a1_G1_G]=a=[1_Ga1_G]=1_G\otimes a$ for all $a,b,c\in G$. Note that this does not require any kind of generalized commutativity for the ternary operation.
I remember noticing this as an undergraduate in the late 60s. My roommate was looking at a less obvious generalization of the associative law. The generalization, as I recall, was $$\big[[abc]de\big]=\big[a[bde][cde]\big]\;.\tag{1}$$ The idea is that if $\otimes$ is a binary operation on a set $G$, one can think of each element $a\in G$ as defining a function $f_a:G\to G:x\mapsto a\otimes x$, and associativity of $\otimes$ is then the statement that $$\operatorname{Comp}(f_a;f_b)=f_{f_a(b)}\;,\tag{2}$$ where $\operatorname{Comp}$ is the composition operator. In the ternary setting think of $a\in G$ as defining a function $$f_a:G\times G\to G:\langle b,c\rangle\mapsto [abc]\;;$$ then $(1)$ becomes
$$\operatorname{Comp}(f_a;f_b,f_c)=f_{f_a(b,c)}\;,$$
generalizing $(2)$.
If I remember correctly, this approach produced more interesting structures, but I no longer remember the details.