A group in which $ba\in\{ab,(ab)^{-1}\}$ for all $a,b$ (call this (Y)) satisfies the weaker property (X): $ab$ commutes with $ba$ for all $a,b$. Changing variable $b$ to $c=ab$, one sees that (X) means that all elements commute with their conjugates; this property has already been consider here at MathSE.
Indeed, (X) implies locally nilpotent: indeed if $G$ satisfies (X) and is generated by $a_1,\dots,a_n$ and if $H_i$ is generated by the conjugacy class of $a_i$, then $H_i$ is an abelian normal subgroup and $G=H_1\dots H_n$. Now use that a product of finitely many nilpotent normal subgroups is abelian.
Now consider the stronger property (Y), with the same change of variables, means that any two conjugate elements are either equal or inverse.
So for non-commuting elements $a,b$, we have $aba^{-1}=b^{-1}$. In particular, $a$ commutes with $b^2$. Hence, if $Z$ is the center of $G$, we have $g^2\in Z$ for all $G$. If $G$ is non-abelian, $G/Z$ has exponent 2 and hence is abelian; in particular $G$ is 2-step nilpotent.
For $G$ satisfying (Y), if $a,b$ don't commute, as already noticed, we have $a^2b^2=1$. But this also applies to $a$ and $b^{-1}$, whence $a^4=b^4=1$. Now let $W$ be the subset of $G$ of elements $g$ of order dividing $4$ (i.e;, $g^4=1$). I claim that this is a subgroup: indeed, for $a,b\in G$, we have $(ab)^2=a^2(a^{-1}bab^{-1})b^2$. This is a product of 3 central elements (since squares and commutators are central), and hence $(ab)^4=a^4b^4$ for all $a,b\in G$. In particular, if $a,b$ are in $W$ then $(ab)^4=1$ (I checked that $g\mapsto g^4$ is an endomorphism if $G$ satisfies (Y).)
So if $G$ satisfies (Y) then $W$ is a subgroup and $W\cup Z=G$ Hence either $W$ or $Z$ equals $G$. In other words, if $G$ is a non-abelian group in (Y), then $W=G$, i.e., $G$ has exponent 4.
Continue with non-abelian $G$ satisfying (Y). I claim that for all non-central elements $a,b$ we have $a^2=b^2$ (hence we obtain a distinguished element of order 2).
Indeed, choose $c,d$ such that $ac\neq ca$ and $bd\neq db$. Since $a,c$ don't commute and $G$ has exponent 4, we have $a^2=c^2$ ($a^2c^2=1$ rewritten); similarly $b^2=d^2$. If any pair among $(a,b),(a,d),(c,b),(c,d)$ is non-commutative, we deduce they have the same squares and deduce $a^2=b^2$. So these four pairs are commuting. But then $a$ does not commute with $bc$, so $a^2=(bc)^2=b^2c^2=a^2c^2$ (the middle equality because $b,c$ commute), so $c^2=1$, and then contradiction, since all elements squaring to 1 are central (recall that every noncentral element is only conjugate to itself and its inverse).
So every non-abelian group $G$ in (Y) has a subgroup $M$ of order $2$ such that every noncentral element has its square the nontrivial element $z$ of $M$. In particular, $G/M$ has exponent 2, so is abelian, and hence $M$ is the derived subgroup of $G$. Moreover, the non-central elements are precisely those of order 4: indeed if by contradiction $g,h$ have order 4 with $g$ non-central and $h$ central, then $gh$ is non-central, so $z=(gh)^2=g^2h^2=zh^2\neq z$, contradiction.
Now we have a converse: let $G$ be a non-abelian group of exponent 4, with a normal subgroup $M=\{1,z\}$ of order 2 such that $G/M$ is abelian and $g^2=z$ for every non-central $g$.
Indeed, for $g,h$ not commuting, $ghg^{-1}h^{-1}=z=h^{-2}$, so $ghg^{-1}=h^{-1}$, which is the required condition.
Best Answer
It's possible to make sense of this question as follows. Let $G$ be a group together with a distinguished central element of order $2$ which we call $-1$; we'll write the product $(-1)a$ as $-a$. Say that two elements $a, b \in G$ anticommute if $ab = -ba$. Then we have, more or less by definition:
So the desired groups are precisely the central extensions of abelian groups by $C_2$. These groups are 2-step nilpotent, and in the finite case every such group must have the form $G = G_1 \times G_2$ where $G_1$ is an abelian group of odd order and $G_2$ is a central extension of an abelian $2$-group by $C_2$.
The two smallest nonabelian examples of such groups are the quaternion group $Q_8$ and the dihedral group $D_8$, which are central extensions of $C_2 \times C_2$ by $C_2$. See the Wikipedia article on extraspecial groups for more. $D_8$ is also isomorphic to the Heisenberg group $H_3(\mathbb{F}_2)$.
The group algebra $k[G]$ of such a group has a distinguished quotient where we identify $-1 \in G$ with the actual element $-1 \in k[G]$ (apologies for the abuse of notation); this is a certain twisted group algebra of $G / \{ 1, -1 \}$. Applying this construction to the quaternion group $Q_8$ over $k = \mathbb{R}$ produces the quaternion algebra $\mathbb{H}$.