This question is related to the question here, which is about generalising the anticommutative property for abelian groups to all groups. The definition there was unclear, so one interpretation put forward was:
Two elements "twisted-commute" if $ab=(ba)^{-1}$.
We therefore have the question:
Is there a specific name for a group where any two elements in the group either commute or "twisted-commute" with one another? That is, for $a,b$ in $G$ either $ab=ba$ or $ab=(ba)^{-1}$.
Best Answer
A group in which $ba\in\{ab,(ab)^{-1}\}$ for all $a,b$ (call this (Y)) satisfies the weaker property (X): $ab$ commutes with $ba$ for all $a,b$. Changing variable $b$ to $c=ab$, one sees that (X) means that all elements commute with their conjugates; this property has already been consider here at MathSE.
Indeed, (X) implies locally nilpotent: indeed if $G$ satisfies (X) and is generated by $a_1,\dots,a_n$ and if $H_i$ is generated by the conjugacy class of $a_i$, then $H_i$ is an abelian normal subgroup and $G=H_1\dots H_n$. Now use that a product of finitely many nilpotent normal subgroups is abelian.
Now consider the stronger property (Y), with the same change of variables, means that any two conjugate elements are either equal or inverse.
So for non-commuting elements $a,b$, we have $aba^{-1}=b^{-1}$. In particular, $a$ commutes with $b^2$. Hence, if $Z$ is the center of $G$, we have $g^2\in Z$ for all $G$. If $G$ is non-abelian, $G/Z$ has exponent 2 and hence is abelian; in particular $G$ is 2-step nilpotent.
For $G$ satisfying (Y), if $a,b$ don't commute, as already noticed, we have $a^2b^2=1$. But this also applies to $a$ and $b^{-1}$, whence $a^4=b^4=1$. Now let $W$ be the subset of $G$ of elements $g$ of order dividing $4$ (i.e;, $g^4=1$). I claim that this is a subgroup: indeed, for $a,b\in G$, we have $(ab)^2=a^2(a^{-1}bab^{-1})b^2$. This is a product of 3 central elements (since squares and commutators are central), and hence $(ab)^4=a^4b^4$ for all $a,b\in G$. In particular, if $a,b$ are in $W$ then $(ab)^4=1$ (I checked that $g\mapsto g^4$ is an endomorphism if $G$ satisfies (Y).)
So if $G$ satisfies (Y) then $W$ is a subgroup and $W\cup Z=G$ Hence either $W$ or $Z$ equals $G$. In other words, if $G$ is a non-abelian group in (Y), then $W=G$, i.e., $G$ has exponent 4.
Continue with non-abelian $G$ satisfying (Y). I claim that for all non-central elements $a,b$ we have $a^2=b^2$ (hence we obtain a distinguished element of order 2).
Indeed, choose $c,d$ such that $ac\neq ca$ and $bd\neq db$. Since $a,c$ don't commute and $G$ has exponent 4, we have $a^2=c^2$ ($a^2c^2=1$ rewritten); similarly $b^2=d^2$. If any pair among $(a,b),(a,d),(c,b),(c,d)$ is non-commutative, we deduce they have the same squares and deduce $a^2=b^2$. So these four pairs are commuting. But then $a$ does not commute with $bc$, so $a^2=(bc)^2=b^2c^2=a^2c^2$ (the middle equality because $b,c$ commute), so $c^2=1$, and then contradiction, since all elements squaring to 1 are central (recall that every noncentral element is only conjugate to itself and its inverse).
So every non-abelian group $G$ in (Y) has a subgroup $M$ of order $2$ such that every noncentral element has its square the nontrivial element $z$ of $M$. In particular, $G/M$ has exponent 2, so is abelian, and hence $M$ is the derived subgroup of $G$. Moreover, the non-central elements are precisely those of order 4: indeed if by contradiction $g,h$ have order 4 with $g$ non-central and $h$ central, then $gh$ is non-central, so $z=(gh)^2=g^2h^2=zh^2\neq z$, contradiction.
Now we have a converse: let $G$ be a non-abelian group of exponent 4, with a normal subgroup $M=\{1,z\}$ of order 2 such that $G/M$ is abelian and $g^2=z$ for every non-central $g$.
Indeed, for $g,h$ not commuting, $ghg^{-1}h^{-1}=z=h^{-2}$, so $ghg^{-1}=h^{-1}$, which is the required condition.