According to Wigner’s theorem, every symmetry operation must be represented in quantum mechanics by a unitary or an anti-unitary operator. To see this, we can see that given any two states $|\psi\rangle$ and $|\psi'\rangle$, you would like to preserve
$$|\langle\psi|\psi\rangle'|^2=|\langle O\psi|O\psi \rangle'|^2$$
under some transformation $O$. If $O$ is unitary, that works. Yet, we can verify that an anti-unitary $A$ operator such that
$$\langle A\psi| A\psi'\rangle= \langle\psi|\psi'\rangle^* ,$$
works too; where ${}^*$ is the complex conjugate. Note that I cannot write it as $\color{red}{\langle \psi| A^\dagger A|\psi\rangle'}$ as $A$ does not behave as usual unitary operators, it is only defined on kets $|A\psi\rangle=A|\psi\rangle$ not bras.
Could one build a version of quantum mechanics using only anti-unitary operations?
Any anti-unitary operation is of the form $A=UK$, where $U$ is a unitary operator and $K$ is the complex conjugation operator (which is itself anti-unitary).
For a state $|\psi\rangle=\sum_\lambda c_\lambda |\lambda\rangle$ in some base $\{|\lambda\rangle\}$, then $K|\psi\rangle = \sum_\lambda c_\lambda^* |\lambda\rangle$.
A common example of an anti-unitary operator is the time-reversal operator.
Assumption
It seems to me that I can plug $K$ all over quantum mechanics to make any unitary transformation anti unitary and get back the same results, as it seems to preserve the actual probabilities that can be measured.
However, part of this question came from an earlier question in Quantum Computing Stack Exchange question of mine, where the answer showed that if anti-unitary operations existed, you could build faster-than-light messaging devices.
So what is wrong with my assumption? Is anti-unitary quantum mechanics problematic?
Best Answer
You can't make continuous transformations anti-unitary because the product of two anti-unitary transformations must be unitary. As a special case, you cannot make transformations that are continuously connected to unity anti-unitary because an identity operator is not anti-unitary.
Thus, the only transformations that can be represented by anti-unitary operators are discrete. And indeed, this does show up in quantum mechanics. For example, the time-reversal operation is represented by an anti-unitary matrix.