I would expect from a reflection operator $\hat{M}$ to leave a wavefunction unchanged if two times applied, thus $\hat{M}^2=1$. However, for a spin-1/2 particle this is not the case when following the standard definition of $\hat{M}$. Is there some scope to define the reflection operator in a different way? Where does the standard definition come from?
By standard definition I mean: The action of a reflection at the yz-plane (with normal in x-direction) $\hat{M}_x$ on the spin part is expressed via an inversion (which has no influence on the spin) and a 180° rotation around the x-axis. Rotations $\hat{R}_{\alpha}(\vec{n})$ of an angle $\alpha$ around the vector $\vec{n}$ can be expressed via:
\begin{equation}
\hat{R}_{\alpha}(\vec{n}) =exp\left(-i\frac{\alpha}{2}\vec{\sigma}\cdot\vec{n}\right) = \cos \left(\frac{\alpha}{2} \right) – i \vec{\sigma}\cdot\vec{n} \sin \left(\frac{\alpha}{2} \right)
\end{equation}
with the Pauli matrices $\vec{\sigma}$. For a 180° rotation around e.g. the $x$-axis we get
\begin{equation}
\hat{R}_{\pi}(\vec{n}_x) = – i \sigma_x = -i \begin{pmatrix}
0 & 1 \\
1 & 0 \\
\end{pmatrix}\, .
\end{equation}
Obviously, $\hat{M}_x^2=\hat{R}^2_{\pi}(\vec{n}_x)=-1$.
Best Answer
A 360° rotation of a spin 1/2 wavefunction does indeed produce a '-' sign. You can find more details in the chapter on angular momentum in Sakurai's Modern Quantum Mechanics.
Of course, this minus sign does not affect any observables, because we are calculating probabilities or expectation values, where the - signs on the bra and ket cancel each other out. However, it can be experimentally verified through interferometry experiments - We use a beam splitter on a monoenergetic beam of neutrons to create two paths. Introduce a phase change (=rotation of the ket, for example, using a magnetic field) in one path, and see whether the max/min interference condition is repeated for a phase change corresponding to a 360° rotation or 720°. Turns out the quantum mechanical prediction is right!