Consider a quantum system described by the Hilbert space $\mathcal{H}$ and consider $A,B\in \mathcal{L}(\mathcal{H},\mathcal{H})$ to be observables. If those observables do not commute there's no simultaneous basis of eigenvectors of each of them. In that case, in general if $|\varphi\rangle$ is eigenvector of $A$ it will not be of $B$.
This leads to the problem of not having a definite value of some quantity in some states.
Now, this is just a mathematical model. It works because it agrees with observations. But it makes me wonder about something. Concerning the Physical quantities associated to $A$ and $B$ (if an example helps consider $A$ to be the position and $B$ the momentum) what is really behind non-commutativity?
Do we have any idea whatsoever about why two observables do not commute? Is there any idea about any underlying reason for that?
Again I know one might say "we don't care about that because the theory agrees with the observation", but I can't really believe there's no underlying reason for some physical quantities be compatible while others are not.
I believe this comes down to the fact that a measurement of a quantity affects the system in some way that interferes with the other quantity, but I don't know how to elaborate on this.
EDIT: I think it's useful to emphasize that I'm not saying that "I can't accept that there exist observables which don't commute". This would enter the rather lengthy discussion about whether nature is deterministic or not, which is not what I'm trying to get here.
My point is: suppose $A_1,A_2,B_1,B_2$ are observables and suppose that $A_1$ and $B_1$ commute while $A_2$ and $B_2$ do not commute. My whole question is: do we know today why the physical quantities $A_1$ and $B_1$ are compatible (can be simultaneously known) and why the quantities $A_2$ and $B_2$ are not?
In other words: accepting that there are incompatible observables, and given a pair of incompatible observables do we know currently, or at least have a guess about why those physical quantities are incompatible?
Best Answer
Observables don't commute if they can't be simultaneously diagonalized, i.e. if they don't share an eigenvector basis. If you look at this condition the right way, the resulting uncertainty principle becomes very intuitive.
As an example, consider the two-dimensional Hilbert space describing the polarization of a photon moving along the $z$ axis. Its polarization is a vector in the $xy$ plane.
Let $A$ be the operator that determines whether a photon is polarized along the $x$ axis or the $y$ axis, assigning a value of 0 to the former option and 1 to the latter. You can measure $A$ using a simple polarizing filter, and its matrix elements are $$A = \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}.$$
Now let $B$ be the operator that determines whether a photon is $+$ polarized (i.e. polarized southwest/northeast) or $-$ polarized (polarized southeast/northwest), assigning them values 0 and 1, respectively. Then $$B = \begin{pmatrix} 1/2 & -1/2 \\ -1/2 & 1/2 \end{pmatrix}.$$
The operators $A$ and $B$ don't commute, so they can't be simultaneously diagonalized and thus obey an uncertainty principle. And you can immediately see why from geometry: $A$ and $B$ are picking out different sets of directions. If you had a definite value of $A$, you have to be either $x$ or $y$ polarized. If you had a definite value of $B$, you'd have to be $+$ or $-$ polarized. It's impossible to be both at once.
Or, if you rephrase things in terms of compass directions, the questions "are you going north or east" and "are you going northeast or southeast" do not have simultaneously well-defined answers. This doesn't mean compasses are incorrect, or incomplete, or that observing a compass 'interferes with orientation'. They're just different directions.
Position and momentum are exactly the same way. A position eigenstate is sharply localized, while a momentum eigenstate has infinite spatial extent. Thinking of the Hilbert space as a vector space, they're simply picking out different directions; no vector is an eigenvector of both at once.