Qualitatively, the tradeoff in uncertainty between two non-commuting observables $\hat{x}$ and $\hat{y}$, could be explained by…
- the Fourier picture where the more one variable is defined (i.e., localized) the more is its Fourier transform undefined (i.e., spread out) in the conjugate domain. Examples include time and frequency (aka. energy), etc.
- the notion that (some amount of) wave function collapse is associated with the measurement process. The measurement retrieves the (supposedly pre-existing) value of the measured observable but irrevocably disturbs the conjugate variable (through the stochastic action of unfathomable hidden variables).
Point #1 seems "cleaner" mathematically whereas #2 is arguably more physical and intuitive. At any rate, to what extent are #1 and #2 related and is #2 even a valid intuition for the uncertainty principle?
PS: Cf. a video by Sean Carroll who claims that #2 is completely unrelated to the uncertainty principle but he doesn't explain why.
Best Answer
It is important to clarify the distinction between two different things that have been confused since ever for historical reasons:
The (Parallel) Uncertainty Relations
The uncertainty relations express the inherent impossibility of preparing a state whose conjoint variance over two observables cannot be reduced arbitrarily. Here, what do we mean by the variance of an observable? We mean that we have infinitely many identical copies of a given state (yes, you cannot clone a state but you can simply prepare identical states) and we perform a measurement of an observable over all those copies. The variance of the observable for the given state is simply the statistical variance of all those observed values. Now, if you want to see how the conjoint variance over two observables behaves for the given state, you perform the measurement of one observable over half of the infinitely many identical copies of the given state and that of another over another half of the infinitely many identical copies of the given state. You simply multiply the statistical variances of the observed values of the two observables to get the conjoint variance.
As you can see, as we spell out what is meant by a product such as $\Delta_\psi x\Delta_\psi p$ (where the $\Delta_\psi$ means that the variance is over the state $\psi$), it is incredibly clear that this has nothing to do with how the measurement of one observable might disturb the value of another observable. We are not doing sequential measurements, we are doing parallel measurements on unentangled states.
Now, why would there be a lower limit on such products of variances? It simply has to do with the fact that non--commuting observables cannot be simultaneously diagonalized. They may or may not be Fourier conjugates but to the extent that they cannot be simultaneously diagonalized, their variances cannot simultaneously vanish over a state. This is formalized by the Robertson-Schrodinger uncertainty relations.
The (Sequential) Uncertainty Principle
Now, the historical uncertainty principle of Heisenberg referred to how the measurement of one observable would disturb the value of another observable. The first thing to keep in mind here is that this was in the early days of quantum mechanics and it was one of the examples of muddled thinking. The equation that Heisenberg associated with his uncertainty principle actually described the inherent impossibility of preparing a state that is a simultaneous eigenstate of two non--commuting observables, i.e., what we described in the previous section. However, his physical reasoning appealed to what would happen in sequential measurements of two observables over a given state -- not parallel measurements. Now, a sequential version of the uncertainty principle (for which the physical reasoning of "disturbance" is correct) also exists -- but it is not what we usually mean when we talk about the uncertainty principle -- unless otherwise specified, we mean what I called the parallel uncertainty relations.
As for how the sequential uncertainty principle works out, it is quite simple to imagine as to why it would work out. Let's say you have a Gaussian wave-packet of width $d$ that minimizes $\Delta_\psi x\Delta_\psi p$ -- so that $\Delta_\psi p = 1/(\sqrt{2}d)$ in natural units. This means that if you take infinitely many copies of this state and measure the momentum over each of the states, the statistical variance in the observed values will be $1/(\sqrt{2}d)$ in natural units. Now, take another fresh batch of infinitely many identically prepared wave-packets described earlier. You now measure $\hat{x}$ first and then measure $\hat{p}$ on the post-measurement states. When you measure $\hat{x}$ on a state, it will make the post-measurement wavefunction spiked near some value of $x$ and make the $\Delta_{\psi'} x$ very small (the more precise your measurement, the smaller will the variance will be). Notice that $\psi'$ in the subscript says that the variance is over the post-measurement state. Now, since the $\Delta_{\psi'}x$ is small for the state $\psi'$, the $\Delta_{\psi'}p$ will be huge for the state $\psi'$ because of the parallel uncertainty relations. And thus, the act of measurement of $x$ on $\psi$ has increased the $\Delta p$ on the post-measurement state $\psi'$. A precise mathematical expression for the sequential uncertainty relations is more complex than the simple parallel uncertainty relations. They have been derived by Paban and Distler (2012), see https://arxiv.org/abs/1211.4169. They have also been discussed earlier in the literature, for example, Gnanaprasagam and Srinivas (1979), see https://link.springer.com/article/10.1007/BF02846858. However, I think the treatment by Paban and Distler is more thorough and modern.
Finally, I hope it is clear that the sequential uncertainty principle relies on the parallel uncertainty relations. In particular, the physical argument of the disturbance works only because the parallel uncertainty relations hold for the post-measurement state.