The equipartition theorem states the following:
Let $H$ be the Hamiltonian describing a system and $x_i, x_j$ be canonical variables. Then, for a canonical ensemble with temperature $T$, it follows that:
$$\langle x_i \frac{\partial H}{\partial x_j}\rangle = \delta_{ij}kT.$$
Now consider a system with two degrees of freedom $x_1, x_2$ and canonical momenta $p_1$, $p_2$, with the Hamiltonian being
$$H(x_1,x_2,p_1,p_2) = \frac{p_1^2+p_2^2}{2m} + \kappa(x_1 – x_2)^2.$$
One can also introduce center of mass coordinates, $R = (x_1 + x_2)/2, r = x_1-x_2$, with the transformed Hamiltonian being:
$$H(R,r,p_R,p_r) = \frac{p_R^2}{2(m+m)} + \frac{p_r^2}{2(m/2)} + \kappa r^2.$$
The equipartition theorem tells us in the first case:
$$2H = p_1\frac{\partial H}{\partial p_1} + p_2\frac{\partial H}{\partial p_2} + x_1\frac{\partial H}{\partial x_1} + x_2\frac{\partial H}{\partial x_2} \Rightarrow \langle H\rangle = 2kT.$$
For the second case (center of mass coordinates):
$$2H = p_R\frac{\partial H}{\partial p_R} + p_r\frac{\partial H}{\partial p_r} + r\frac{\partial H}{\partial r} \Rightarrow \langle H\rangle = (3/2)kT.$$
Where is my mistake?
Best Answer
Indeed, even for finite size systems, it would be problematic since going to the center of mass frame is just a change of variable with jacobian one (and a nice symplectomorphism, so everything should be consistent). And, this is the hamiltonian of a diatomic particle, it was worked on during the 1880s. Two problems can arise, first, the Hamiltonian is free in the center of mass frame. That is problematic because it means that you'll need to regularize the integral to get sensitive answers (by adding a box for example, or taking the infinite size limit after performing the integral). Secondly, the bounds in your center of mass frames are highly non trivial. In the derivation of the generalised equipartition theorem, you use the fact that some quantities vanish at the boundaries (see wikipedia article about equipartition theorem), while here, the situation is a bit more complex because the integration on $R$ depends on the value of the integration boundaries on $r$ (see this MSE post). Of course at the end you take the limit of infinite size, but still, I think it invalidates the proof using integration by part given in the Wikipedia page (see Edit 1.).
If you do it by hand, you will obtain $\dfrac{3}{2}k_b T$ as expected.
So everything is fine. Note that, the NVT or NVE ensemble, considers that the center of mass is also a variable that can fluctuate. You are right (I think it was your thought process) to think that, in a realistic system, the center of mass would be conserved, as the momentum and the angular momentum. This is for example what happens in molecular dynamics and this has to be accounted for, see for example this article. As mentionned by Quillo however, for large system sizes, the difference between a system conserving the momentum, center of mass and angular momentum (molecular dynamics with infinite size) and one that does not (for example if we simulate the NVT ensemble through monte carlo) is not measurable.
Edit 1: proof that the wikipedia proof does not hold in this case.
Wikipedia places itself in the NVT ensemble:$$1 = {\displaystyle \dfrac{1}{Z}\int e^{-\beta H(p,q)}d\Gamma =\dfrac{1}{Z}\int d[x_{k}e^{-\beta H(p,q)}]d\Gamma _{k}-\dfrac{1}{Z}\int x_{k}{\frac {\partial e^{-\beta H(p,q)}}{\partial x_{k}}}d\Gamma ,}$$
Where $d\Gamma = dx_kd\Gamma_k$. Then they integrate by part and obtain:
$${\displaystyle {\dfrac{1}{Z}}\int \left[e^{-\beta H(p,q)}x_{k}\right]_{x_{k}=a}^{x_{k}=b}d\Gamma _{k}+{\dfrac{1}{Z}}\int e^{-\beta H(p,q)}x_{k}\beta {\frac {\partial H}{\partial x_{k}}}d\Gamma =1,}$$
They argue that the first integral is 0 either because the Hamiltonian diverges when evaluated at $a$ and $b$ or that $x_a = x_b = 0$. They then find that: $$\beta\langle x_j \frac{\partial H}{\partial x_j}\rangle = 1$$ as expected.
Now, let's assume your Hamiltonian, without the kinetic part: $$H = (x -y)^2 = r^2$$ this implies that:
$$\lim_{l\rightarrow\infty}\dfrac{1}{Z}\int_{-l}^{l}dx\int_{-l}^{l}dy e^{-\beta (x -y)^2} = \lim_{l\rightarrow\infty}\dfrac{1}{Z}\int_{-l}^{l}dR\int_{-2(l - R)}^{2(L - R)}dr e^{-\beta r^2} = 1$$
If we turn to the proof given by wikipedia with this Hamiltonian, we have the following integration by part:
$$\lim_{l\rightarrow\infty}{\dfrac{1}{Z}}\int_{l}^{l} dR \left[e^{-\beta r^2}r\right]_{-2(l - R)}^{2(l - R)}$$
And this won't be 0 (I might have messed up the boundary terms for the integrals... But in any case, the final result won't be 0 but something like $2/\pi$.)
Thus you see that wikipedia argument does not hold when phase space coordinates are interdependent. This also happens for the momenta, thus the equipartition theorem does not hold for them neither.