Newton's shell theorem has two corollaries:
-
The gravitational attraction of a spherically symmetric body acts as if all its mass were concentrated at the center, and
-
The gravitational acceleration inside the cavity of a spherically symmetric body vanishes.
Consider a spaceship floating freely in space. In a homogeneous universe, the combined attraction from all matter should cancel out, and the spaceship should stay motionless. Nevertheless, I'm free to divide the attraction into several parts originating from different parts of the universe: In the figure below, I've divided the universe into a red sphere centered on some arbitrary point (×) with my spaceship located at the edge of the sphere, plus infinitely many shells centered on the same point.
By corollary #1, the gravitational attraction of the red sphere equals that of all its mass centered at the point ×. By corollary #2, the combined acceleration of the spaceship from all mass in the green shell vanishes. The same can be said for the blue shell, the orange shell, and so on ad infinitum.
Hence my spaceship should start accelerating toward ×. By choosing the sphere large enough, I should be able to make it accelerate arbitrarily fast, and by choosing the location of × I can make it accelerate in any direction.
Of course this doesn't work, but why?
My best guess is that, even in an infinite universe, you can't keep adding spheres because you'll exit the observable universe, in which case there's no way to feel the gravity in part of the shell so that it's no longer symmetric. Perhaps also the expansion of the universe matters. But see the last two points below.
A few more things to consider:
-
The mass of the red sphere increases with the chosen radius $r$ as $r^3$, while the acceleration it generates is proportional to $r^{-2}$; hence the acceleration increases linearly with the chosen $r$.
-
Our universe — the "Universe" — has an average density of some $10^{-29}\,\mathrm{g}\,\mathrm{cm}^{-3}$. Hence if I set $r$ equal to the radius of the observable Universe (46.3 billion light-years), the acceleration is a minuscule $10^{-7}\,\mathrm{cm}\,\mathrm{s}^{-2}$. If that bothers you, choose another universe where $\rho$ is ten orders of magnitude higher.
-
Our Universe is not really homogeneous, but on large enough scales ($\gtrsim$ half a billion light-years) it seems it is. Still, the acceleration of the spaceship will be dominated by nearby sources. If that bothers you, choose a sufficiently homogeneous universe.
-
On the scales we're considering, the Universe is not governed by Newtonian dynamics, but by general relativity. If that bothers you, use Birkhoff's theorem instead — I think the issue is the same.
-
If the issue is really that the size of the observable universe matters, then my intuition tells me that I can just choose an arbitrarily old universe where the asymmetric contribution from the most distant shells is arbitrarily small.
-
If the issue is that the universe expands (so that gravity from the far side of a shell is somehow weakened, or "redshifted"), then my intuition tells me that I can just choose a sufficiently static universe.
Best Answer
The problem lies in the boundary conditions. Ignoring factors of $G$ and $\pi$, gauss's law of gravitation relates the gravitational potential $\Phi$ to the mass density $\rho$ by $$\rho=-\nabla^2 \Phi. $$ In order to have a unique, well-defined solution, we need to specify boundary conditions for $\Phi$. Usually, we assume that $\rho$ dies off sufficiently quickly at spatial infinity that a reasonable choice of boundary condition is $\Phi(|\vec x|\to\infty)=0$ is. The shell theorem relies on this assumption. However in your example $\rho$ does not die off at infinity and is instead non-zero everywhere and therefore the shell theorem fails.
Often when a given scenario in physics doesn't, but almost, satisfies the 'if' part of a theorem, it can be helpful to try and modify the problem so that it does. Therefore we can use a window function $W_\epsilon(x-x_0)$ that dies off quickly as $x\to\infty$ but $\lim_{\epsilon\to0} W_\epsilon =1$ to regulate the charge density. [e.g. take $W_\epsilon(x-x_0)=e^{-\epsilon (\vec x-\vec x_0)^2}$.] Then we can replace your uniform charge density $\rho$ by $$\rho\to\rho_{\epsilon,x_0}\equiv \rho W_\epsilon(x-x_0) .$$ In this case, the shell theorem does hold. However, the result we get is not regulator-independent, that is if we solve for $\Phi_{\epsilon,x_0}$ using the charge distribution $\rho_{\epsilon,x_0}$ and then send $\epsilon \to0$, we find that our answer still depends on choice of $x_0$. This is the mathematically rigorous way to see that there really is an ambiguity when applying the shell theorem to such a situation!
Edit: There seems to be some debate in the comments as to whether the shell theorem should be proved with forces or with Gauss's law. In reality, it doesn't matter, but I will address what goes wrong if you just use forces. Essentially, Newton's laws are only guaranteed to be valid if there is a finite amount of matter in the universe. Clearly if there is uniform mass density throughout all of space, then there is an infinite amount of matter, so the shell theorem fails. The requirement that $\rho(|\vec x|\to \infty)\to 0$ 'sufficiently quickly' from above is more precisely that $\int d^3 x \rho(x) <\infty$, which is just the condition that there is a finite amount of matter in the universe.