Problem
The Moon tidally brakes Earth's rotation, and as a result distances itself away from Earth as a result of conservation of total angular momentum*. Certainly, tidal braking slows the rotation of the Earth. However, and specifically, does the distancing of the Moon away from the Earth over time contribute to the increase or decrease of the Earth's rotational period (day length)?
Context
We are physics laypersons trying to understand what to us appears a dilemma, thought probably has a simple answer that escapes us. We understand that the Earth experiences tidal breaking from the Moon, i.e. the Moon slows down Earth's rotation velocity, rendering days longer. Over time, this causes the Moon to have a further orbital distance from Earth*. However, as the Moon distances itself from Earth, tidal breaking should lessen, deceleration decreases, and the Earth's rotation should increase, rendering days shorter. However, we disagree whether this is the case, and whether the contribution of (lessening) tidal braking has any impact over the conservation of total angular momentum, which suggests that a drop in Earth's rotational momentum due to tidal braking necessarily means that Earth must slow down as the Moon recedes, no matter what, which would render days longer as the Moon gradually recedes. Some of us also think that a receding Moon would result in the pulling in of Earth's ovoid tidal shape toward a more spherical shape, which would increase the rotational velocity (just as pulling in the arms of a figure skater would cause them to spin faster), rendering days shorter.
I have found this sketchy-looking paper from what may be a predatory journal which states
The earth rotation period is going to be faster [decreased, i.e. shorter days?] when the earth-moon
distance increases.
In parallel, a friend (BSc in astrophysics) has given us the exact opposite answer:
as the Moon gains "orbital" angular momentum (distance from Earth), you need to subtract angular momentum from the "rotation" angular momentum of the Earth, to conserve total angular momentum of the (closed) system
Could someone please explain how to correctly understand this problem, and settle the confusion between all of us curious about the physics? We are trying our best to understand the complete picture. Apologies for the layman's naiveté. Thank you.
*Total angular momentum is always conserved; as Earth's (spin/rotational) angular momentum decreases due to tidal braking, the Moon's (orbital) angular momentum increases to maintain total angular momentum; resultantly, the Moon's orbit grows further out.
Best Answer
A first look
Let’s call the angular momentum of the Earth’s rotation about its axis $L_E = I_E \omega_E$, where $I_E \lesssim \frac25 M_E R_E^2$ is the moment of inertia for a sphere whose density is greater at the core, and the frequency $\omega_E$ is $2\pi$ radians per day. This is somewhat less than half of the angular momentum in the Earth-Moon system.
Similarly the moon has angular momentum $L_M=I_M \omega_M$ from spinning around its axis once per month. That’s negligible compared to $L_E$, because $I_M \approx \frac{I_E}{80 \cdot 4^2}$ is small and $\omega_M \approx \omega_E/30 $ is slow.
Somewhat more than half of the angular momentum in the Earth-Moon system is from the orbital term. We have $I_{EM} / I_E \approx \frac{60^2}{80} \approx 60$, because the Moon is about 1/80th of Earth’s mass but it is 60 Earth radii away.* But the month $\omega_{ME} = \omega_M$ is about thirty times slower than the day $\omega_E$, so the angular momentum product comes out comparable.
If we approximate that the Moon orbits in Earth’s equatorial plane (it doesn’t), so that all the angular momenta are parallel, we have the conserved angular momentum
$$ L_\text{total} = I_E \omega_E + I_{EM} \omega_M = \text{constant} $$
and the two terms contribute similarly.
Tidal locking happens as rotational energy is extracted to do dissipative work. The rotational energy is
\begin{align} T &= \frac12 I_E \omega_E^2 + \frac12 I_{EM} \omega_M^2 \\ &= \frac{L_E^2}{2 I_E} + \frac{L_{EM}^2}{2 I_{EM}} \end{align}
Since the angular momenta $L_E$ and $L_{EM}$ are within a factor two or so, but the Earth has a smaller moment $I_E$ and larger frequency $\omega_E$, then nearly all of the kinetic energy is from the spinning-Earth term. Furthermore Earth’s moment of inertia is fixed,† while the Earth-Moon distance is free to vary. Therefore any increase in Earth’s rotational speed would increase the total rotational energy of the system:
\begin{align} L_E &\to L_E + \delta L \\ L_{EM} &\to L_{EM} - \delta L \\ T &\to T + \delta L\cdot \left( \frac{L_E}{I_E} - \frac{L_{EM}}{I_{EM}} \right) + \frac{\mathcal O (\delta L^2)}{I} \\ &\approx T + \delta L \cdot ( \omega_E - \omega_M ) \end{align}
Tidal locking therefore happens as the total kinetic energy $T$ is decreased by the transfer of angular momentum in the direction $L_E \to L_{EM}$. This will continue until the frequencies $\omega_E, \omega_M$ become the same. If $I$ is fixed, the way to release kinetic energy is to transfer angular momentum from the higher-frequency subsystem to the lower-frequency one.
More careful: use total energy
What about the Moon, whose orbital moment $I_{EM}$ is not fixed? Could the tidal locking process perhaps speed up its orbit? To answer this, we have to remember that the Moon is held in its orbit by gravity, with centripetal acceleration
\begin{align} \frac{\text{constant}}{R_{EM}^2} &= R_{EM} \omega_{EM}^2 \\ \text{constant} &= R_{EM}^3\omega_{EM}^2 \end{align}
If we increase the orbital angular momentum via some interaction, the new orbit will still follow this constraint:
\begin{align} L & \propto R^2\omega & L &\to (1 + \epsilon) L \\ & & \frac{L^2}{R^3\omega^2} & \to \frac{1+2\epsilon} {\text{constant}} \frac{L^2}{R^3\omega^2} \\ R &\propto \frac{L^2}{R^3\omega^2} & R &\to (1 + 2\epsilon) R \\ \omega &\propto \frac{L}{R^2} & \omega &\to \frac{1+\epsilon}{1-4\epsilon}\omega \approx (1 - 3\epsilon)\omega \end{align}
So for a gravitational orbit, increasing its angular momentum will necessarily make it bigger and slower. If you allow the size of the orbit to change you have to include changes in the gravitational potential energy $U_{EM} = -\frac{G M_E M_M}{ R_{EM}}$ in your energy calculations. After a nontrivial amount of algebra, we end up with the same result for the total energy $E = T + U$ that we got above for the fixed-distance case,
$$ \frac{\mathrm dE}{\mathrm dL_E} = \omega_E - \omega_M, $$
or in words, the total energy of the system is reduced when angular momentum flows from the faster rotation to the slower one.
Note that this is a "mechanism-free" argument. Another answer here, and the Wikipedia page about Phobos (on which more below), go into detail about the locations of the "tidal bulges." But the "tidal bulge" model is mostly bogus, especially for Earth's oceans. This model says that if the mechanical energy of the Earth-Moon system is reduced, without changing $L$, it must also transfer angular momentum out of Earth's spin and into the Moon's orbit. You can push the Moon away by improving the efficiency of your local tidal-powered electrical generator.
Stability
Let's emphasize that this analysis also applies to other planets and their orbiting moons with a change of notation. We'll talk about "a planet" with mass $M$ and radius $R$, with "a moon" having mass $m<M$ and radius $r$, in an equatorial, circular orbit with radius $a > R+r$. The angular momenta for the planet $L_p$ and the orbit $L_o$ are related to the total energy $E = T+U$ by
\begin{align} L &= L_p + L_o = I_p\omega_p + I_o \omega_o \\ E &= \frac{L_p^2}{2I_p} - \frac{L_o^2}{2I_o} \end{align}
This formulation makes it clear that the angular momentum $L_m = I_m \omega_m$ is negligible unless the moon in spinning very fast, or is very large and very close. The moment of inertia for the orbit is $I_o = \mu a^2$, where $\mu = (\frac1m + \frac1M)^{-1} \lesssim m < M$ is the "reduced mass." This orbital moment is always§ larger than the moon's rotational moment $I_m \lesssim mr^2 < m(r+R)^2 < ma^2$.
Shifting angular momentum to the planet while holding $L$ constant changes the energy by
$$ \frac{\mathrm dE}{\mathrm dL_p} = \omega_p - \omega_o $$
That is, energy is released when angular momentum flows from the faster-rotating part of the system to the slower-rotating part of the system. For the Earth-Moon system this has the effect of making both the spin and the orbital frequencies slower: the extra angular momentum causes the Moon's orbit to move out to a larger radius.
But for a system with $\omega_p < \omega_m$, energy will be released when angular momentum flows to the planet. The extra spin energy for the planet, as well as whatever energy was released, has to come from the gravitational energy $U=-\frac{Gmm}{a}$ of the moon, which would move into a lower, faster orbit. It's fair to ask, then: is a tidally-locked orbit a stable configuration? Or, having achieved a tidal lock, will the system continue to release energy by having the moon move closer and faster, until it crashes into the planet's surface?
The condition for stability is that the second derivative of the energy be positive:
\begin{align} \frac{\mathrm d^2E}{\mathrm dL_p^2} = \frac{1}{I_p} - \frac{3}{I_o} &> 0 \\ I_o &> 3I_p \end{align}
We've already seen that, in the Earth-Moon system, $\frac{I_{EM}}{I_E} \approx 10^2$, and getting larger as the Moon moves away. But for a different system with a smaller, nearer moon, it's possible for the orbital moment $I_o \approx ma^2$ to be smaller than the planet's moment $I_p \lesssim MR^2$.
Note that since we always have $I_m<I_o$, it's nearly always stable for the moon to become tidally locked to the planet.
Interesting cases
Let’s look at the consequences of these relationship in a few interesting scenarios.
A planet spinning faster than the orbit of its moon will evolve towards a stable tidal lock. This is the Earth-Moon system. We can figure out the eventual separation $a$ and frequency $\omega$ by fixing the constraints \begin{align} (I_p + \mu a^2) \omega &= L \\ a^3 \omega^2 &= G(M+m) \end{align} Using this value of $I_E \approx \frac13 M_E R_E^2$ for Earth's moment of inertia, I get an equilibrium distance of about 40% larger than the current Earth-Moon distance, with a period of about 47 days.
I have read that the Earth-Moon tidal lock may not finish happening before the Sun turns into a red giant and swallows them both, but I haven't tried to compute that.
On the graph below, this is the middle region.
An planet spinning very much faster than is moon is only stable in isolation. If the equilibrium distance is too far, the moon will eventually wander out of the planet's "Hill sphere" and be lost. I think this is related to the fact that the rapidly-spinning planet can give the total system energy $E>0$, which is gravitationally unbound. The Earth-Moon system currently has $E>0$, but would have $E<0$ at the equilibrium distance I described. The radius of Earth's Hill sphere is about three or four current Earth-Moon distances.
Mars's outer moon, Deimos, could migrate out of Mars's Hill sphere without materially changing the length of the Martian day.
A planet rotating more slowly than the orbit of a distant moon can evolve towards a stable, tidally-locked equilibrium. However, the equilibrium is achieved by speeding up both the planet's rotation and the moon's orbit. On the graph below, this is the region on the right.
A planet rotating more slowly than the orbit of a nearby moon is in for an adventure.
The tidally-locked equilibria in the energy curve come in pairs: a distant, stable equilibrium with $I_o = \mu a^2 > 3I_p$, and a nearer, unstable equilibrium with $\mu a^2 < 3I_p$. A system where the moon is nearer than the inner equilibrium will (as in case 3) release energy by transferring angular momentum to the planet, moving the moon closer. For the too-near moon, this inward motion is away from the equilibrium, instead of towards, and will end with the moon crashing into the planet's surface.
In the limit of a zero-mass satellite, the unstable equilibrium is the geosynchronous orbit. This is why atmospheric drag makes low-Earth satellites fall out of orbit: they trade angular momentum with Earth by interacting dissipatively with the atmosphere, rather than tidally. In the longer term, this is the fate of Mars's moon Phobos, which rises in Mars's western sky about three times a day.
There is no guarantee that the pair of equilibrium points exists, in which case the moon is also doomed; see below.
Here's a plot showing the total energy $E$ and frequencies for the Earth $\omega_E$ and Moon $\omega_M$, in orbits where the Moon is tidally locked to the Earth and the total angular momentum $L$ for the system is equal to the $L$ that we have today:
The horizontal axis is the Earth-Moon distance, in units of the current Earth-Moon distance $\renewcommand{\anow}{a_\text{now}} \anow$. Currently the Moon orbits at $a = \anow$, and the Earth spins about thirty times faster than the Moon rotates. You can see the stable equilibrium at $a \approx 1.4\anow$. The unstable equilibrium for our large Moon is extremely close to Earth, at only about $0.04\anow \approx 2.2 R_E$. That's actually quite close to where the Moon formed — perhaps the model making that prediction is that the Moon formed from an accretion disk straddling the unstable equilibrium.
If you change this model so that Earth spins more rapidly while the Moon orbits at its current distance, the $\omega_E$ curve shifts up and the two equilibria move further apart. If you give the Earth a very fast retrograde rotation, it's possible to move the $\omega_E$ curve down enough that there is no stable equilibrium.
tl;dr summary
Interactions between co-orbiting objects which reduce their mechanical energy do so by moving angular momentum from the higher-frequency rotation modes to the lower-frequency rotation modes. The Earth and Moon are both slowing towards an equilibrium in which they would be mutually tidally locked; that equilibrium will be stable because the Moon is large and far from the Earth. However, moons that are very small, or very near their planets, or in retrograde orbits, may be unstable against tidal interactions; those moons will eventually fall out of the sky.
Notes
* I’m being intentionally sloppy about the $\frac25$, which is wrong anyway because of Earth’s dense core. I’m also ignoring, for now, Earth’s orbital motion about the Earth-Moon barycenter.
† Earth’s moment of inertia changes a little bit as the shape of the Earth changes, e.g. during quakes, landslides, volcanic eruptions, and the falling of leaves in the autumn. In the longer term Earth's moment of inertia is changed by convection currents in the mantle.
§ Unless (possibly) you get into a weird edge case where the two bodies are very close and the lower-mass object has a very low density. We'd have to revisit a lot of our assumptions for such a system.