Without knowing the mass of the Earth, calculating the gravitational constant is impossible from $g$ and the acceleration of the Moon. The best you can do is calculate the product of the gravitational constant and the Earth's mass (GM). This is why Cavendish's experiments with the gravity of lead weights was important, since the mass of the body providing the gravitational force was known. Once $G$ was calculated from this experiment, the Earth could then be weighed from using either $g$ or the Moon's acceleration (both hopefully yielding the same answer).
The suggestion in the previous paragraph that Cavendish's experiment resulting in a value for $G$ is still not quite right. While a value for $G$ could have been determined from the experiment, Cavendish only reported the specific gravity (the ratio of a density to water's density) of Earth. According to Wikipedia, the first reference in the scientific literature to the gravitational constant is in 1873--75 years after Cavendish's experiment and 186 years after Newton's Principia was first published:
Cornu, A.; Baille, J. B. (1873). "Détermination nouvelle de la constante de l'attraction et de la densité moyenne de la Terre" [New Determination of the Constant of Attraction and the Average Density of Earth]. C. R. Acad. Sci. (in French). Paris. 76: 954–958.
Click on the link if you read French or can find a translator. Also, the symbol $f$ is used instead of $G$.
Newton's Principia can be downloaded here:
https://archive.org/stream/newtonspmathema00newtrich#page/n0/mode/2up
Follow up questions copied from the comments (in case the comment-deletion strike force shows up):
So how exactly did Newton express his universal gravitational law. Was it like this "$F_g$ is equal to $GMm/r^2,$ but I must avow that I doth not know neither $G$ nor big $M$". Or did he just assign some number "$X$" to the gravitational effect due to the Earth, which ended up being $GM$?
Philip Wood: I'm pretty sure that Newton never wrote his law of gravitation in algebraic form, nor thought in terms of a gravitational constant. In fact the Principia looks more like geometry than algebra. Algebra was not the trusted universal tool that it is today. Even as late as the 1790s, Cavendish's lead balls experiment was described as 'weighing [finding the mass of] the Earth', rather than as determining the gravitational constant. Interestingly, Newton estimated the mean density of the Earth pretty accurately (how, I don't know) so he could have given a value for G if he'd thought algebraically
Mark H: Philip Wood is correct. Newton wrote Principia in sentences, not equations. The laws of gravity were described in two parts (quoting from a translation): "Tn two spheres mutually gravitating each towards the other, ... the weight of either sphere towards the other will be reciprocally as the square of the distance between their centres." And, "That there is a power of gravity tending to all bodies, proportional to the several quantities of matter which they contain." This is the full statement of the behavior of gravity. No equations or constants used.
Who first measured the standard gravitational acceleration 9.80 m/s/s? I assume that was well known by the time of Newton?
After a quick search, I can't find who first measured
$g=9.8m/s^2$. It's not a difficult measurement, but would require accurate clocks with subsecond accuracy. This is an interesting article: https://en.wikipedia.org/wiki/Standard_gravity
Actually, on page 520, Newton lists the acceleration due to gravity at Earth's surface like so: "the same body, ... falling by the impulse of the same centripetal force as before [Earth's gravity], would, in one second of time, describe 15 1/12 Paris feet." So, the value was first measured sometime between Galileo's experiments and Newton's Principia.
Was Newton (and therefore all of us!) just a tiny bit luck y that the ratios worked out so nicely. I'm not putting down Sir Isaac (perhaps the smartest bloke who's ever drawn breath in tights), but even I might notice that
$\frac{g(Earth)}{a_c(Moon)}=3600=\left(\frac{r(Earth−to−Moon)}{r(Earth)}\right)^2$.
If the ratio had been a little messier, say one to 47½, it might have been a little harder to spot the connection.
Newton knew that the moon was not exactly 60 earth-radii distant. He quotes a number of measurements in Principia: "The mean distance of the moon from the centre of the earth, is, in semi-diameters of the earth, according to Ptolemy, Kepler in his Ephemerides, Bidliuldus, Hevelius, and Ricciolns, 59; according to Flamsted, 59 1/3; according to Tycho, 56 1/2; to Vendelin, 60; to Copernicus, 60 1/3; to Kircher, 62 1/2 (p . 391, 392, 393)." He used 60 as an average, which results in an easily calculable square, but squaring isn't a difficult calculation anyway.
The inverse square law was already being talked about by many scientists at the time, including Robert Hooke. Newton used the Moon as a confirmation of the inverse square law, not to discover it. He already knew what the answer should be if the inverse square law was true. In fact, it was the orbital laws discovered by Johannes Kepler--especially the constant ratio of the cube of the average distance from the central body and the square of the orbital period--that provided the best evidence for the inverse square law.
In "The System of the World" part of Newton's Principia, he uses astronomical data to show that gravity is a universal phenomena: the planets around the Sun, the moons around Jupiter, the moons around Saturn, and the Moon around Earth. For the last, in order to establish the ratio of forces and accelerations, you need at least two bodies. Since Earth only has one moon, he made the comparison with terrestrial acceleration.
I would love to read a proof (requiring less mathematical nous than Sir Isaac had at his disposal) for the connection from Kepler's 3rd law to Newton's inverse square. Do you know of one?
A simple version of Kepler's Third Law to the inverse square law can be shown for circular orbits pretty easily. Define $r$ as the constant radius of the orbit, $T$ as the time period of the orbit, $v$ as the planet's velocity, $m$ as the mass of the orbiting planet, $F$ as the gravitational force, and $k$ as some constant.
\begin{align}
\frac{r^3}{T^2} = k &\iff r^3 = kT^2 \\
&\iff r^3 = k\left(\frac{2\pi r}{v}\right)^2 \\
&\iff r = \frac{4\pi^2k}{v^2} \\
&\iff \frac{v^2}{r} = \frac{4\pi^2k}{r^2} \\
&\iff \frac{mv^2}{r} = \frac{4\pi^2km}{r^2} \\
&\iff F = \frac{4\pi^2km}{r^2}
\end{align}
The quantity $v^2/r$ is the centripetal acceleration necessary for constant speed circular motion.
Best Answer
It's true that if you know the masses of, e.g. two orbiting stars $M_1$ and $M_2$, their orbital period $T$, and the distance $d$ between them, then you know $G$. And we can measure $T$ pretty well and $d$ fairly well.
But how do you think we figure out the masses of the stars? We can't just count the amount of stuff in them; we have to infer the mass from how hard they pull on other objects. So we actually determine $M$ using the known value of $G$. Since we don't know stellar masses any other way, we can't flip the measurement around to get a better value for $G$.
You might think we could calculate stellar masses directly using what we know about fusion, but that doesn't work either: a star needs to exert enough outward pressure to cancel its weight, and that weight is proportional to $G$. In other words, $G$ is an input, so it can't be an output.