$(\log M(z)'' = - \varpi^{-2} \wp_i(z/\varpi)$ so Gauss was into the theory elliptic functions.
There is an easy way to show that for $k\ge 4$ $$G_k(i)=\sum_{m,n\ne (0,0)}\frac1{(m+in)^k} \text{ is in } G_4(i)^{k/4} \Bbb{Q}$$
Note that $G_k(i)=0$ and $G_{2k}(i)=0$ for $k$ odd.
Let
$$\wp_i(z)=\sum_{n,m} \frac1{(m+in+z)^2}-\frac{1_{(m,n)\ne (0,0)}}{(m+in)^2}$$
$\wp_i(z)$ is even, meromorphic, $\Bbb{Z}+i\Bbb{Z}$ periodic. Expand in Laurent series around $0$
$$\wp_i(z) =z^{-2}+\sum_{k\ge 1} \frac{(2k+1)! G_{2k+2}(i) }{(2k)!}z^{2k}= z^{-2}+3 G_4(i) z^2+O(z^6)$$
$$\wp_i''(z)= 6 z^{-4}+6G_4(i)+O(z^2), \qquad \wp_i(z)^2 = z^{-4}+6G_4(i)+O(z^2)$$
So $\wp_i''(z G_4(i))-6\wp_i(z)^2+30 G_4(i)$ is entire, it vanishes at $0$, and since it is $\Bbb{Z}+i\Bbb{Z}$ periodic it must be constant $=0$.
Let $G_4(i)=\frac{\varpi^4}{15}$ and $f(z)= \varpi^{-2}\wp_i(z/\varpi)$ then
$$f''(z) =6 f(z)^2-2 $$
Equating the Laurent series, you'll find by induction that $\color{red}{\text{the Laurent coefficients of f are all rational}}$. If Gauss had a rigorous proof then it was for sure equivalent to thisone, obtaining a differential equation for something closely related to $\wp_i(z)$.
The relation between $\varpi$ and $G_4(i)$ is obtained from
$$\int_0^1 \frac{dw}{\sqrt{1-w^4}}=\int_0^1 \frac{du}{2 \sqrt{u(1-u^2)}}=\frac12 \int_0^\infty \frac{du}{2 \sqrt{u(1-u^2)}}$$ On the other hand with $g_2(i)=60 G_4(i)$ and $\wp_i'(z)^2=4\wp_i(z)^3-g_2(i)\wp(z)$ (obtained similarly to the previous differential equation)
$$-\frac{1+i}{2} =\int_{(1+i)/2}^0 \frac{d\wp_i(z)}{\wp_i'(z)}=
\int_{(1+i)/2}^0 \frac{d\wp_i(z)}{\sqrt{4\wp_i(z)^4-g_2(i)\wp_i(z)}}$$ $$=
\int_0^\infty \frac{x}{\sqrt{4x^3-g_2(i)x}}$$
Best Answer
It is not quite accurate to say that Gauss discovered the prime number theorem. He made tables of the quantity
$$\int_2^x\frac{dt}{\ln t}$$
which are reprinted in Harold Edwards' book Riemann's Zeta Function. According to Edwards citing Gauss's Werke, Vol. II Gauss claims (in an 1849 letter) to have conjectured that the density of primes was $1/\log x$ around 1792 or 1793.
According to Edwards, Gauss presented no analytic basis for the conjecture but gave empirical data in the form of tables suggesting the relation.
According to the Wiki entry on the PNT Legendre in 1797 or 1798 conjectured a relation of the form $a/(A\ln a + B)$ which he later sharpened. Chebyshev subsequently showed something akin to the prime number theorem but shy of it: that if
$$ \lim_{x\to\infty}\frac{\pi(x)}{\int_2^x\frac{dt}{\ln t}}$$ exists the limit is 1.
Edwards and one of his sources say that the relation given by Legendre had been around for some time. Since Gauss didn't publish it the original source of the general idea is arguably obscure.
By the time the prime number theorem was finally proven in 1896 by Hadamard and de la Vallee Poussin (separately) is was much more than a guess and not quite a theorem.
Gauss deserves credit for the numerical observation and for the unpublished conjecture because his credibility in matters of priority is high (which Edwards and others discuss). Properly speaking we can give him credit for the prime number conjecture.