(Too long for a comment.)
The equation can be written as $\,(t-c)^2 = c^2-c\,$ then by the triangle inequality with $\lambda=|c| \lt 1\,$:
$$
|t-c|^2 = |c|\,|1-c| \le |c|(1+|c|) \quad\implies\quad |t| \le |t-c|+|c| \le \lambda + \sqrt{\lambda(1+\lambda)}
$$
Therefore $\,f(\lambda)=\lambda + \sqrt{\lambda(1+\lambda)}\,$ is an upper bound for the magnitude of roots $\,|t|\,$, but it does not insure that $\,|t| \le 1\,$ since $\,f(\lambda)\,$ can take values larger than $\,1\,$ e.g. $\,f(\lambda) \gt 1\,$ for $\,\forall \lambda \gt \frac{1}{3}\,$.
It also follows that $\,|c| \lt \frac{1}{3}\,$ is a sufficient condition for the roots to have magnitude less than $\,1\,$.
[ EDIT ] $\;$ My answer here covers this as the case $\,\beta = \gamma = c\,$. The necessary and sufficient conditions for both roots to be inside the unit circle, according to $\,(8)\,$ in that post:
$$
2 \left(|c|^2 + |c^2 - c|\right) - 1 \;\lt\; |c|^2 \;\lt\; 1
$$
Here $\,|c| \lt 1\,$, so the second inequality is always satisfied. This leaves the condition:
$$
\begin{align}
2 \left(|c|^2 + |c^2 - c|\right) - 1 &\;\lt\; |c|^2 \quad\iff\quad |c|^2 + 2\,|c^2 - c| \lt 1
\end{align}
$$
After straightforward algebraic manipulations, the condition reduces to:
$$
3\,|c|^4 + 2\, \big(3 - 4\,\text{Re}(c)\big)\,|c|^2 \lt 1
$$
For real $\,c\,$, the condition reduces to $\,3 c^4 - 8 c^3 + 6 c^2 - 1 \lt 0 \iff c \in \left(- \dfrac{1}{3}, \,1\right)\,$.
Best Answer
Assuming that $\omega, \omega_n,\zeta \in \mathbb{R}$, and using that $\,|z|^2 = z \cdot \bar z\,$:
$$ \big|G(j\omega)\big|^2 = \frac{\omega^{2}_{n}}{j\omega(j\omega + 2\zeta \omega_{n})} \cdot \frac{\omega^{2}_{n}}{-j\omega(-j\omega + 2\zeta \omega_{n})} = \frac{\omega_n^{\,4}}{\omega^2 \left(4 \zeta^2\omega_n^{\,2}+\omega^2\right)} $$