The comments on the question provide a good amount of insight to finding a solution, but I noticed there were still unanswered questions on how to solve the original problem.
As you said, the straightforward way is to find the real and imaginary parts, then use the Pythagorean formula to find the magnitude. separating into real and imaginary parts is usually done as @fleablood stated by multiplying the top and bottom of the fraction by the denominator's complex conjugate. Then, the fraction can be separated into real and imaginary parts by the terms in the numerator.
In your case, let's start with the $H(j\omega)$ version of your rational function, as it seems to be what you're looking for. (It can be done but gets more complicated if you are working in the $s$ domain where $s=\sigma + j\omega$).
$$H(j\omega)=\frac{1−j\omega}{(j\omega)^2+j\omega+1}$$
The first simplification is in the denominator, as $j^2=-1$
$$H(j\omega)=\frac{1−j\omega}{-\omega^2+j\omega+1}$$
$\omega$ is a real valued number, so the complex conjugate for the denominator just has the sign of the complex term changed: $1-\omega^2-j\omega$
Now, rationalizing the fraction:
$$H(j\omega)=\frac{1−j\omega}{1-\omega^2+j\omega}\times\frac{1-\omega^2-j\omega}{1-\omega^2-j\omega}$$
$$ = \frac{j\omega^3-2\omega^2-2j\omega+1}{\omega^4-\omega^2+1}$$
$$ = \frac{-2\omega^2+1}{\omega^4-\omega^2+1}+j\frac{\omega^3-2\omega}{\omega^4-\omega^2+1}$$
With the real and imaginary parts separated, we can find the magnitude:
$$\lvert{H(j\omega)}\rvert = \sqrt{\left(\frac{-2\omega^2+1}{\omega^4-\omega^2+1}\right)^2+\left(\frac{\omega^3-2\omega}{\omega^4-\omega^2+1}\right)^2}$$
$$= \sqrt{\frac{\left(-2\omega^2+1\right)^2+\left(\omega^3-2\omega\right)^2}{\left(\omega^4-\omega^2+1\right)^2}}$$
$$= \sqrt{\frac{\omega^6+1}{\left(\omega^4-\omega^2+1\right)^2}}$$
$$= \sqrt{\frac{\left(\omega^2+1\right)\left(\omega^4-\omega^2+1\right)}{\left(\omega^4-\omega^2+1\right)^2}}$$
$$\lvert{H(j\omega)}\rvert = \sqrt{\frac{\omega^2+1}{\omega^4-\omega^2+1}}$$
There may be other ways of doing the algebraic manipulation, but this seemed the most straightforward. (Note that factoring $\omega^6+1$ uses the Sum of Cubes Identity recognizing that $\omega^6+1 = \left(\omega^2\right)^3+1$.
The final result was verified with the Maxima 5.43.0 cabs function, which results in the equivalent expression:
$$\lvert{H(j\omega)}\rvert = \sqrt{\frac{\omega^2+1}{\left(1-\omega^2\right)^2+\omega^2}}$$
Best Answer
You're almost there: \begin{align} & \sqrt{a^2+b^2} = k \\[10pt] & a^2+b^2 = k^2 \\[10pt] & a^2 = k^2 - b^2 \\[10pt] & a = \sqrt{k^2 - b^2} \end{align} and we don't need to say $\text{“}{\pm}\text{''}$ because we know $a\ge0.$