Here are two circles $C_1 = [ x^2 + y^2 = 1]$ and $C_2 = [ (x-2)^2 + y^2 = 3 ]$. The radii are $1$ and $3$ and the two circles are orthogonal to each other $C_1 \cdot C_2 = 0$. What is the equation of a third circle cutting the other two at right angles? I suspect there are only two solutions which are equivalent by a reflection.
Looking for a geometric solution if possible. Counting the number of solutions, finding the equation of the circles and maybe some insights into the method.
Related: Find a circle orthogonal to two other circles
The Inner Product of Two Circles
If $C = [ a(x^2 + y^2) – 2px – 2qy + r = 0]$ the inner product between two such circles would be:
$$ [C_1 \cdot C_2] = \frac{1}{2}[2p_1p_2 + 2q_1q_2 – r_1a_2 – r_2a_1] $$
So we are trying to find three circles $C_1, C_2, C_3$ the first two given, such that:
$$ [C_1 \cdot C_2] = [C_2 \cdot C_3] = [C_3 \cdot C_1 ]= 0 $$
The inner product can also be written in terms of the radii of the circles and the distance between the centers:
$$ [C_1 \cdot C_2 ] = \frac{1}{2} [ R_1^2 + R_2^2 – D^2]$$
and if $C_1 \perp C_2$ we have that this inner product is zero, $[C_1 \cdot C_2 ] = 0$.
Not too many resources on the "algebra of circles", e.g. Geometry a Comprehensive Coruse by Daniel Pedoe.
Descartes theorem deals with mutually tangent circles.
Best Answer
Inversive Geometry
We can find the family of perpendicular circles using Inversive Geometry. If we invert the circles $x^2+y^2=1$ (green) and $(x-2)^2+y^2=3$ (blue) through a circle (gray) whose center is their intersection at $\left(\frac12,\frac{\sqrt3}2\right)$ and that passes through their intersection at $\left(\frac12,-\frac{\sqrt3}2\right)$, the circles are mapped to a pair of (dashed) lines.
Perpendicular Circles
Since inversion is conformal, to find a circle perpendicular to both the green and blue solid circles, we simply need to find the inverse of a circle perpendicular to both the green and blue dashed lines. That is, any circle centered at the intersection of those lines.
The red dashed circles are centered at the intersection of the green and blue dashed lines (which are the inverses of the green and blue solid circles), and are perpendicular to both the green and blue dashed lines. The red solid circles are the inverses of the red dashed circles, and are therefore perpendicular to both the green and blue solid circles.
Computing The Inverse Circles
Since the inversion center and the centers of the red dashed circles are on the line $x=\frac12$, the centers of the red solid circles will also be on that same line. Given that the radius of the inversion circle is $\sqrt3$ and the radius of the dashed red circle is $r$, we have
a point at a distance of $\sqrt3-r$ below the inversion point is mapped to a point at at distance of $\frac3{\sqrt3-r}=\frac{3\left(\sqrt3+r\right)}{3-r^2}$ below the inversion point (negative values are above).
a point at a distance of $\sqrt3+r$ below the inversion point is mapped to a point at at distance of $\frac3{\sqrt3+r}=\frac{3\left(\sqrt3-r\right)}{3-r^2}$ below the inversion point (negative values are above).
The radius of the red solid circle is half the distance between these points: $\left|\frac{3r}{3-r^2}\right|$.
The center of the red solid circle is the average of these distances below (or above for negative values) the inversion center: $\frac{\sqrt3}2-\frac{3\sqrt3}{3-r^2}=-\frac{\sqrt3}2\frac{3+r^2}{\left(3-r^2\right)}$