Got the answer... Brutally... Took about 1 hour work and a page of letter size by hand. It at least seemed to me that using inversion did not make things simpler... But maybe someone can work this out?
Anyway this is what I did. Suppose that $a < 9b$. (Need to check what should be corrected if $a > 9b$).
Step 1: Compute $l$.
See there is a right triangle. By the Pythagorean theorem,
$$
l^2 = (a+b)^2 - (a-b)^2 \quad \Rightarrow \quad l = 2\sqrt{ab}.
$$
Step 2: Compute $d$. This is the bloody part.
If the radius of the giant circle is $R$, the two segments connecting the centers should be of length $R-a$ and $R-b$, respectively. Use two Pythagorean theorem to calculate their length gives the equation
$$
a - b = \sqrt{(d+b)^2 + (a+2\sqrt{ab})^2} - \sqrt{(d+a)^2 + a^2}.
$$
Typically this involves squaring the equation twice. First, move the terms:
$$
\sqrt{(d+a)^2 + a^2} + a - b = \sqrt{(d+b)^2 + (a+2\sqrt{ab})^2}.
$$
Take square on both sides. Simplify it, YOLO!
$$
-a(\sqrt a - 3\sqrt b) - (\sqrt a - \sqrt b)d = (\sqrt a - \sqrt b)\left(\sqrt{(d+a)^2 + a^2}\right)
$$
Take square another time. Observe that $d^2$ term cancelled and it is indeed a linear equation. Thank god...
$$
\begin{aligned}
2(\sqrt a - \sqrt b)d(2\sqrt b) &= a(-a - 2\sqrt{ab} + 7b)\\
d &= \frac{a(-a - 2\sqrt{ab} + 7b)}{4\sqrt b (\sqrt a - \sqrt b)}.
\end{aligned}
$$
Step 3: Compute $R$.
Well there is nothing interesting... $R = \sqrt{(d+a)^2 + a^2} + a$. One thing worth mentioning is that the formula $(m^2 - n^2)^2 + (2mn)^2 = (m^2+n^2)^2$. Here
$$
d+a = \frac{-a(\sqrt a - 3\sqrt b)(\sqrt a + \sqrt b)}{4\sqrt b(\sqrt a - \sqrt b)}.
$$
Using $m = \sqrt a - \sqrt b$, $n = 2\sqrt b$, we get
$$
(d+a)^2 + a^2 = \left(\frac{a}{4\sqrt b(\sqrt a - \sqrt b)}\right)^2(a - 2\sqrt{ab} + 5b)^2.
$$
Therefore,
$$
R = \sqrt{(d+a)^2 + a^2} + a = \frac{a(\sqrt a + \sqrt b)^2}{4\sqrt b(\sqrt a - \sqrt b)}.
$$
Step 4: Compute $c$.
Using the Pythagorean theorem on the right triangle shown above, we have
$$
(a+c)^2 = a^2 + (R-a-d-c)^2.
$$
As
$$
R-a-d = \frac{a(\sqrt a + \sqrt b)}{2\sqrt b},
$$
we get the equation
$$
2ac + c^2 = \left(\frac{a(\sqrt a + \sqrt b)}{2\sqrt b} - c\right)^2.
$$
Finally,
$$
c = \frac{a(\sqrt a + \sqrt b)^2}{4\sqrt b(\sqrt a + 3\sqrt b)}.
$$
Best Answer
In fact, you already almost have the small circle's radius.
The height of the right triangle can be $r+\dfrac {3} {4} a$.(r: small circle's radius) also, the hypotenuse of the right triangle can be $a-r$, and the base of the triangle is $\dfrac a 2 $. You can try it from here.