Have you considered finding the intersections using an implicit form for the circles, $$\frac{x^2}{r^2} + \frac{y^2}{r^2} + ax + by + c = 0?$$ This representation doesn't have any coefficients that diverge as the circle approaches a straight line. To find intersections, you'll have to solve a quadratic equation whose leading coefficient could be zero or arbitrarily close to it, but the alternative form of the quadratic formula should be able to deal with that robustly.
You'll then have to do some jiggery-pokery to figure out whether the intersection points lie within the arcs. If the arc's bending angle is smaller than $\pi$, a projection onto the line joining the endpoints will suffice.
(Disclaimer: While all of this feels like it should work, I haven't analyzed it in any detail. Also, there could still be a problem when the circle is close to a line and you want the longer arc. But I can't imagine that's a case that would turn up in any practical application.)
Update: For a concrete example, here is the equation for a circular arc passing through the three points $(0,0)$, $(0.5, h)$, and $(1,0)$: $$\kappa^2 x^2 + \kappa^2 y^2 - \kappa^2 x - 2\eta y = 0,$$ where $$\begin{align}\kappa &= \frac{8h}{4h^2 + 1}, \\ \eta &= \frac{8h(4h^2-1)}{(4h^2+1)^2}.\end{align}$$ As you can see, the coefficients remain bounded as $h \to 0$.
Update 2: Wait, that equation becomes trivial if $h = 0$, which is bad. We really want something like $x^2/r + y^2/r + ax + by + c,$ i.e. multiply the previous expression through by $r$. Then for the same example, our equation becomes $$\kappa x^2 + \kappa y^2 - \kappa x - 2\eta' y = 0,$$ where $\eta' = (4h^2-1)/(4h^2+1)$. Here are some explicit values.
$h = 1/2$: $$2 x^2 + 2 y^2 - 2 x = 0,$$ $h = 0.01$: $$0.07997 x^2 + 0.07997 y^2 - 0.07997 x + 1.998 y = 0,$$ $h = 0$: $$2 y = 0.$$
By the way, in this format, the linear terms will always be simply $-2(x_0/r)x$ and $-2(y_0/r)y$, where the center of the circle is at $(x_0,y_0)$. As the center goes to infinity but the endpoints remain fixed, these coefficients remain bounded and nonzero (i.e. not both zero).
In addition to the answer pointed out by @n_b I would like to show how to reach there:
The final result:
$$\LARGE\boxed{R=\frac H2+\frac{W^2}{8H}}$$
float cos_t = (R-H)/R;
P_x=(rect.width/2);
P_y=-R.cos_t;
Process:
Without using calculus(actually it would never be solved with calculus) and only using trigonometry:
The component of R along $\theta$ is $R.\cos\theta$ and perpendicular to it is $R.\sin\theta$
Now remaining component is $R-Rcos\theta=R(1-\cos\theta)$
Now we have:
$$
W=2.R.\sin\theta\implies\sin\theta=\frac W{2R}\\
H=R(1-\cos\theta)\implies\cos\theta=1-\frac HR=\frac{R-H}R
$$
Using the identity $\sin^2\theta+\cos^2\theta=1$
$$\left(\frac W{2R}\right)^2+\left(\frac{R-H}R\right)^2=1$$
$$\frac{W^2}{4R^2}+\frac{R^2+H^2-2RH}{R^2}=1\\\text{ I assume you are familiar with }(a\pm b)^2=a^2+b^2\pm 2ab$$
$$\frac{W^2}4+R^2+H^2-2RH=R^2$$
$$\frac{W^2}4+H^2=2RH$$
$$R=\frac{W^2}{8H}+\frac H2$$
It is easy to see that center is at the midpoint of width.
Also y-coordinate is $-R.\cos\theta$
Best Answer
Assuming that your hole is indeed circular, it can be done like this.
Of course we'll call the radius $r$. Also, I'm using $w$ as half the measured width. In this diagram, $$ \sin \theta = \frac{w}{r} \quad\quad\quad \cos\theta = \frac{r - h}{r} $$
One way to combine these expressions is by using the identity $$ \sin^2\theta +\cos^2\theta = 1 $$
After some simplification, this produces $$ r = \frac{w^2 + h^2}{2h} $$
We can also find the angle $\theta$ now $$ \sin \theta = \frac{w}{r} \\ \theta = \arcsin\frac{w}{r} \\ \theta = \arcsin\frac{2hw}{w^2+r^2} $$
Combining these together we can find the arclength $l$ $$ l = r(2 \theta) \\ l = 2 \frac{w^2 + h^2}{2h} \arcsin\left(\frac{2hw}{w^2 + h^2}\right) \\ l = \frac{w^2 + h^2}{h} \arcsin\left(\frac{2hw}{w^2 + h^2}\right) $$
For $h = 5$ and $w = 11$, $l = 24.915...$