Fix some circle (e.g. unit circle as a 1D variety), and a probability measure on it. The probability space we start with is the one of all $P=(P_0,P_1,P_2,P_3,P_4)$ tuples of five points on the circle. Denote this configuration space by $C(5)$. Measure on spaces (derived from probability spaces) will always have total mass zero. Let $\Pi=\Pi(P)=\Pi(P_0,P_1,P_2,P_3,P_4)$ be the "solid" area / set of points obtained in the following manner.
- Consider one of the points among the five, denote it by $A_0$, then define $A_1,A_2,A_3,A_4$ to be the points so that as sets $\{A_k\ : k\}$ and $\{P_k\ : k\}$ are equal, and so that the $A$-points come in cyclic trigonometric order (anti-clockwise) on the circle.
- Let $C_0(5)$ be the space of such $A$-configurations. If i correctly understood which probability should be computed, the event we want to measure is factorizing through this step.
- For $A=(A_0,A_1,A_2,A_3,A_4)$ in $C_0(5)$ let $\Pi(A)$ be the "star" delimited the following five half-spaces:
- the half-space delimited by $A_0A_2$ which does not contain $A_1$, but contains $A_3,A_4$
- and its "cyclic cousins", determined by $A_{k-1}A_{k+1}$, but not containing $A_k$. (Indices are taken modulo five.)
- It may be useful for a bonus to also have a notation for the "inner pentagon", which is the intersection of these half-spaces, we use $\Pi^*(A)$ for it.
- Set $\Pi(P)=\Pi(A)$, $\Pi^*(P)=\Pi^*(A)$.
- Because of rotational symmetry, we can work in the configuration space of tuples $A$ where $A_0$ is fixed.
- We will work now with the circle of length $2$, so that $A_0=0$, and $A_1,A_2,A_3,A_4$ are (identified with) points in this order in the interval $[0,2]$ (instead of $[0,2\pi]$, denote this new configuration space by $\Delta(5)$. (It is a part of a $4$-dimensional simplex.) It will be convenient to write $x_0,x_1,x_2,x_3,x_4)$ for an element in this space.
We will write below some integrals on configuration spaces, the mass will be omitted, (so we simply write $\int f$ as a "functional in $f$" instead of the measure theoretical notation $\int f\; d\mu$).
The passage from one configuration space to the other is done by the transport formula.
Then we can compute the probability $p$ that $O$ is inside $\Pi(P)$.
The complement, opposite event is the one where $O$ is in one of the regions denote by OP as "region $A,B,C,D,E$". Each such region has the same probability to contain $O$, and exactly one contains it. By rotational symmetry we compute the probability $q$ that $O$ is in the "wrong half-space" w.r.t. both lines $A_0A_2$ and $A_1A_3$, i.e. that the corresponding chords/segments have lengths $\ge 1$. The probability $q$ is thus:
$$
\begin{aligned}
q &
=\int_{C(5)}1_{O\text{ in region $A$ w.r.t. }\Pi(P)}
\\
&=\int_{C_0(5)}1_{O\text{ in region $A$ w.r.t. }\Pi(A)}
\\
&=\int_{\Delta(5)}1_{O\text{ in region $A$ w.r.t. }x}
\\
&=\int_{\Delta(5)}1_{\substack{
0=x_0\le x_1\le x_2\le x_3\le x_4\le 2\\
x_2-x_0 \ge 1\\
x_3-x_1 \ge 1
}}
\\
&=
\frac
{\operatorname{Volume}
\left(
\begin{split}
x\ :\ 0=x_0\le x_1\le x_2\le x_3\le x_4\le 2
\\
1\le x_2\\
0\le x_1\le 1\\
1+x_1, x_2\le x_3\le 2\\
x_3\le x_4\le 2
\end{split}
\right)}
{\operatorname{Volume}(x\ :\ 0=x_0\le x_1\le x_2\le x_3\le x_4\le 2)}
\\
&=
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_0^1dx_1
\int_{\max(1+x_1, x_2)}^2dx_3
\int_{x_3}^2dx_4
\\
&=
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_0^{x_2-1}dx_1
\int_{x_2}^2dx_3
\int_{x_3}^2dx_4
+
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_{x_2-1}^1dx_1
\int_{1+x_1}^2dx_3
\int_{x_3}^2dx_4
\\
&=
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_0^{x_2-1}dx_1
\int_{x_2}^2(2-x_3)\;dx_3
+
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_{x_2-1}^1dx_1
\int_{1+x_1}^2(2-x_3)\;dx_3
\\
&=
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_0^{x_2-1}\frac 12(2-x_2)^2\;dx_1
+
\frac 1{2^4/4!}
\int_1^2 dx_2
\int_{x_2-1}^1\frac 12(1-x_1)^2\;dx_1
\\
&=
\frac 1{2^4/4!}
\int_1^2 \frac 12(2-x_2)^2(x_2-1)\;dx_2
+
\frac 1{2^4/4!}
\int_1^2 \frac 16(2-x_2)^3\;dx_2
\\
&=
\frac 1{2^4/4!}
\left(\frac 1{24}+\frac 1{24}\right)
=
\frac{24}{16}\cdot \frac 2{24}
=\frac 2{16}=\frac 18\ .
\end{aligned}
$$
This leads to the value of the wanted probability $p$:
$$
\bbox[lightyellow]{\qquad
p=1-5q=1-\frac 58=\frac 38\ .\qquad}
$$
Bonus: Let us also compute the probability $p^*$ that $O$
is in the inner pentagon $\Pi^*$.
$$
\begin{aligned}
p^* &
=\int_{C(5)}1_{O\in \Pi(P)}
=\int_{C_0(5)}1_{O\in \Pi(A)}
=\int_{\Delta(5)}1_{\substack{
0=x_0\le x_1\le x_2\le x_3\le x_4\le 2\\
x_2-x_0 \le 1\\
x_3-x_1 \le 1\\
x_4-x_2 \le 1\\
2 - x_3 \le 1\\
2+x_1-x_4\le 1
}}
\\
&=
\frac
{\operatorname{Volume}\left(
\begin{split}
x\ :\ 0=x_0\le x_1\le x_2\le x_3\le x_4\le 2
\\
x_2\le 1\\
x_3-x_1 \le 1\\
x_4-x_2 \le 1\\
2 - x_3 \le 1\\
2+x_1-x_4\le 1
\end{split}
\right)}
{\operatorname{Volume}(x\ :\ 0=x_0\le x_1\le x_2\le x_3\le x_4\le 2)}
\\
&=
\frac 1{2^4/4!}
\int_{x_2\in[0,1]}dx_2
\int_{x_1\in[0,x_2]}dx_1
\int_{x_3\in[1,1+x_1]}dx_3
\int_{x_4\in[1+x_1,1+x_2]}dx_4
\\
&=
\frac 1{2^4/4!}
\int_{x_2\in[0,1]}dx_2
\int_{x_1\in[0,x_2]}dx_1
\cdot x_1(x_2-x_1)
=
\frac 1{2^4/4!}
\int_{x_2\in[0,1]}\frac 16x2^3\;dx_2
\\
&=\frac 1{2^4/4!}\cdot\frac 1{4!}=\frac 1{2^4}=\frac 1{16}\ .
\end{aligned}
$$
Simulation: We let $x$ be a random configuration with five components, one of them equal to zero, in the interval $[0,2]$, and check that no two consecutive cords have length bigger one. The code runs through the sage interpreter.
import random
TRIALS = 10**6
count = 0 # we count successes, so far none
x0 = 0
for trial in range(TRIALS):
ok = True
x1, x2, x3, x4 = [random.uniform(0, 2) for k in [1,2,3,4]]
x1, x2, x3, x4 = sorted([x1, x2, x3, x4])
pattern = [x2 - x0 > 1, x3 - x1 > 1, x4 - x2 > 1,
2 - x3 > 1, 2+x1 - x4 > 1]
for k in [0, 1, 2, 3, 4]:
if pattern[k] and pattern[(k + 1)%5]:
ok = False # and do not count
break
if ok:
count += 1
print(f"Monte-Carlo simulation: p ~ {count} / {TRIALS} = {count/TRIALS.n()}")
(The pattern is something like the sample [False, False, False, True, True], and shows True in the bad case of a chord bigger one. The decision to not count is when two cyclically consecutive True values occur.
And this time i've got:
Monte-Carlo simulation: p ~ 374674 / 1000000 = 0.374674000000000
sage: 3./8.
0.375000000000000
We can modify the do-not-count condition to also simulate $p^*$, by just testing that there is no True entry in the pattern, so ok = True not in pattern, a one liner.
import random
TRIALS = 10**6
count = 0 # we count successes, so far none
x0 = 0
for trial in range(TRIALS):
x1, x2, x3, x4 = sorted([random.uniform(0, 2) for k in [1,2,3,4]])
pattern = [x2 - x0 > 1, x3 - x1 > 1, x4 - x2 > 1,
2 - x3 > 1, 2+x1 - x4 > 1]
if True not in pattern:
count += 1
print(f"Monte-Carlo simulation: p* ~ {count} / {TRIALS} = {count/TRIALS.n()}")
And this time i've got:
Monte-Carlo simulation: p* ~ 62733 / 1000000 = 0.0627330000000000
sage: 1./16
0.0625000000000000
Which supports the computed result in the bonus part...
The moral is that "similar probabilities" (using possibly more points, and special patterns of the position of the center $O$ w.r.t. the sides and diagonals of the cyclically ordered polygon) are volumes of polyhedra (in more dimensions), and can be computed via Fubini as above.
Best Answer
Simulation with your sample-size of $n=10^7$ gave you an observed proportion $\hat p=0.4285833,$ so an approximate confidence interval for $p:=P(a^2<bc)$, with, say a $95\%$ "confidence level", is $$\hat p\pm 1.96\,\sqrt{\hat p(1-\hat p)\over n}=(0.4283, 0.4289)$$
Although $3/7=0.42857...$ does lie in this interval, the interval only esimates $p$ to about three digits of precision. Simulations using a larger sample-size suggest, with a very high degree of confidence/credibility, that $p\ne {3\over 7},$ the difference showing up in the fourth decimal place. Here's a picture of what I find with sample-size $10^{11}$:
The posterior distribution is $\text{Beta}(a,b)$, with $(a,b)=(n\hat p+{1\over 2}, n(1-\hat p)+{1\over 2}).$
The $100(1-\alpha)\%$ credibility interval is therefore $$(B_{\alpha\over 2},B_{1-{\alpha\over 2}})$$ where $B_q$ is the $q$-th quantile of the $\text{Beta}(a,b)$ distribution.
The $100(1-\alpha)\%$ confidence interval is $$\hat p\pm z_{\alpha\over 2}\,\sqrt{\hat p(1-\hat p)\over n}$$ where $z_q$ is the $q$-th quantile of the standard normal distribution.
With the sample-size this large the various approaches to confidence/credibility intervals for $p$ should all give practically the same results (assuming a non-informative prior distribution in the Bayesian methods); e.g., they agree to about six decimal places for a $99.9\%$ confidence/credibility interval, namely
$$0.428790 \pm 0.000005 = (0.428785, 0.428795).$$
(I found no OEIS entries for decimal expansions in that interval, so I can't guess at the exact value.)