Let $a \le b \le c$ be the sides of a triangle inscribed inside a fixed circle such that the vertices of the triangle are distributed uniformly on the circumference.
Question 1: Is it true that the probability that $ac > b^2$ is $\displaystyle \frac{1}{5}$. I ran a simulation by generating $1.75 \times 10^9$ triangle and counting the number of times $ac > b^2$. The experimental data seems to suggest that probability converges to about $0.2$.
Note: For any triangle with $a \le b \le c$, the triangle inequality implies $b < a+c < 3b$. Now the condition $ac > b^2$ implies that $2b < a+c < 3b$; here the lower bound follows from AM-GM inequality. Hence all triangles for which $b < a+c < 2b$ are ruled out. For our problem, the condition $2b < a+c < 3b$ is necessary but not sufficient.
Update: Changed the title in light of the comments and answer that relaxing the condition $a\le b \le c$ is easier to handle
Related question: If $(a,b,c)$ are the sides of a triangle and $x \ge 1$, what is the probability that $a+b > cx$?
Best Answer
Assume that the circle is the unit circle centred at the origin, and the vertices of the triangle are:
$A(\cos(-Y),\sin(-Y))$ where $0\le Y\le2\pi$
$B(1,0)$
$C(\cos X,\sin X)$ where $0\le X\le2\pi$
Relax the requirement that $a \le b \le c$, and let:
$a=BC=2\sin\left(\frac{X}{2}\right)$
$b=AC=\left|2\sin\left(\frac{2\pi-X-Y}{2}\right)\right|=\left|2\sin\left(\frac{X+Y}{2}\right)\right|$
$c=AB=2\sin\left(\frac{Y}{2}\right)$
$\therefore P[ac>b^2]=P\left[\sin\left(\frac{X}{2}\right)\sin\left(\frac{Y}{2}\right)>\sin^2\left(\frac{X+Y}{2}\right)\right]$ where $0\le X\le2\pi$ and $0\le Y\le2\pi$
This probability is the ratio of the area of the shaded region to the area of the square in the graph below.
Rotate these regions $45^\circ$ clockwise about the origin, then shrink by factor $\frac{1}{\sqrt2}$, then translate left $\pi$ units, by letting $X=x+\pi-y$ and $Y=x+\pi+y$.
$\begin{align} P[ac>b^2]&=P\left[\sin\left(\frac{x+\pi-y}{2}\right)\sin\left(\frac{x+\pi+y}{2}\right)>\sin^2(x+\pi)\right]\\ &=P\left[\cos(x+\pi)-\cos y<-2\sin^2(x+\pi)\right]\text{ using sum to product identity}\\ &=P\left[-\cos x-\cos y<-2\sin^2 x\right]\\ &=P\left[-\arccos(2\sin^2x-\cos x)<y<\arccos(2\sin^2x-\cos x)\right]\\ &=\dfrac{\int_0^{\pi/3}\arccos(2\sin^2x-\cos x)\mathrm dx}{\frac{\pi^2}{2}} \end{align}$
Numerical evidence suggests that the integral equals $\frac{\pi^2}{5}$. (I've posted this integral as another question.) If that's true, then the probability is $\frac25$.
If the probability without requiring $a \le b \le c$ is $\frac25$ , it follows that the probability with requiring $a \le b \le c$ is $\frac15$, as @joriki explained in the comments.
Update:
The integral has been shown to equal $\frac{\pi^2}{5}$, thus showing that the answer to the OP is indeed $1/5$.
The simplicity of the answer, $1/5$, suggests that there might be a more intuitive solution, but given the amount of attention the OP has received, an intuitive solution seems to be quite elusive. We might have to chalk this one up as another probability question with a simple answer but no intuitive explanation. (Other examples of such probability questions are here and here.)