Assume that the circle is the unit circle centred at the origin, and the vertices of the triangle are:
$A(\cos(-Y),\sin(-Y))$ where $0\le Y\le2\pi$
$B(1,0)$
$C(\cos X,\sin X)$ where $0\le X\le2\pi$
Relax the requirement that $a \le b \le c$, and let:
$a=BC=2\sin\left(\frac{X}{2}\right)$
$b=AC=\left|2\sin\left(\frac{2\pi-X-Y}{2}\right)\right|=\left|2\sin\left(\frac{X+Y}{2}\right)\right|$
$c=AB=2\sin\left(\frac{Y}{2}\right)$
$\therefore P[ac>b^2]=P\left[\sin\left(\frac{X}{2}\right)\sin\left(\frac{Y}{2}\right)>\sin^2\left(\frac{X+Y}{2}\right)\right]$ where $0\le X\le2\pi$ and $0\le Y\le2\pi$
This probability is the ratio of the area of the shaded region to the area of the square in the graph below.
Rotate these regions $45^\circ$ clockwise about the origin, then shrink by factor $\frac{1}{\sqrt2}$, then translate left $\pi$ units, by letting $X=x+\pi-y$ and $Y=x+\pi+y$.
$\begin{align}
P[ac>b^2]&=P\left[\sin\left(\frac{x+\pi-y}{2}\right)\sin\left(\frac{x+\pi+y}{2}\right)>\sin^2(x+\pi)\right]\\
&=P\left[\cos(x+\pi)-\cos y<-2\sin^2(x+\pi)\right]\text{ using sum to product identity}\\
&=P\left[-\cos x-\cos y<-2\sin^2 x\right]\\
&=P\left[-\arccos(2\sin^2x-\cos x)<y<\arccos(2\sin^2x-\cos x)\right]\\
&=\dfrac{\int_0^{\pi/3}\arccos(2\sin^2x-\cos x)\mathrm dx}{\frac{\pi^2}{2}}
\end{align}$
Numerical evidence suggests that the integral equals $\frac{\pi^2}{5}$. (I've posted this integral as another question.) If that's true, then the probability is $\frac25$.
If the probability without requiring $a \le b \le c$ is $\frac25$ , it follows that the probability with requiring $a \le b \le c$ is $\frac15$, as @joriki explained in the comments.
Update:
The integral has been shown to equal $\frac{\pi^2}{5}$, thus showing that the answer to the OP is indeed $1/5$.
The simplicity of the answer, $1/5$, suggests that there might be a more intuitive solution, but given the amount of attention the OP has received, an intuitive solution seems to be quite elusive. We might have to chalk this one up as another probability question with a simple answer but no intuitive explanation. (Other examples of such probability questions are here and here.)
The answer appears to be no. The four significant figures quoted in the question are correct, but the fifth figure for the actual probability is less than it would be for the Basel constant minus 1. To six figures: $0.644904$ from the calculation below versus $0.644934$ for $\pi^2/6-1$.
As noted in the comments, the problem is a bit wonky because a triangle has three sides and whether the proposed inequality holds for a given triangle may depend on which side is chosen as $c$. A Monte Carlo-type simulation reveals that picking the minimum side as $c$ always holds but picking the maximilum side fails almost $75$% of the time. To get a probability close to the claimed $\pi^2/6-1$ requires picking the side at random with equal probability ($1/3$) for each side of any triangle.
The problem is then equivalent to drawing a chord through the unit circle with its endpoints uniformly distributed around the circumference, then selecting a third point with the same uniform distribution to complete the triangle and designating the original chord as $c$.
Rendering the circle as $x^2+y^2=1$ in the coordinate plane, we can wlog assume the plane is rotated so that the chord is parallel to the $x$ axis and has a positive $y$ coordinate. Thus the chord lies on $y=\cos\theta$ with $0\le\theta\le\pi/2$. For any theta, we define the probability $p(\theta)$ that the chord will be long enough to satisfy $a+b>c^{3/c}$ when the third point is selected.
Clearly $c=2\sin\theta$ and the bounding value for $a+b$ would be
$2s(\theta)=c^{3/c}=(2\sin\theta)^{[3/(2\sin\theta)]}$ Eq. 1
where $s(\theta)$ is the semimajor axis of an ellipse with foci at the endpoints of the chord $(\pm\sin\theta,\cos\theta)$. Then the third point, uniformly distributed about the circumference, must fall outside this ellipse for the inequality to hold. Using standard methods for determining thr equation of an ellipse with major and minor axes parallel to the coordinate axes we obtain its equation
$\dfrac{x^2}{s(\theta)^2}+\dfrac{(y-\cos\theta)^2}{s(\theta)^2-\sin^2\theta}=1$
Plugging $x^2=1-y^2$ and solving the resulting quadratic equation for $y$ gives two roots corresponding to the $y$ coordinates where the ellipse intersects the circle, to wit:
$y_+=\dfrac{s(\theta)^2\cos(\theta)+\sqrt{s(\theta)^4\cos^2(\theta)-\sin^2\theta(2s(\theta)^2-s(\theta)^4-\sin^2\theta)}}{\sin^2\theta}$ Eq. 2
$y_-=\dfrac{s(\theta)^2\cos(\theta)-\sqrt{s(\theta)^4\cos^2(\theta)-\sin^2\theta(2s(\theta)^2-s(\theta)^4-\sin^2\theta)}}{\sin^2\theta}$ Eq. 3
Here $s(\theta)$ is as defined implicitly in Eq. 1.
There are now three cases:
Case 1: For $\theta\le\pi/6$, the quadratic equation has a nonpositive discriminant and the ellipse either does not exist or, in the degenerate case at $\theta=\pi/6$, is just the chord itself. We regard the entire circle as lying outside the ellipse (the chord endpoints carry zero probability), so for this case
$p(\theta)=1.$
Case 2: When $\theta$ is just slightly above $\pi/6$, the ellipse intersects the circle at four points and the outer portions lie above $y=y_+$ or below $y=y_-$. In this case
$p(\theta)=\dfrac{\arccos(y_+)+\arccos(-y_-)}{\pi},$
where $y_+$ and $y_-$ are given in Eqs. 2 and 3. This case involves only a narrow range of $\theta$ values, for at $\theta\approx 0.534462\approx\pi/6+0.010863$ the $y_+$ value hits $1$ and th upper region disappears. This leads to:
Case 3: When $\theta$ exceeds the approxinate bound given in Case 2, the quadratic equation still gives roots for $y_+$ and $y_-$, but now only the latter gives a real inverse cosine corresponding to the existence of a region for which $c$ meets the inequality constraint. So the probability function reduces to
$p(\theta)=\dfrac{\arccos(-y_+)}{\pi}$
So now we may integrate these cases to get the overall probability with $\theta$ uniformly distributed between $0$ and $\pi/2$:
$P=\dfrac{2}{\pi}\int_{\theta=0}^{\theta=\pi/2}p(\theta) d\theta$
$=\dfrac13+\dfrac{2}{\pi^2}\int_{\theta=\pi/6}^{\theta=\pi/2}([y_+\le1]\arccos(y_+)+\arccos(-y_-)) d\theta$
Given the complicated forms used to define $y_{\pm}$ in Eqs. 2 and 3 and $s(\theta)$ in Eq. 1, this integral can be expected to be nonelementary. For the interval from $\pi/6$ to $\pi/2$, trapezoidal integration was used with varying numbers $n$ of uniformly spaced divisions. This led to the following calculated probabilities (including the $1/3$ term from Case 1):
$n=200\implies 0.644890659$
$n=500\implies 0.644907762$
$n=1000\implies 0.644907059$
$n=2000\implies 0.644904958$
With a bounded derivative for the integrand, the error would be $O(1/n^2)$, but the boundaries between the cases introduce square-root singularities that impart $O(1/n^{3/2})$ error terms. Therefore the above results were fitted to a least regression of the form
$P(n)=A+B/n^{3/2}+C/n^2$
from which $A\approx 0.644904$ is rendered as the intercept corresponding to $n\to\infty$. The standard error of the regression is $5.1×10^{-7}$ and $r^2$ is $0.99867$. To the same accuracy $\pi^2/6-1$ would be $0.644934$.
Best Answer
I have a formula for the probability, which for large $x$ is roughly $$\frac4{\pi^2}(\frac2x+\frac2{9x^3}+\frac2{25x^5}+\cdots)$$ The series continues in the same form up to at least the $x^{-11}$ term. There is an integral form at the end of this answer.
Solution:
The angles in the triangle are all twice the corresponding angles at the centre of the circle. So the trio of angles $(\alpha,\beta,\gamma)$ is equally distributed over the flat region in 3space with vertices $(\pi,0, 0),(0,\pi,0),(0,0,\pi)$.
The area of a triangle with sides $a,b,c$ and angles $\alpha,\beta,\gamma$ is $$\Delta=\frac12ab\sin\gamma=\frac12ca\sin\beta=\frac12bc\sin\alpha$$
So for example $a=\frac{abc}{2\Delta}\sin\alpha$, and the ratio we want is $$\frac{a+b}c=\frac{\sin\alpha+\sin\beta}{\sin\gamma}\\=\frac{2\sin\frac{\alpha+\beta}2\cos\frac{\alpha-\beta}2}{\sin(\alpha+\beta)}\\ =\frac{\cos\frac{\alpha-\beta}2}{\cos\frac{\alpha+\beta}2}$$
The region of interest is $0\le\alpha,0\le \beta,\alpha+\beta\le\pi$. Change variables to $$\theta=\frac{\alpha+\beta}2,\phi=\frac{\alpha-\beta}2$$
The region is now $$0\le\theta\le\pi/2,\\-\theta\le\phi\le\theta$$ with area $\pi^2/4$.
The ratio of interest is greater than $x$ when $$\cos\theta\lt\frac{\cos\phi}x$$
For a given $\phi$, Theta ranges over an interval of length $$\pi/2-\arccos(\frac{\cos\phi}x)\\=\arcsin(\frac{\cos\phi}x)$$
Integrate that over the region to give $$\frac4{\pi^2}\int^{\pi/2}_{-\pi/2}\arcsin(\frac{\cos\phi}x)d\phi$$
I used a series expansion in arcsine $x+\frac{x^3}6+\frac{3x^5}{40}+\cdots$ to get the series at the top.