Repeat your reasoning starting with this form of triangular inequalities:
$$
|a-b|<c,\quad |b-c|<a,\quad |c-a|<b,
$$
that is:
$$
(a-b)^2<c^2,\quad (b-c)^2<a^2,\quad (c-a)^2<b^2.
$$
Assume that the circle is the unit circle centred at the origin, and the vertices of the triangle are:
$A(\cos(-Y),\sin(-Y))$ where $0\le Y\le2\pi$
$B(1,0)$
$C(\cos X,\sin X)$ where $0\le X\le2\pi$
Relax the requirement that $a \le b \le c$, and let:
$a=BC=2\sin\left(\frac{X}{2}\right)$
$b=AC=\left|2\sin\left(\frac{2\pi-X-Y}{2}\right)\right|=\left|2\sin\left(\frac{X+Y}{2}\right)\right|$
$c=AB=2\sin\left(\frac{Y}{2}\right)$
$\therefore P[ac>b^2]=P\left[\sin\left(\frac{X}{2}\right)\sin\left(\frac{Y}{2}\right)>\sin^2\left(\frac{X+Y}{2}\right)\right]$ where $0\le X\le2\pi$ and $0\le Y\le2\pi$
This probability is the ratio of the area of the shaded region to the area of the square in the graph below.
Rotate these regions $45^\circ$ clockwise about the origin, then shrink by factor $\frac{1}{\sqrt2}$, then translate left $\pi$ units, by letting $X=x+\pi-y$ and $Y=x+\pi+y$.
$\begin{align}
P[ac>b^2]&=P\left[\sin\left(\frac{x+\pi-y}{2}\right)\sin\left(\frac{x+\pi+y}{2}\right)>\sin^2(x+\pi)\right]\\
&=P\left[\cos(x+\pi)-\cos y<-2\sin^2(x+\pi)\right]\text{ using sum to product identity}\\
&=P\left[-\cos x-\cos y<-2\sin^2 x\right]\\
&=P\left[-\arccos(2\sin^2x-\cos x)<y<\arccos(2\sin^2x-\cos x)\right]\\
&=\dfrac{\int_0^{\pi/3}\arccos(2\sin^2x-\cos x)\mathrm dx}{\frac{\pi^2}{2}}
\end{align}$
Numerical evidence suggests that the integral equals $\frac{\pi^2}{5}$. (I've posted this integral as another question.) If that's true, then the probability is $\frac25$.
If the probability without requiring $a \le b \le c$ is $\frac25$ , it follows that the probability with requiring $a \le b \le c$ is $\frac15$, as @joriki explained in the comments.
Update:
The integral has been shown to equal $\frac{\pi^2}{5}$, thus showing that the answer to the OP is indeed $1/5$.
The simplicity of the answer, $1/5$, suggests that there might be a more intuitive solution, but given the amount of attention the OP has received, an intuitive solution seems to be quite elusive. We might have to chalk this one up as another probability question with a simple answer but no intuitive explanation. (Other examples of such probability questions are here and here.)
Best Answer
Assume that the circle is centred at the origin, and the vertices of the triangle are:
$A(\cos(-2Y),\sin(-2Y))$ where $0\le Y\le\pi$
$B(\cos(2X),\sin(2X))$ where $0\le X\le\pi$
$C(1,0)$
Let:
$a=BC=2\sin X$
$b=AC=2\sin Y$
$c=AB=\left|2\sin\left(\frac{2\pi-2X-2Y}{2}\right)\right|=|2\sin(X+Y)|$
$P\left[\frac1a+\frac1b\ge\frac1c\right]=1-P\left[\frac1a+\frac1b<\frac1c\right]$
$P\left[\frac1a+\frac1b<\frac1c\right]=P\left[\frac{1}{\sin X}+\frac{1}{\sin Y}<\frac{1}{|\sin(X+Y)|}\right]$
This last probability is the ratio of the area of the shaded region to the area of the square in the graph below.
Rotate these regions $45^\circ$ clockwise about the origin, then shrink by factor $\frac{1}{\sqrt2}$, by letting $X=x-y$ and $Y=x+y$.
Using symmetry, we only need to consider the left half of the blue "diamond". Note that in the left half, $0<x<\pi/2$, so $|\sin(2x)|=\sin(2x)$.
$P\left[\frac{1}{\sin X}+\frac{1}{\sin Y}<\frac{1}{|\sin(X+Y)|}\right]$
$=P\left[\frac{1}{\sin(x-y)}+\frac{1}{\sin(x+y)}<\frac{1}{\sin(2x)}\right]$
Now we want to express the inequality as $-f(x)<y<f(x)$ for some $f(x)$.
$=P\left[\sin(2x)(\sin(x+y)+\sin(x-y))<\sin(x-y)\sin(x+y)\right]$
$=P\left[2\sin(2x)(\sin x)(\cos y)<(\sin^2x)(\cos^2y)-(\cos^2x)(\sin^2y)\right]$
$=P\left[2\sin(2x)(\sin x)(\cos y)<(\sin^2x)(\cos^2y)-(\cos^2x)(1-\cos^2y)\right]$
$=P\left[\cos^2y-2\sin(2x)(\sin x)(\cos y)-\cos^2x>0\right]$
Solving the quadratic in $\cos y$ gives
$=P\left[\cos y>(\cos x)\left(2\sin^2x+\sqrt{1+4\sin^4x}\right)\right]$
$=P\left[-f(x)<y<f(x)\right]$
where $\color{red}{f(x)=\arccos\left((\cos x)\left(2\sin^2x+\sqrt{1+4\sin^4x}\right)\right)}$
Note that $f(\arccos(1/4))=0$ and $f(\pi/2)=\pi/2$.
So we have $P\left[\frac1a+\frac1b\ge\frac1c\right]=1-\dfrac{\int_{\arccos(1/4)}^{\pi/2}f(x)\mathrm dx}{\frac12 \left(\frac{\pi}{2}\right)^2}$
Here it is shown that $\int_{\arccos(1/4)}^{\pi/2}f(x)\mathrm dx=\dfrac{\pi^2}{40}$.
$\therefore P\left[\frac1a+\frac1b\ge\frac1c\right]=1-\dfrac15=\dfrac45$.