This answer may not be the simplest, but it is straightforward.
I particularized the problem by making one side of the large equilateral triangle the segment between the points $(-8,0)$ and $(8,0)$. Then simple geometry tells us the third vertex is at $(8\sqrt{3},0)$, the circumcenter of the triangle is at $A(0,\dfrac{8\sqrt{3}}3)$, and the radius of the circumcircle is $\dfrac{16\sqrt{3}}3$.
The equation of the circumcircle is then
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\left(\frac{16\sqrt{3}}3\right)^2$$
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\frac{256}3$$
The one side of the small equilateral triangle, $\overline{FH}$, is on the line $y=-\sqrt{3}x$. That gives us two equations in two unknowns for the coordinates of point $H$ which is on both the circle and the line. Substitute the expression for $y$ in the linear equation into the quadratic equation and we get a quadratic equation for $x$:
$$x^2+\left(-\sqrt 3x-\frac{8\sqrt 3}3\right)^2=\frac{256}3$$
$$4x^2+16x-64=0$$
$$x^2+4x-16=0$$
Solving this gives us this positive value for $x$:
$$x=2\sqrt 5-2$$
The side of the small equilateral triangle is twice the $x$-coordinate of point $H$, so the side of the triangle is
$$4\sqrt 5-4$$
The final answer to your problem, given the side is $a\sqrt b-c$, is
$$a=4, \quad b=5, \quad c=4$$
I checked this answer numerically with Geogebra, the source of my diagram above, and my answer checks.
Best Answer
$\angle OBC = 30^\circ$, Hence $BM=r\cos 30^\circ = \frac{\sqrt3 r}{2}.$
Task given to you, find the length of $BC$.