This answer may not be the simplest, but it is straightforward.
I particularized the problem by making one side of the large equilateral triangle the segment between the points $(-8,0)$ and $(8,0)$. Then simple geometry tells us the third vertex is at $(8\sqrt{3},0)$, the circumcenter of the triangle is at $A(0,\dfrac{8\sqrt{3}}3)$, and the radius of the circumcircle is $\dfrac{16\sqrt{3}}3$.
The equation of the circumcircle is then
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\left(\frac{16\sqrt{3}}3\right)^2$$
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\frac{256}3$$
The one side of the small equilateral triangle, $\overline{FH}$, is on the line $y=-\sqrt{3}x$. That gives us two equations in two unknowns for the coordinates of point $H$ which is on both the circle and the line. Substitute the expression for $y$ in the linear equation into the quadratic equation and we get a quadratic equation for $x$:
$$x^2+\left(-\sqrt 3x-\frac{8\sqrt 3}3\right)^2=\frac{256}3$$
$$4x^2+16x-64=0$$
$$x^2+4x-16=0$$
Solving this gives us this positive value for $x$:
$$x=2\sqrt 5-2$$
The side of the small equilateral triangle is twice the $x$-coordinate of point $H$, so the side of the triangle is
$$4\sqrt 5-4$$
The final answer to your problem, given the side is $a\sqrt b-c$, is
$$a=4, \quad b=5, \quad c=4$$
I checked this answer numerically with Geogebra, the source of my diagram above, and my answer checks.
Let $CE=CF=x,BD=BF=y,AD=AE=z$.
We have $yz=11$. Since the triangle is a right triangle, you can have
$$(z+y)^2=(z+x)^2+(x+y)^2,$$
i.e.
$$x^2+xz+xy=11.$$
So, the area is
$$\frac{1}{2}(x+z)(x+y)=\frac{1}{2}(x^2+xy+xz+yz)=\frac{11+11}{2}$$
Best Answer
$$\alpha=\frac sR$$ $$\beta = \pi-\alpha=\pi-\frac sR$$
Draw $OD$ perpendicular to $AB$
$$c=AB=AD+DB=2AD$$ $$AD=R\sin\frac\beta2$$ $$c=2R\sin(\frac\pi2-\frac s{2R})=2R\cos\frac s{2R}$$