NB: I am presenting this answer as a frame challenge. The primary motivation behind this answer is to make more permanent some of the comments left in response to the question and other answers, as well as to incorporate some ideas from a now deleted answer.
The Importance of Definitions
Mathematics is a human endeavor. The words we use to describe mathematical ideas are a human invention, hence it is important to recognize that different humans might use the same word to describe different ideas, or different words to describe the same idea. When one is trying to understand a mathematical idea presented by another, it is important to understand the presenter's definitions. From the definitions, further deductions may be made.
For example, in the question above, we have the definition:
Definition: An isosceles triangle is a triangle with at least two congruent sides.
An equilateral triangle has three congruent sides, and three is "at least" two. Therefore, per this definition, every equilateral triangle must be isosceles.
However, there are authors who give a different definition of isosceles triangles. Joel Reyes Noche notes that many primary school instructors define an isosceles triangle to be one with exactly two congruent sides. Indeed, this is the definition given by Euclid himself!:
- Further, of trilateral figures, an equilateral triangle is that which has its three sides equal, an isosceles triangle that which has two of its sides alone equal, and a scalene triangle that which has its three sides unequal. [Euclid's Elements, as translated by Thomas Heath]
Per this definition, no isosceles triangle is equilateral, and no equilateral triangle is isosceles.
This answer may not be the simplest, but it is straightforward.
I particularized the problem by making one side of the large equilateral triangle the segment between the points $(-8,0)$ and $(8,0)$. Then simple geometry tells us the third vertex is at $(8\sqrt{3},0)$, the circumcenter of the triangle is at $A(0,\dfrac{8\sqrt{3}}3)$, and the radius of the circumcircle is $\dfrac{16\sqrt{3}}3$.
The equation of the circumcircle is then
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\left(\frac{16\sqrt{3}}3\right)^2$$
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\frac{256}3$$
The one side of the small equilateral triangle, $\overline{FH}$, is on the line $y=-\sqrt{3}x$. That gives us two equations in two unknowns for the coordinates of point $H$ which is on both the circle and the line. Substitute the expression for $y$ in the linear equation into the quadratic equation and we get a quadratic equation for $x$:
$$x^2+\left(-\sqrt 3x-\frac{8\sqrt 3}3\right)^2=\frac{256}3$$
$$4x^2+16x-64=0$$
$$x^2+4x-16=0$$
Solving this gives us this positive value for $x$:
$$x=2\sqrt 5-2$$
The side of the small equilateral triangle is twice the $x$-coordinate of point $H$, so the side of the triangle is
$$4\sqrt 5-4$$
The final answer to your problem, given the side is $a\sqrt b-c$, is
$$a=4, \quad b=5, \quad c=4$$
I checked this answer numerically with Geogebra, the source of my diagram above, and my answer checks.
Best Answer
Let's name vertices, sides and angles of the triangle as in figure below, with $c\le a\le b$ and consequently $\gamma\le \alpha\le \beta$. Let then $PQR$ be the inscribed equilateral triangle, with $P\in AC$, $Q\in AB$ and $\theta=\angle AQP$. Applying the sine rule to triangles $APQ$, $BQR$, it is not difficult to find the side $l=PQ$ as a function of $\theta$: $$ l={c\sin\alpha\sin\beta\over \sin\alpha\sin(60°+\theta-\beta) +\sin\beta\sin(\alpha+\theta)}. $$
As in the isosceles case, $dl/d\theta=0$ when $l$ reaches its minimum value. The maximum is reached for a limiting value of $\theta$. There are two cases to discuss.
If $\alpha>60°$, the limiting values of $\theta$ are reached when $Q=B$ (that is $\theta =\beta-60°$) or $Q=A$ (that is $\theta =180°-\alpha$), corresponding to the values of $l$: $$ l_1={c\sin\alpha\over\sin(\alpha+\beta-60°)}, \quad l_2={c\sin\beta\over\sin(\alpha+\beta-60°)}. $$ In this case $l_2>l_1$, hence the maximum of $l$ is given by $l_2$.
If $\alpha\le60°$, the limiting values of $\theta$ are reached when $Q=B$ (that is $\theta=\beta-60°$) or $R=B$ (that is $\theta =120°$). The first value corresponds to $l_1$ as written above, while the second value gives $$ l_2'={c\sin\alpha\over\cos(\alpha-30°)}. $$ In this case the maximum of $l$ is the greater between $l_1$ and $l_2'$.