This answer may not be the simplest, but it is straightforward.
I particularized the problem by making one side of the large equilateral triangle the segment between the points $(-8,0)$ and $(8,0)$. Then simple geometry tells us the third vertex is at $(8\sqrt{3},0)$, the circumcenter of the triangle is at $A(0,\dfrac{8\sqrt{3}}3)$, and the radius of the circumcircle is $\dfrac{16\sqrt{3}}3$.
The equation of the circumcircle is then
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\left(\frac{16\sqrt{3}}3\right)^2$$
$$x^2+\left(y-\frac{8\sqrt{3}}3\right)^2=\frac{256}3$$
The one side of the small equilateral triangle, $\overline{FH}$, is on the line $y=-\sqrt{3}x$. That gives us two equations in two unknowns for the coordinates of point $H$ which is on both the circle and the line. Substitute the expression for $y$ in the linear equation into the quadratic equation and we get a quadratic equation for $x$:
$$x^2+\left(-\sqrt 3x-\frac{8\sqrt 3}3\right)^2=\frac{256}3$$
$$4x^2+16x-64=0$$
$$x^2+4x-16=0$$
Solving this gives us this positive value for $x$:
$$x=2\sqrt 5-2$$
The side of the small equilateral triangle is twice the $x$-coordinate of point $H$, so the side of the triangle is
$$4\sqrt 5-4$$
The final answer to your problem, given the side is $a\sqrt b-c$, is
$$a=4, \quad b=5, \quad c=4$$
I checked this answer numerically with Geogebra, the source of my diagram above, and my answer checks.
To restate your question more clearly, you are asking what kind of integer number $N$ can be the hypothenuse of a right-angled triangle, so that the other two sides are of integer length too.
The answer is well known: $N$ must be (the multiple of) the sum of two perfect squares: $N=a^2+b^2$ (or $N=k(a^2+b^2)$). If so, the other two sides (which in your problem are half the modified side and "the line drawn from the opposite vertex to the mid-point of the altered side") are given by $a^2-b^2$ and $2ab$ (multiplied by $k$ if needed).
Example: $5=2^2+1^2$, $3=2^2-1^2$, $4=2\cdot(2\cdot1)$.
Best Answer
Sorry for not typing! There is an easy way to find $k$ if we use some elementary trigonometry!