Let $a$, $b$, $c$ be the lengths of the sides opposite $A$, $B$, $C$, and let points $D$, $E$, $F$ lie on the sides opposite $A$, $B$, $C$. (The latter is slightly different than in the original question). Let $r = |DE| = |EF| = |FD|$ be the length of the sides of the equilateral triangle; let $s = |AF| = |BD| = |CE|$ be the lengths of the congruent sub-segments of the sides.
The Law of Cosines for each $r$-side in $\triangle AEF$, $\triangle BFD$, $\triangle CDE$ gives
$$\begin{align}
r^2 = s^2 + (b-s)^2 - 2 s (b-s) \cos A \\
r^2 = s^2 + (c-s)^2 - 2 s (c-s) \cos B \\
r^2 = s^2 + (a-s)^2 - 2 s (a-s) \cos C
\end{align}$$
Setting the right-hand sides equal and subtracting $2s^2$ yields:
$$b^2-2bs-2s(b-s)\cos A = c^2-2cs-2s(c-s)\cos B = a^2 - 2 a s - 2 s(a-s) \cos C$$
Using the Law of Cosines (in $\triangle ABC$) to re-write each of $\cos A$, $\cos B$, $\cos C$, and then simplifying a bit, gives
$$\begin{align}
&\phantom{=}\;a\left( s^2\left(-a^2+b^2+c^2\right)+ s b\left(a^2-\left(b+c\right)^2\right)+b^3 c\right) \\
&= b\left( s^2\left(\phantom{-}a^2-b^2+c^2\right)+ s c\left(b^2-\left(c+a\right)^2\right)+c^3 a\right) \\
&= c\left( s^2\left(\phantom{-}a^2+b^2-c^2\right)+ s a\left(c^2-\left(a+b\right)^2\right)+a^3 b\right)
\end{align}$$
We can break the triple equality into a system of two quadratic equations in the parameter $s$. Eliminating $s$ gives a polynomial equation, $p(a,b,c)=0$; the factors of $p$ provide these equations:
$$\begin{align}
abc (a+b+c) &= 0 &(1) \\
a^2 b\left(a-b\right)+b^2c\left(b-c\right)+c^2 a\left(c-a\right) &= 0 & (2) \\
a b\left(a-b\right)\left(a\left(b-c\right)\left(a^2-b^2\right)-b^2c^2\right) \\
+b c\left(b-c\right)\left(b\left(c-a\right)\left(b^2-c^2\right)-c^2a^2\right) \\
+c a\left(c-a\right)\left(c\left(a-b\right)\left(c^2-a^2\right)-a^2b^2\right) &= 0 & (3)
\end{align}$$
Solutions in each case correspond to ostensible solutions to the problem.
We can ignore (1), which has only trivial solutions. I believe (2)'s only non-trivial solutions (with the Triangle Inequality in play) require $a=b=c$, but I need to double-check this.
As for (3), clearly $a=b=c$ works; and, indeed, if (say) $a=b$, then $a=b=c$. So any non-equilateral solutions would need to be strictly scalene; it turns out that (3) admits scalene solutions.
Below is Mathematica's plot of (3), with $c=1$; the horizontal axis is $a$, and the vertical is $b$. The lines are $a+b=c$ and $b+c=a$ and $c+a=b$, which bound the region of feasibility according to the Triangle Inequality. The point $(1,1)$ seems to have been missed by the implicit plotter, but lots of other points appear in blue. (I believe there are even more. Zooms show an island near the $a+b=c$ border, for instance.)
Here's the figure for $a=8$, $b=8.58667$, $c=1$:
Equilateral $\triangle DEF$ extends beyond the confines of $\triangle ABC$. This may be why @Brian was having difficulty ruling-out non-equilateral $\triangle ABC$s.
I suspect equation (3) can be expressed in a way that's more illuminating; it's not clear to me, for instance, whether every solution to (3) corresponds to an "external" triangle. There's probably a whole other approach to this problem that makes everything obvious, but this is the best I have at the moment.
Best Answer
Well, since the distances form a Pythagorean triple the choice was not that random. You are on the right track and reflection is a great idea, but you need to take it a step further.
Check that in the (imperfect) drawing below $\triangle RBM$, $\triangle AMQ$, $\triangle MPC$ are equilateral, since they each have two equal sides enclosing angles of $\frac{\pi}{3}$. Furthermore, $S_{\triangle ARM}=S_{\triangle QMC}=S_{\triangle MBP}$ each having sides of length 3,4,5 respectively (sometimes known as the Egyptian triangle as the ancient Egyptians are said to have known the method of constructing a right angle by marking 12 equal segments on the rope and tying it on the poles to form a triangle; all this long before the Pythagoras' theorem was conceived)
By construction the area of the entire polygon $ARBPCQ$ is $2S_{\triangle ABC}$
On the other hand
$$ARBPCQ= S_{\triangle AMQ}+S_{\triangle MPC}+S_{\triangle RBM}+3S_{\triangle ARM}\\=\frac{3^2\sqrt{3}}{4}+\frac{4^2\sqrt{3}}{4}+\frac{5^2\sqrt{3}}{4}+3\frac{1}{2}\cdot 3\cdot 4 = 18+\frac{25}{2}\sqrt{3}$$
Hence
$$S_{\triangle ABC}= 9+\frac{25\sqrt{3}}{4}$$