Given a scalene triangle $ABC$ and an inscribed equilateral triangle whose vertices lie on different sides of $\triangle ABC$, what is the maximal ratio of the area of the equilateral triangle to that of the original triangle?
I would expect an answer as a ratio of polynomials in sines and cosines of the angles of $\triangle ABC$. I got such an expression via a clunky unsymmetrical method, but it but it was so messy that I gave up trying to simplify it. However, a more intelligent method might well yield a formula of reasonable length.
Best Answer
We start with the process of solving the problem of "inserting" an equilateral triangle in the given triangle. We select the side of the triangle in front the largest angle to create the locus the smaller angles of the equilateral triangle.
When you have (the drawing on the right) the two circles with centers $U$ (radius $r_1$) and $S$ (radius $r_2$), you select the side of the equilateral triangle $GH$, in the larger circle (center $S$ in the drawing), you stretch it to intersect the smaller circle (not shown) and then you draw the line through this intersection and the third vertex of the equilateral triangle intersecting the other circle.
The triangle you created is the smallest triangle subscribing the equilateral triangle and is similar to the original triangle. You scale up the equilateral triangle by the similarity factor to get the largest equilateral triangle.
Note that the largest equilateral triangle is sharing a side with the triangle in which it is inscribed and that the $60^\circ$ angle of the equilateral triangle is included in the largest angle of the original triangle.
This means that to draw the largest equilateral triangle you do not need to go through the lengthy process; you draw a $60^\circ$ angle on the larger side of the largest angle of the triangle and from that vertex to the intersection with the opposite side is the largest equilateral triangle.
Let me know if you have a question.