The last paragraph at the Mathworld piece on equilateral triangles gives the answer, and cites Madachy, J. S. Madachy's Mathematical Recreations. New York: Dover, pp. 115 and 129-131, 1979.
EDIT (in response to request from Taha Akbari for more detail): Let the square have horizontal and vertical sides. Consider an equilateral triangle with one vertex at the lower left corner, $A$, of the square, and one vertex at the upper left corner, $B$, of the square, and the third vertex, $Z$, inside the square. Now consider moving the triangle vertex at $B$ to the right, toward the upper right corner, $C$, of the square, while moving $Z$ so as to keep the triangle equilateral. This increases the area of the triangle, since it increases the length of the side of the triangle, since the second vertex, $X$, of the triangle is moving away from the first vertex of the triangle.
Eventually, the triangle vertex $Z$ lies on the right side of the square, and you can't move $X$ any farther right without pushing $Z$ outside the square, so you've made the triangle as large as possible. Now the question is, why are the angles $BAX$ and $ZAD$ 15 degrees (where $D$ is the lower right corner of the square)?
The triangles $BAX$ and $ZAD$ are congruent, since $BA=AD$, $AX=AZ$, and the angles at $B$ and $D$ are equal. So the angles $BAX$ and $ZAD$ are equal. But they, together with the 60 degree angle $XAZ$, add up to the 90 degree angle $BAD$. So, they measure 15 degrees.
Consider an arbitrary triangle PQR. Excluding the trivial case of an equilateral triangle, either one or two of the angles are at least 60 degrees, and wlog let P be the minimal example of these. I.e. P is the smallest angle of at least 60 degrees, and Q is either the largest angle or the second largest after P. Then the largest enclosed equilateral triangle (call it E) has one vertex at P.
If Q < 60 (i.e. P is the only angle >=60), then both other vertices are on QR.
If Q > 120 - P/2, then the second vertex lies on QR, at the intersection with a line drawn at an angle of 60 degrees from PR.
Otherwise the second vertex is at Q.
Motivation: The interesting case to consider is a triangle of angles 62, 89, 29 degrees. (Actually this is almost exactly the OP's triangle ABC above.
If Q<90, there will be a local maximum E based on P and Q. The diagram shows the perpendicular dropped from P to QR, and shows that there will be an equal E if it is possible to rotate this about P so that the vertex on QR is the same distance on the opposite side of the perpendicular. In this case, since angle Q is 89 degrees, we need to be able to rotate E through 2 degrees, which is exactly possible. Of course if angle Q is more than 90 degrees, rotation will always increase the size of E.
This is a sketch of an answer; it depends on proving that one vertex of E is at a vertex of PQR, and a messy set of cases for optimisation. But I hope I have captured the distinction between the two cases illustrated by (62, 89, 29).
Best Answer
Since the triangle is equilateral, the lengths of the sides must be equal; that is, $$ 2x^2=x^2-2x+2\tag1 $$ The solutions to $(1)$ are $x=-1\pm\sqrt3$. The only one that would apply to the case pictured above would be $-1+\sqrt3$. This would make the area of the triangle $$ x-\tfrac12x^2=2\sqrt3-3\tag2 $$