Consider an arbitrary triangle PQR. Excluding the trivial case of an equilateral triangle, either one or two of the angles are at least 60 degrees, and wlog let P be the minimal example of these. I.e. P is the smallest angle of at least 60 degrees, and Q is either the largest angle or the second largest after P. Then the largest enclosed equilateral triangle (call it E) has one vertex at P.
If Q < 60 (i.e. P is the only angle >=60), then both other vertices are on QR.
If Q > 120 - P/2, then the second vertex lies on QR, at the intersection with a line drawn at an angle of 60 degrees from PR.
Otherwise the second vertex is at Q.
Motivation: The interesting case to consider is a triangle of angles 62, 89, 29 degrees. (Actually this is almost exactly the OP's triangle ABC above.
If Q<90, there will be a local maximum E based on P and Q. The diagram shows the perpendicular dropped from P to QR, and shows that there will be an equal E if it is possible to rotate this about P so that the vertex on QR is the same distance on the opposite side of the perpendicular. In this case, since angle Q is 89 degrees, we need to be able to rotate E through 2 degrees, which is exactly possible. Of course if angle Q is more than 90 degrees, rotation will always increase the size of E.
This is a sketch of an answer; it depends on proving that one vertex of E is at a vertex of PQR, and a messy set of cases for optimisation. But I hope I have captured the distinction between the two cases illustrated by (62, 89, 29).
If I've understood the question correctly, the setup is impossible: it assumes that the distances between two opposite vertices of the hexagon, and two opposite sides of the hexagon, are equal to the side length of the square.
Imagining the hexagon inscribed in a circle makes it obvious that opposite vertices are further apart than opposite sides, so the two distances can't be equal.
So what you've got isn't a regular hexagon, and it's refusing to behave like one.
Best Answer
Intuition is that if no side of the square lies along a side of the triangle then there is room left to "jiggle" the square until one of its sides lines up with a side of the triangle, then "inflate" it larger. Therefore the maximal square must have one side on a side of the triangle, and the other two vertices on the other sides of the triangle.
Proving that rigorously is not trivial, though. John M. Sullivan's paper Polygon in Triangle: Generalizing a Theorem of Post proves the following more general statements for arbitrary convex polygons inside a triangle, not just squares: