geometry
Consider a unit square. What is the largest, and smallest, equilateral triangle with vertices touching the sides?
Clearly the sides have to be larger than one, and it looks like the biggest would be with side $\frac{1}{\cos(\pi/12)}=\sqrt{\frac{4}{\sqrt{3}+2}}$, by placing one vertex in a corner of the square, but
You could call it an Ikea type problem, how to fit your box into the back of the van..
Let the unknown radius be $r$. Then the vertical distance between the centres of $B,C$ is $r-9$, hence by Pythagoras their horizontal distance is $\sqrt{(9+r)^2-(r-9)^2}=6\sqrt{r}$. Likewise, the vertical distance between centres of $A,C$ is $r-4$ and therefore the horizontal distance is $\sqrt{(4+r)^2-(r-4)^2}=4\sqrt r$. Thus the horizontal distance between centres of $A,B$ is $6\sqrt r-4\sqrt r=2\sqrt r$ and the vertical distance is $\sqrt{13^2-4r}$, but it is also $2r-13$. We conclude $$13^2-4r=(2r-13)^2 $$ and so ($r=0$ or) $$r=12. $$
Consider an arbitrary triangle PQR. Excluding the trivial case of an equilateral triangle, either one or two of the angles are at least 60 degrees, and wlog let P be the minimal example of these. I.e. P is the smallest angle of at least 60 degrees, and Q is either the largest angle or the second largest after P. Then the largest enclosed equilateral triangle (call it E) has one vertex at P.
If Q < 60 (i.e. P is the only angle >=60), then both other vertices are on QR.
If Q > 120 - P/2, then the second vertex lies on QR, at the intersection with a line drawn at an angle of 60 degrees from PR.
Otherwise the second vertex is at Q.
Motivation: The interesting case to consider is a triangle of angles 62, 89, 29 degrees. (Actually this is almost exactly the OP's triangle ABC above.
If Q<90, there will be a local maximum E based on P and Q. The diagram shows the perpendicular dropped from P to QR, and shows that there will be an equal E if it is possible to rotate this about P so that the vertex on QR is the same distance on the opposite side of the perpendicular. In this case, since angle Q is 89 degrees, we need to be able to rotate E through 2 degrees, which is exactly possible. Of course if angle Q is more than 90 degrees, rotation will always increase the size of E.
This is a sketch of an answer; it depends on proving that one vertex of E is at a vertex of PQR, and a messy set of cases for optimisation. But I hope I have captured the distinction between the two cases illustrated by (62, 89, 29).
Best Answer
The whole set of equilateral triangles inscribed in a unit square $ABCD$ can be generated through the following procedure:
By symmetry it is pretty clear that the minimum area of the inscribed equilateral triangle is attained when $P$ is the midpoint of $AB$ and the maximum area is attained when a vertex of the inscribed equilateral triangle is a vertex of the original square. In the former case the mentioned triangle has unit side length, in the latter the side length equals $\frac{1}{\cos 15^\circ}=\sqrt{6}-\sqrt{2}$.
It is interesting to point out that if $P$ is a vertex of an inscribed equilateral triangle close to the midpoint of $AB$, the opposite sides goes through a fixed point $Q$, and the area of the inscribed equilateral triangle just equals $\frac{1}{\sqrt{3}} PQ^2$. Here it is an animation, too: