Let the ellipse be in standard position, with (fraction-free) equation
$$b^2 x^2 + a^2 y^2 = a^2 b^2$$
Let our equilateral triangle have circumcenter $(p,q)$ and circumradius $r$. Note that maximizing the area of the triangle is equivalent to maximizing $r$.
For some angle $\theta$ ---actually, for three choices of $\theta$--- the vertices of the triangle have coordinates
$$
(p,q) + r \; \mathrm{cis}\theta \qquad (p,q)+r\;\mathrm{cis}\left(\theta+120^{\circ}\right) \qquad (p,q) + r\;\mathrm{cis}\left(\theta-120^{\circ}\right)
$$
where I abuse the notation "$\mathrm{cis}\theta$" to indicate the vector $(\cos\theta,\sin\theta)$.
Substituting these coordinates into the ellipse equation gives a system of three equations in four parameters $p$, $q$, $r$, $\theta$. I used Mathematica's Resultant[]
function to help me eliminate $r$ and $\theta$, arriving at a huge polynomial equation in $p$ and $q$. One factor of the polynomial gives rise to this equation:
$$p^2 b^2\left( a^2+3b^2 \right)^2 + q^2 a^2\left(3a^2+b^2\right)^2= a^2b^2\left(a^2-b^2\right)^2$$
This says that the family of circumcenters $(p,q)$ lie on their own ellipse! We can therefore write
$$p = \frac{a\left(a^2-b^2\right)}{a^2+3b^2}\cos\phi \qquad q = \frac{b\left(a^2-b^2\right)}{3a^2+b^2}\sin\phi$$
for some $\phi$. Back-substituting into the system of equations gives this formula for $r$:
$$r = \frac{4 a b \sqrt{a^2\left(a^2+3b^2\right)^2-\left(a-b\right)^3\left(a+b\right)^3 \cos^2\phi}}{\left(a^2 + 3 b^2\right)\left(3a^2+b^2\right)}$$
The maximum value, $R$, is attained when $\cos\phi = 0$, so
$$R := \frac{4a^2b}{3a^2+b^2}$$
The area of the maximal triangle is
$$\frac{3\sqrt{3}}{4}R^2 = \frac{12a^4 b^2\sqrt{3}}{\left(3a^2+b^2\right)^2}$$
Perhaps-unsurprisingly, the corresponding triangles are centered horizontally within the ellipse, with a vertex at either the top or bottom of the minor axis.
Now, I should point out that my big $pq$ polynomial has other factors, namely, $p$ itself, $q$ itself, and a giant I'll call $f$.
One can verify that the cases $p=0$ and $q=0$ lead to the same results as above. (Specifically, they correspond to the respective cases $\cos\phi=0$ and $\sin\phi=0$.) Intuitively, if a circle's center lies on an axis of the ellipse, then the points of intersection with the ellipse have reflective symmetry over that axis. If there are four distinct points (or two, or none), then we cannot choose three to be the vertices of our equilateral triangle; consequently, there must be only three points of intersection, with one of them on the axis, serving as the point of tangency for the circle and ellipse.
As for the case $f=0$ ... I'll just irresponsibly call it extraneous. (The method of resultants tends to spawn such things.)
Let us look at $\triangle AED$ and $\triangle CED$:
- $\angle EDA = \angle EDC = 90\,^{\circ}$
- $DC= DA = 13$ cm
- $ED$ is common
So, by $SAS$ congruence, those two triangles are congruent, and hence $EC = AE$. Now let $BE$ be $x$. So, by Pythagoras Theorem, $(EC)^2=x^2+100$. But, $(EC)^2= (AE)^2 = (24-x)^2$. Therefore, we get an equation, $(24-x)^2= 576 + x^2 - 48x = x^2+100$. Which upon solving, we get, $x = \frac{119}{12}$ .
Back to $\triangle AED$, we see that $(ED)^2 = (24-x)^2 - 169$. So, we know $x$ and we can get $ED$ as well. The final touch, $$Area(BEDC) = Area(ABC) - Area(EDA) = \frac{24 \cdot 10}{2} - \frac{ED \cdot 13}{2}$$
Edit: Details of calculation:
$(ED)^2 = (24 - x)^2 - 169 = (24 - \frac{119}{12})^2 - 169 = (\frac{169}{12})^2 - 169 = 169(\frac{169}{144}- 1)= \frac{169 \cdot 25 }{144}$
Therefore, $ED = \sqrt{\frac{169 \cdot 25 }{144}} = \frac{13 \cdot 5 }{12}$.
So, $$Area(BEDC) = \frac{24 \cdot 10}{2} - \frac{ED \cdot 13}{2} = 120 - \frac{13 \cdot 13 \cdot 5}{24} = 120- \frac{845}{24} = \frac{2035}{24} = 84.791666...$$
Best Answer
Hint:
$\hspace{60pt}$
The red solid line is the height dropped onto the hypotenuse, i.e. $h = \frac{ab}{c}$ and the red dotted lines are of the same length. The green parallel lines are unnecessary, but might get you some intuitions.
Good luck! ;-)