Let the ellipse be in standard position, with (fraction-free) equation
$$b^2 x^2 + a^2 y^2 = a^2 b^2$$
Let our equilateral triangle have circumcenter $(p,q)$ and circumradius $r$. Note that maximizing the area of the triangle is equivalent to maximizing $r$.
For some angle $\theta$ ---actually, for three choices of $\theta$--- the vertices of the triangle have coordinates
$$
(p,q) + r \; \mathrm{cis}\theta \qquad (p,q)+r\;\mathrm{cis}\left(\theta+120^{\circ}\right) \qquad (p,q) + r\;\mathrm{cis}\left(\theta-120^{\circ}\right)
$$
where I abuse the notation "$\mathrm{cis}\theta$" to indicate the vector $(\cos\theta,\sin\theta)$.
Substituting these coordinates into the ellipse equation gives a system of three equations in four parameters $p$, $q$, $r$, $\theta$. I used Mathematica's Resultant[] function to help me eliminate $r$ and $\theta$, arriving at a huge polynomial equation in $p$ and $q$. One factor of the polynomial gives rise to this equation:
$$p^2 b^2\left( a^2+3b^2 \right)^2 + q^2 a^2\left(3a^2+b^2\right)^2= a^2b^2\left(a^2-b^2\right)^2$$
This says that the family of circumcenters $(p,q)$ lie on their own ellipse! We can therefore write
$$p = \frac{a\left(a^2-b^2\right)}{a^2+3b^2}\cos\phi \qquad q = \frac{b\left(a^2-b^2\right)}{3a^2+b^2}\sin\phi$$
for some $\phi$. Back-substituting into the system of equations gives this formula for $r$:
$$r = \frac{4 a b \sqrt{a^2\left(a^2+3b^2\right)^2-\left(a-b\right)^3\left(a+b\right)^3 \cos^2\phi}}{\left(a^2 + 3 b^2\right)\left(3a^2+b^2\right)}$$
The maximum value, $R$, is attained when $\cos\phi = 0$, so
$$R := \frac{4a^2b}{3a^2+b^2}$$
The area of the maximal triangle is
$$\frac{3\sqrt{3}}{4}R^2 = \frac{12a^4 b^2\sqrt{3}}{\left(3a^2+b^2\right)^2}$$
Perhaps-unsurprisingly, the corresponding triangles are centered horizontally within the ellipse, with a vertex at either the top or bottom of the minor axis.
Now, I should point out that my big $pq$ polynomial has other factors, namely, $p$ itself, $q$ itself, and a giant I'll call $f$.
One can verify that the cases $p=0$ and $q=0$ lead to the same results as above. (Specifically, they correspond to the respective cases $\cos\phi=0$ and $\sin\phi=0$.) Intuitively, if a circle's center lies on an axis of the ellipse, then the points of intersection with the ellipse have reflective symmetry over that axis. If there are four distinct points (or two, or none), then we cannot choose three to be the vertices of our equilateral triangle; consequently, there must be only three points of intersection, with one of them on the axis, serving as the point of tangency for the circle and ellipse.
As for the case $f=0$ ... I'll just irresponsibly call it extraneous. (The method of resultants tends to spawn such things.)
One way to solve this problem is using affine transform, i.e transformation of the form:
$$\mathbb{R}^2 \ni (u,v)\quad\mapsto\quad (x,y) \in\mathbb{R}^2 \;\;\text{ with}\;\;
\begin{cases}x &= a u + b v + c\\y &= du + ev + f\end{cases}$$
where $a,b,c,d,e,f \in \mathbb{R}$ subject to the constraint $\Delta = ae - bd \ne 0$.
The key properties that we need to use are
- under an affine transform, the area of all geometric figure get scaled by same factor $|\Delta|$. As a result, the ratio of areas of any two geometric figures is invariant under such a transform.
- Given any two triangles or two circles/ellipses, there is an affine transform that send one to another.
- Under an affine transform, triangles map to triangles, circles/ellipses map to circles/ellipses.
Several geometric relationships are invariant under affine transform. e.g.
- if two curves are tangent to each other at some point, so does their images.
- if a point is the mid-point of another two points, so does their images.
This means given any triangle $T$, the problem of finding the largest ellipse (in sense of area) inside $T$ can be solved as follows:
- By 2. there is an affine transform $\phi$ which maps $T$ to an equilateral triangle $T'$.
- By 3. if $E$ is any circle or ellipse inside $T$, $E' = \phi(E)$ is a circle or ellipse inside $T'$.
- By 1.
$$\frac{\text{Area}(E)}{\text{Area}(T)} = \frac{\text{Area}(E')}{\text{Area}(T')}$$
As a result, the problem of finding the largest ellipse $E$ in $T$ is equivalent to one finding the largest ellipse $E'$ in $T'$.
- Intuitively, the largest circle/ellipse inside $T'$ is its incircle $C_{in}$. Assume this
is the case, the largest circle/ellipse inside $T$ will be $\phi^{-1}(C_{in})$.
To identify $E_{max} = \phi^{-1}(C_{in})$ geometrically, we use following fact:
The in-circle of an equilateral triangle is a circle tangent to the mid-points of its three sides.
By 4. we know $E_{max}$ will be an ellipse tangent to the mid points of the three sides.
Since each point tangent to a side is equivalent to specify two points on a curve
and 5 points is enough to completely specify a conic. We have more than enough to fully specify $E_{max}$.
Such an ellipse is called the Steiner inellipse of triangle $T$. For other interesting geometric properties of it, please consult the wiki pages and the references there.
There is one loose end to close, that is the assertion that
The in-circle $C_{in}$ is the largest ellipse inside $T'$.
To show this, we use 2. again. For any ellipse $E'$ in $T'$, there is always an affine transform $\psi$ which send $E'$ to the unit circle $C$. Let $T'' = \psi(T')$. Since
$$\frac{\text{Area}(E')}{\text{Area}(T')} = \frac{\text{Area}(C)}{\text{Area}(T'')}$$
The problem of finding the largest $E'$ inside $T'$ is equivalent to finding the smallest
triangle $T''$ that encloses the unit circle $C$.
If $T''$ is the smallest triangle, it is obvious its three sides are tangent to the unit circle $C$. Let $2\alpha, 2\beta, 2\gamma$ be the three angles of $T''$. It is clear they satisfy following constraint
$$0 < \alpha, \beta, \gamma \quad\text{ and }\quad
\alpha+\beta+\gamma = \frac{\pi}{2}\tag{*1}$$
and the area of $T''$ is given by
$$\text{Area}(T'') = \cot(\alpha) + \cot(\beta) + \cot(\gamma) \tag{*2}$$
Notice $\cot(\theta)$ is a strictly convex function in $\theta$. By Jensen's
inequality, the configuration that minimize $(*2)$ subject to constraint $(*1)$ are those
where $\alpha = \beta = \gamma$. This implies $T''$ is an equilateral triangle and hence $\psi$ is simply a scaling transform.
As a corollary, the largest ellipse $E'$ inside $T'$ is the in-circle.
Best Answer
By AM-GM $$S_{\Delta}=(a-x)y=(a-x)b\sqrt{1-\frac{x^2}{a^2}}=ab\left(1-\frac{x}{a}\right)\sqrt{1-\frac{x^2}{a^2}}=$$ $$=\frac{ab}{\sqrt3}\cdot\sqrt{\left(1-\frac{x}{a}\right)^3\left(3+\frac{3x}{a}\right)}\leq\frac{ab}{\sqrt3}\cdot\sqrt{\left(\frac{3\left(1-\frac{x}{a}\right)+3+\frac{3x}{a}}{4}\right)^4}=\frac{3\sqrt3ab}{4}.$$ The equality occurs for $1-\frac{x}{a}=3+\frac{3x}{a}$, which says that the equality indeed occurs,
which says that $\frac{3\sqrt3ab}{4}$ is a maximal value.
Done!