Let the ellipse be in standard position, with (fraction-free) equation
$$b^2 x^2 + a^2 y^2 = a^2 b^2$$
Let our equilateral triangle have circumcenter $(p,q)$ and circumradius $r$. Note that maximizing the area of the triangle is equivalent to maximizing $r$.
For some angle $\theta$ ---actually, for three choices of $\theta$--- the vertices of the triangle have coordinates
$$
(p,q) + r \; \mathrm{cis}\theta \qquad (p,q)+r\;\mathrm{cis}\left(\theta+120^{\circ}\right) \qquad (p,q) + r\;\mathrm{cis}\left(\theta-120^{\circ}\right)
$$
where I abuse the notation "$\mathrm{cis}\theta$" to indicate the vector $(\cos\theta,\sin\theta)$.
Substituting these coordinates into the ellipse equation gives a system of three equations in four parameters $p$, $q$, $r$, $\theta$. I used Mathematica's Resultant[] function to help me eliminate $r$ and $\theta$, arriving at a huge polynomial equation in $p$ and $q$. One factor of the polynomial gives rise to this equation:
$$p^2 b^2\left( a^2+3b^2 \right)^2 + q^2 a^2\left(3a^2+b^2\right)^2= a^2b^2\left(a^2-b^2\right)^2$$
This says that the family of circumcenters $(p,q)$ lie on their own ellipse! We can therefore write
$$p = \frac{a\left(a^2-b^2\right)}{a^2+3b^2}\cos\phi \qquad q = \frac{b\left(a^2-b^2\right)}{3a^2+b^2}\sin\phi$$
for some $\phi$. Back-substituting into the system of equations gives this formula for $r$:
$$r = \frac{4 a b \sqrt{a^2\left(a^2+3b^2\right)^2-\left(a-b\right)^3\left(a+b\right)^3 \cos^2\phi}}{\left(a^2 + 3 b^2\right)\left(3a^2+b^2\right)}$$
The maximum value, $R$, is attained when $\cos\phi = 0$, so
$$R := \frac{4a^2b}{3a^2+b^2}$$
The area of the maximal triangle is
$$\frac{3\sqrt{3}}{4}R^2 = \frac{12a^4 b^2\sqrt{3}}{\left(3a^2+b^2\right)^2}$$
Perhaps-unsurprisingly, the corresponding triangles are centered horizontally within the ellipse, with a vertex at either the top or bottom of the minor axis.
Now, I should point out that my big $pq$ polynomial has other factors, namely, $p$ itself, $q$ itself, and a giant I'll call $f$.
One can verify that the cases $p=0$ and $q=0$ lead to the same results as above. (Specifically, they correspond to the respective cases $\cos\phi=0$ and $\sin\phi=0$.) Intuitively, if a circle's center lies on an axis of the ellipse, then the points of intersection with the ellipse have reflective symmetry over that axis. If there are four distinct points (or two, or none), then we cannot choose three to be the vertices of our equilateral triangle; consequently, there must be only three points of intersection, with one of them on the axis, serving as the point of tangency for the circle and ellipse.
As for the case $f=0$ ... I'll just irresponsibly call it extraneous. (The method of resultants tends to spawn such things.)
The equation $$ax^2+2hxy+by^2=c$$ represents a conic rotated about the axis (depending on $a,b,h,c$) with centre $(0,0)$.
You can always rotate the axis to remove the $xy$ term. This makes it easy to find the focus, length of major axis, etc.
To remove $xy$ term without shifting origin, lets rotate the axis by an angle $\theta$.
Let $X,Y$ represent the new coordinate system. Then,
$$x=X\cos\theta-Y\sin\theta$$
$$y=X\sin\theta+Y\cos\theta$$
Substituting and letting coefficient of $XY=0$, you get $$\tan2\theta=\frac{2h}{a-b}$$
For example, lets find $e$ of the following equation (which represents an ellipse): $$4x^2+2y^2+2\sqrt{3}xy=1$$
Raotate the axis by $$\tan2\theta=\frac{2h}{a-b}=\sqrt{3}$$
$$\theta=\frac{\pi}{6}$$
Now, substitute $$x=X\frac{\sqrt{3}}{2}-Y\frac{1}{2}$$
$$y=Y\frac{\sqrt{3}}{2}+X\frac{1}{2}$$
You will get $$5X^2+Y^2=5$$
Now you can find eccentricity using the formula you specified. (Rotating the ellipse or hyperbola will not change the eccentricity and the length of major\minor axis.
This is the graph for the two equations:
(Black represents the new ellipse)
Best Answer
Let the ellipse $$\frac{x^2}{a^2}+\frac{(y-b)^2}{b^2}=1\quad(1)$$ and the semicircle $$y=\sqrt{1-x^2}\quad(2)$$ The ellipse is clearly tangent to $x$-axis.
If we substitute the semicircle equation in the ellipse equation, we get: $$(a^2-b^2)y^2-2a^2by+b^2=0 \quad(3)$$ As the semicircle and the ellipse are tangent the discriminant of equation $(3)$ must be zero. Therefore $$b=\sqrt{a^2(1-a^2)} \quad(4)$$ But we know that the ellipse area can be calculated by: $$A=\pi a b \quad(5)$$ From $(4)$ and $(5)$, we get: $$A= \pi \sqrt{a^4(1-a^2)} \quad (6)$$ To calculate the maximum of $A$ we can use the AM-GM inequality: $$(\frac{a^2}{2}\frac{a^2}{2}(1-a^2))^{\frac{1}{3}} \leq \frac{\frac{a^2}{2}+\frac{a^2}{2}+(1-a^2)}{3}=\frac{1}{3} \Rightarrow$$ $$a^2a^2(1-a^2)\leq \frac{4}{27}. \quad(7)$$ We know that the equality holds for $$\frac{a^2}{2}=(1-a^2).\quad(8) $$ Therefore $$a=\sqrt{{\frac{2}{3}}}\quad(9)$$ From $(4)$ and $(9)$, we get: $$b=\frac{\sqrt{2}}{3}$$ Finally we get $$e=\sqrt{\frac{2}{3}}.$$