This is a nice example of a problem where the "right" choice for the independent variable makes a solution reasonably tractable, whereas other choices will lead to rather rough going. I tried a straight-ahead Cartesian approach (which led to a quintic equation) and the trigonometric approach you attempted (which gave a derivative function running up to fourth powers of sine which was going to be difficult to solve -- though it could be shown that one would obtain the expected angle for an equilateral triangle).
Instead we will work with the slope $ \ m \ $ of the sides of the isosceles triangle. If we center the circle of radius $ \ r \ $ on the origin, the "base" will lie along the line $ \ y \ = \ -r \ $ ; we will work with one vertex at $ \ ( \ a, \ -r \ ) \ $ , continuing your notation. The apex of this triangle is then located at $ \ ( \ 0, \ h \ - \ r \ ) \ $ and the area of the triangle is $ \ A \ = \ ah \ $ . The slope of the marked side is then $ \ m \ = \ -\frac{h}{a} \ $ .
We need some sort of constraint for this optimization: that is provided by the fact that the circumscribed triangle must be tangent to the circle on each of its sides. We will call the tangent point on the marked side $ \ ( \ X, \ Y \ ) \ $ . The equation of the line along which this side lies is $ \ y \ = \ ( \ h \ - \ r \ ) \ - \ \frac{h}{a} x \ $ .
Since the radius of the circle which extends to the tangent point is perpendicular to the tangent line, the equation of the line for this radius is $ \ y \ = \ -\frac{1}{m} x \ $ . Its intersection with the tangent line is thus given by
$$ \ -\frac{1}{m} X \ = \ ( \ h \ - \ r \ ) \ + \ m X \ \ \Rightarrow \ \ X \ = \ -\frac{m}{m^2 \ + \ 1} \ ( \ h \ - \ r \ ) $$
$$ \Rightarrow \ \ Y \ = \ \frac{1}{m^2 \ + \ 1} \ ( \ h \ - \ r \ ) \ \ . $$
Now we apply the constraint: the tangent point lies on the circle, so
$$ X^2 \ + \ Y^2 \ = \ r^2 \ \ \Rightarrow \ \ \left[ \ \left(-\frac{m}{m^2 \ + \ 1}\right)^2 \ + \ \left(\frac{1}{m^2 \ + \ 1}\right)^2 \ \right] \ ( \ h \ - \ r \ )^2 \ = \ r^2 $$
$$ \Rightarrow \ \ \left(\frac{1}{m^2 \ + \ 1}\right) \ ( \ h^2 \ - \ 2rh \ + \ r^2 ) \ = \ r^2 \ \ . $$
If, for the moment, we call $ \ \mu \ = \ \frac{1}{m^2 \ + \ 1} \ $ , then we can solve the quadratic equation $ \ \mu \ h^2 \ - \ 2 \mu r \ h \ + \ ( \ \mu \ - \ 1 \ ) r^2 \ = \ 0 \ $ to obtain
$$ h \ = \ \frac{2 \mu r \ \pm \ \sqrt{4 \mu^2 r^2 \ - \ 4 \mu \ ( \mu \ - \ 1 \ ) \ r^2}}{2 \mu} \ = \ r \ \left( 1 \ \pm \ \frac{1}{\sqrt{\mu}} \right) \ \ ; $$
since it is plainly the case that $ \ h \ > \ r \ $ , we discard the "negative solution" to produce $ \ h \ = \ r \ ( \ 1 \ + \ \sqrt{m^2 \ + \ 1} \ ) \ $ .
We have made our way through the first thicket and prepare to enter the second one. The area function is now found from
$$ m \ = \ -\frac{h}{a} \ \ \Rightarrow \ \ a \ = \ -\frac{h}{m} $$
$$ \Rightarrow \ \ A \ = \ ah \ = \ -\frac{h^2}{m} \ = \ - r^2 \ \left( \frac{m^2 \ + \ 2 \ + \ 2\sqrt{m^2 \ + \ 1}}{m} \right) \ \ . $$
It is this latter function for which we wish to locate the "critical value" for $ \ m \ $ ; hence,
$$ \frac{dA}{dm} \ = \ - r^2 \ \left( 1 \ - \ \frac{2}{m^2} \ - \ \frac{2}{m^2 \ \sqrt{m^2 \ + \ 1}} \right) $$
$$ = \ - r^2 \ \left( \frac{m^2 \ \sqrt{m^2 \ + \ 1} \ - \ 2 \sqrt{m^2 \ + \ 1} \ - \ 2} {m^2 \ \sqrt{m^2 \ + \ 1}} \right) \ = \ 0 \ \ . $$
Since the denominator cannot be zero (as $ \ m \ $ is non-zero), we only need to consider when the numerator is zero:
$$ m^2 \ \sqrt{m^2 \ + \ 1} \ - \ 2 \sqrt{m^2 \ + \ 1} \ - \ 2 \ = \ 0 \ \ \Rightarrow \ \ (m^2 \ - \ 2 ) \ \sqrt{m^2 \ + \ 1} \ = \ 2 $$
$$ \Rightarrow \ \ m^2 \ + \ 1 \ = \ \frac{4}{(m^2 \ - \ 2)^2} \ \ \Rightarrow \ \ m^6 \ - \ 3 \ m^4 \ = \ m^4 \ ( \ m^2 \ - \ 3 \ ) \ = \ 0 \ \ . $$
[squaring both sides of the equation and simplifying in this last line]
We have already rejected zero as a possible solution, so we have $ \ m \ = \ \pm \sqrt{3} \ $ . But these are the slopes of the sides of an equilateral triangle above the (horizontal) base; this then is our circumscribed isosceles triangle of minimal area.
If we shrink the ellipse together with an inscribed triangle by a factor of $1/5$ along the $x$-axis and by a factor of $1/4$ along the $y$-axis, the ratio of the area of the triangle to the area of the ellipse remains unchanged. The ellipse is now transformed into the unit circle. It suffices to find the largest triangle inscribed in the unit circle.
Take an arbitrary chord of the unit circle as a side of the triangle. The triangle has the largest possible area if it is the point on the circle which is farthest away from the chord, which is the farther (from the chord) point of intersection of the perpendicular bisector of the chord and the circle. This implies that the triangle must be isosceles.
Let $x$ be the distance of the chord from the centre, then the area $A$ of the triangle is $(1+x)\sqrt{1-x^2}=\sqrt{(1-x)(1+x)^3}$.
$\dfrac{dA^2}{dx}=3(1-x)(1+x)^2-(1+x)^3=(1+x)^2(2-4x)$.
It is easy to check that $A$ is the maximum when $x=1/2$. The maximum area is $\dfrac{3\sqrt{3}}{4}$.
The maximum area of the inscribed triangle of the ellipse is $\displaystyle \dfrac{3\sqrt{3}}{4}\times 5\times 4=15\sqrt{3}$.
Best Answer
HINT:
Let $$\frac{dr}{d\theta}=0$$ and then rewrite the numerator completely in terms of $\sin \theta$.
If you need any more help please don't hesitate to ask :)
EDIT:
As you correctly found, the numerator of the derivative is equal to $$a(1-2\sin^2\theta-\sin^3\theta)$$ Hence, $$a(1-2\sin^2\theta-\sin^3\theta)=0$$ as when a fraction is equal to $0$ the denomiantor plays no part in making the fraction equal to $0$; the numerator must be equal to $0$. Now, assuming $a\ne0$, we can divide through by $a$ to obtain $$1-2\sin^2\theta-\sin^3\theta=0\implies \sin^3\theta+2\sin^2\theta-1=0$$ Now we can simply verify that $\sin\theta=\frac{\sqrt5 -1}{2}$ is a solution the equation above, and we are done.
Alternatively, we can find this solution directly. Notice that substituting $\sin\theta=-1$ gives us $0$; hence by the factor theorem $(\sin\theta+1)$ is a factor of $\sin^3\theta+2\sin^2\theta-1$. Hence, $$\sin^3\theta+2\sin^2\theta-1=(\sin\theta+1)(\sin^2\theta+\sin\theta-1)=0$$ Solving for the second factor we obtain the required solution. Notice that the other $2$ solutions are outside of the range of allowed solutions for $\theta$.