This is a nice example of a problem where the "right" choice for the independent variable makes a solution reasonably tractable, whereas other choices will lead to rather rough going. I tried a straight-ahead Cartesian approach (which led to a quintic equation) and the trigonometric approach you attempted (which gave a derivative function running up to fourth powers of sine which was going to be difficult to solve -- though it could be shown that one would obtain the expected angle for an equilateral triangle).
Instead we will work with the slope $ \ m \ $ of the sides of the isosceles triangle. If we center the circle of radius $ \ r \ $ on the origin, the "base" will lie along the line $ \ y \ = \ -r \ $ ; we will work with one vertex at $ \ ( \ a, \ -r \ ) \ $ , continuing your notation. The apex of this triangle is then located at $ \ ( \ 0, \ h \ - \ r \ ) \ $ and the area of the triangle is $ \ A \ = \ ah \ $ . The slope of the marked side is then $ \ m \ = \ -\frac{h}{a} \ $ .
We need some sort of constraint for this optimization: that is provided by the fact that the circumscribed triangle must be tangent to the circle on each of its sides. We will call the tangent point on the marked side $ \ ( \ X, \ Y \ ) \ $ . The equation of the line along which this side lies is $ \ y \ = \ ( \ h \ - \ r \ ) \ - \ \frac{h}{a} x \ $ .
Since the radius of the circle which extends to the tangent point is perpendicular to the tangent line, the equation of the line for this radius is $ \ y \ = \ -\frac{1}{m} x \ $ . Its intersection with the tangent line is thus given by
$$ \ -\frac{1}{m} X \ = \ ( \ h \ - \ r \ ) \ + \ m X \ \ \Rightarrow \ \ X \ = \ -\frac{m}{m^2 \ + \ 1} \ ( \ h \ - \ r \ ) $$
$$ \Rightarrow \ \ Y \ = \ \frac{1}{m^2 \ + \ 1} \ ( \ h \ - \ r \ ) \ \ . $$
Now we apply the constraint: the tangent point lies on the circle, so
$$ X^2 \ + \ Y^2 \ = \ r^2 \ \ \Rightarrow \ \ \left[ \ \left(-\frac{m}{m^2 \ + \ 1}\right)^2 \ + \ \left(\frac{1}{m^2 \ + \ 1}\right)^2 \ \right] \ ( \ h \ - \ r \ )^2 \ = \ r^2 $$
$$ \Rightarrow \ \ \left(\frac{1}{m^2 \ + \ 1}\right) \ ( \ h^2 \ - \ 2rh \ + \ r^2 ) \ = \ r^2 \ \ . $$
If, for the moment, we call $ \ \mu \ = \ \frac{1}{m^2 \ + \ 1} \ $ , then we can solve the quadratic equation $ \ \mu \ h^2 \ - \ 2 \mu r \ h \ + \ ( \ \mu \ - \ 1 \ ) r^2 \ = \ 0 \ $ to obtain
$$ h \ = \ \frac{2 \mu r \ \pm \ \sqrt{4 \mu^2 r^2 \ - \ 4 \mu \ ( \mu \ - \ 1 \ ) \ r^2}}{2 \mu} \ = \ r \ \left( 1 \ \pm \ \frac{1}{\sqrt{\mu}} \right) \ \ ; $$
since it is plainly the case that $ \ h \ > \ r \ $ , we discard the "negative solution" to produce $ \ h \ = \ r \ ( \ 1 \ + \ \sqrt{m^2 \ + \ 1} \ ) \ $ .
We have made our way through the first thicket and prepare to enter the second one. The area function is now found from
$$ m \ = \ -\frac{h}{a} \ \ \Rightarrow \ \ a \ = \ -\frac{h}{m} $$
$$ \Rightarrow \ \ A \ = \ ah \ = \ -\frac{h^2}{m} \ = \ - r^2 \ \left( \frac{m^2 \ + \ 2 \ + \ 2\sqrt{m^2 \ + \ 1}}{m} \right) \ \ . $$
It is this latter function for which we wish to locate the "critical value" for $ \ m \ $ ; hence,
$$ \frac{dA}{dm} \ = \ - r^2 \ \left( 1 \ - \ \frac{2}{m^2} \ - \ \frac{2}{m^2 \ \sqrt{m^2 \ + \ 1}} \right) $$
$$ = \ - r^2 \ \left( \frac{m^2 \ \sqrt{m^2 \ + \ 1} \ - \ 2 \sqrt{m^2 \ + \ 1} \ - \ 2} {m^2 \ \sqrt{m^2 \ + \ 1}} \right) \ = \ 0 \ \ . $$
Since the denominator cannot be zero (as $ \ m \ $ is non-zero), we only need to consider when the numerator is zero:
$$ m^2 \ \sqrt{m^2 \ + \ 1} \ - \ 2 \sqrt{m^2 \ + \ 1} \ - \ 2 \ = \ 0 \ \ \Rightarrow \ \ (m^2 \ - \ 2 ) \ \sqrt{m^2 \ + \ 1} \ = \ 2 $$
$$ \Rightarrow \ \ m^2 \ + \ 1 \ = \ \frac{4}{(m^2 \ - \ 2)^2} \ \ \Rightarrow \ \ m^6 \ - \ 3 \ m^4 \ = \ m^4 \ ( \ m^2 \ - \ 3 \ ) \ = \ 0 \ \ . $$
[squaring both sides of the equation and simplifying in this last line]
We have already rejected zero as a possible solution, so we have $ \ m \ = \ \pm \sqrt{3} \ $ . But these are the slopes of the sides of an equilateral triangle above the (horizontal) base; this then is our circumscribed isosceles triangle of minimal area.
For solving $(b)$, we can find the total height $(h_T)$ of the isosceles triangle by adding the value for $h$ found in $(a)$ to the other height $6$:
$$h_T=6+3=9.$$
So, we can say that the total height of the triangle is $9$ units. Then we can find the base:
$$b=\sqrt{36-3^2}=\sqrt{36-9}=\sqrt{27}=3\sqrt{3}.$$
We can then divide the base by $2$ to get the length of the base for half the triangle (by the $AAS$ Similar Triangles Theorem) to get the base to be $\frac{3\sqrt{3}}{2}$. We can now focus on the right half of the triangle divided by the dotted line going through the vertex point of the circle. We can tell that the dotted line through the vertex makes a right angle. Thus, we can use Trigonometry to solve for $\alpha$. So,
$$\alpha=\tan\frac{\text{opposite}}{\text{adjacent}}=\tan\frac{\frac{3\sqrt{3}}{2}}{9}=\tan\frac{27\sqrt{3}}{2}.$$
Since we have found what $\alpha$ is, we can find the function of it with respect to $\alpha$:
$$\boxed{\alpha=\tan\left(\frac{(6+h)(\sqrt{36-h^2})}{2}\right)},$$
or as an incorporated function:
$$\frac12bh=\frac12(6\sin\alpha)(12\cos\alpha)=\boxed{36\sin\alpha\cos\alpha}.$$
And we are done with $(b)$!
For $(c)$, By the Isoperimetric Theorem, it states
Theorem: Among all triangles inscribed in a given circle, the equilateral one has the largest area.
Therefore, the equilateral triangle has the maximum area.
Best Answer
Let $\theta$ be one-half of the vertex angle (less than a right angle) of the isosceles triangle.
Differentiating $A$ and setting it to $0$, you will be left with the problem of solving
$\cos\theta = -\cos2 \theta.$
Plugging these into $A$ we get the answer: $\dfrac{3\sqrt 3 r^2}{4}.$