Since you are having trouble findind the area in terms of median length and cosine of vertex angle, I suggest you do it differently.
The shape of an isosceles triangle, up to congruence, has two degrees of freedom. Choose two measurable objects that determine the triangle. Then find expressions for the other relevant properties, namely the length of the median, the area, and the cosine of the vertex angle. Then set the expression for the median length equal to a constant and solve for one of your original objects in terms of the other. Substitute that into the expression for area, and you now have an expression for area in terms of only one variable. Now use calculus to maximize that area. Substitute that into your expression for the cosine of the vertex angle, and you are done.
I left that explanation very general, because I can think of many ways to define an isosceles triangle in terms of two variables. Here is one way that more directly uses the cosine of the vertex angle.
I have placed the isosceles triangle on a Cartesian coordinate system, with the apex vertex at the origin and one of the equal sides on the positive $x$-axis. We can let $c$ be the length of the equal sides and $\alpha$ be the measure of the apex angle. Then it is easy to find expressions for the vertices coordinates, midpoint of one side, median length, altitude, and triangle area. You can continue from there.
In your comments you say you want your two variables that define the isosceles triangle to be $c$, the length of the two equal sides, and $b$, the length of the other side. That does also work, though that does not directly include what your final answer will be, the cosine of the apex angle.
As you wrote, we have
$$m^2=\frac{c^2+2b^2}{4}$$
We can easily solve this for $c$ and we get
$$c^2=4m^2-2b^2$$
Your area formula $A=\frac 12ch$ is not useful here, since finding $h$ directly is difficult for you. Instead, use the base and height from my second diagram. The base is obviously $b$, and side $c$ is the hypotenuse of a right triangle with sides $\frac b2$ and $h'$, where $h'$ is the altitude on side $\overline{AB}$. We then find that
$$h'=\sqrt{c^2-\frac 14b^2}$$
We then get for the area of the triangle,
$$\begin{align}
A&=\frac 12bh' & \\[2 ex]
&= \frac 12b\sqrt{c^2-\frac 14b^2} & (\text{substituted for }h')\\[2 ex]
&= \frac 12b\sqrt{(4m^2-2b^2)-\frac 14b^2} & (\text{substituted for }c^2)\\[2 ex]
&= \frac 12b\sqrt{4m^2-\frac 94b^2} \\[2 ex]
&= \frac 14b\sqrt{16m^2-9b^2}
\end{align}$$
The median, $m$, is a constant, so we finally have a formula for the area in terms of only one variable, the side $b$.
You should be able to continue from here, finding the value of $b$ that maximizes the area (or the square of the area) and finding an expression for $\cos\alpha$ in terms of $b$ then in terms of just constants.
Best Answer
Use:
In triangle $ABD:$
$$\angle D=90^{\circ},\angle A=30^{\circ} \Rightarrow BD=\frac12AB=10$$