You replaced $\sin\theta$ with $\exp(i\theta)$ for $\theta=(a+b)/2$ and $\theta=(a-b)/2$, but these things are not equal.
From $e^{i\theta}=\cos\theta+i\sin\theta$ we may conclude the actual formulas:
$$\cos\theta = \frac{e^{i\theta}+e^{-i\theta}}{2} \qquad \sin\theta = \frac{e^{i\theta}-e^{-i\theta}}{2} $$
Now give it a try with the correct substitutions.
That is correct, you can use the quadratic formula for $c = 0$. And your work is all fine.
But note, you can save yourself time by simply factoring your equaton, and noting that when once has factors $a, b$, then $$a b = 0 \;\text{ if and only if } \;a= 0 \;\text{ or }\;b = 0$$
$$\begin{align} 2y^2 - 3\sqrt 3y = 0 & \iff y\,(2y - 3\sqrt 3) = 0\\ \\ &\iff y = 0\;\text{ or }\; \left(2y - 3\sqrt 3 = 0 \iff \;y = \frac{3\sqrt 3}2\right)\end{align}$$
Don't forget that $y = \cos x$, so we need to solve for $x$ given $$\cos x =y = 0\text{ or } \cos x = y = \dfrac {3\sqrt 3}{2}$$ and you are correct that we can throw out the possibility that $\cos x = \frac{3\sqrt 3}2 > |1|$.
So, for $x\in [0, 2\pi) = [0^\circ, 360^\circ) $ then indeed, $x = \pi/2 = 90^\circ, $ or $\,x = 3\pi/2 = 270^\circ$
Best Answer
I would have written it as follows: $$\begin{align}c^2&=8^2+5^2-(2\times5\times8\times0.5)\\&=64+25-40\\&=89-40\\&=49=7^2\\\implies c&=\sqrt{7^2}=7\end{align}$$
Perhaps the computer wanted to see "$c=7$" as the final line.