Added: As explained in the comments, certain trigonometric equations such as the linear equations in $\sin x$ and $\cos x$ can be solved by a resolvent quadratic equation. One method is to write the $\sin x$ and $\cos x$ functions in terms of the same trigonometric function. Since all [direct] trigonometric functions of the simple angle can be expressed rationally as a function of the $\tan$ of the half-angle, such a conversion is adequate for these equations.
Since $$\cos \alpha =\frac{1-\tan ^{2}\frac{\alpha }{2}}{1+\tan ^{2}\frac{%
\alpha }{2}}$$
and
$$\sin \alpha =\frac{2\tan \frac{\alpha }{2}}{1+\tan ^{2}%
\frac{\alpha }{2}}$$
your equation
$$m\cos \alpha +\sin \alpha =1$$
is equivalent to
$$m-m\tan ^{2}\frac{\alpha }{2}+2\tan \frac{\alpha }{2}=1+\tan ^{2}\frac{\alpha }{2}.$$
One may set $x=\tan \frac{\alpha }{2}$ ($\alpha =2\arctan x$), and thus get
the quadratic equation
$$\left( 1+m\right) x^{2}-2x+1-m=0.$$
Its solutions are: $x=\frac{1}{m+1}\left( -m+1\right) $ (if $m\neq -1$) or $%
x=1$ (if $m=-1$), which gives
i) If $m\neq -1$,
$$\alpha =2\arctan x=2\arctan \frac{1-m}{m+1},$$
ii) If $m=-1,$
$$\alpha =2\arctan 1=\frac{\pi }{2}.$$
A different technique to solve a linear equation in $\sin \alpha $ and $\cos
\alpha $ is to use an auxiliary angle $\varphi $. If you set $m=\tan \varphi
$, your equation takes the form
$$\sin \alpha +\tan \varphi \cdot \cos \alpha =1$$
or
$$\sin (\alpha +\varphi )=\cos \varphi =\frac{1}{\pm \sqrt{1+\tan ^{2}\varphi
}}=\pm \sqrt{\frac{1}{1+m^{2}}},$$
and obtain
$$\alpha =\pm \arcsin \sqrt{\frac{1}{1+m^{2}}}-\arctan m.$$
Detailed derivation: from $m\cos \alpha +\sin \alpha =1$ and $m=\tan \varphi
$, we get
$$\sin \alpha +\tan \varphi \cdot \cos \alpha =1\iff\sin \alpha +\dfrac{\sin \varphi }{\cos \varphi }\cdot \cos \alpha =1$$
$$\iff\dfrac{\sin \alpha \cdot \cos \varphi +\sin \varphi \cdot \cos \alpha }{\cos
\varphi }=1\iff\dfrac{\sin \left( \alpha +\varphi \right) }{\cos \varphi }=1$$
$$\iff\sin \left( \alpha +\varphi \right) =\cos \varphi .$$
The identity
$$\cos \varphi =\pm \sqrt{\dfrac{1}{1+\tan ^{2}\varphi }}$$
can be obtained as follows
$$\sin ^{2}\varphi +\cos ^{2}\varphi =1\iff\dfrac{\sin ^{2}\varphi }{\cos ^{2}\varphi }+1=\dfrac{1}{\cos ^{2}\varphi }$$
$$\iff\tan ^{2}\varphi +1=\dfrac{1}{\cos ^{2}\varphi }\iff\cos ^{2}\varphi =\dfrac{1}{1+\tan ^{2}\varphi }.$$
Therefore
$$\sin \left( \alpha +\varphi \right) =\pm \sqrt{\dfrac{1}{1+\tan ^{2}\varphi }}\iff\alpha +\varphi =\arcsin \left( \pm \sqrt{\dfrac{1}{1+\tan ^{2}\varphi }}\right) $$
$$\iff\alpha +\arctan m=\arcsin \left( \pm \sqrt{\dfrac{1}{1+m^{2}}}\right) \qquad (m=\tan \varphi,\ \varphi =\arctan m)$$
and finally
$$\alpha =\arcsin \left( \pm \sqrt{\dfrac{1}{1+m^{2}}}\right) -\arctan m.$$
Best Answer
Use the tan half-angle subsitution $z = \tan (x/2)$ or $$x = 2\, {\rm atan}(z)$$
Note that
$$\begin{aligned} \cos(x) & = \frac{ (1-z^2)}{ (1+z^2)} \\ \sin(x) & = \frac{ (2 z)}{ (1+z^2)} \end{aligned}$$
which is substituted into the equation to get
$$ \begin{aligned} 5 \sin(x) + 4 \cos(x) - 10 \sin(x) \cos(x) & = 2 \\ \hline \\ 5 \frac{ (2 z)}{ (1+z^2)} + 4 \frac{ (1-z^2)}{ (1+z^2)} - 10 \frac{ (2 z)}{ (1+z^2)} \frac{ (1-z^2)}{ (1+z^2)} & = 2 \end{aligned} $$
The above is simplified to
$$ -\frac{ 2 z^4 -15 z^3 + 5 z -2}{ (1+z^2)^2 } = 1 $$
or
$$ 3 z^4 - 15 z^3 + 2 z^2 + 5 z = 1 $$
$$ (z^2 -5 z +1) (3 z^2 -1) = 0 $$
The four (primary) solutions are
$$ \begin{aligned} z & = \begin{cases} \frac{1}{\sqrt{3}} \\ - \frac{1}{\sqrt{3}} \\ \frac{5}{2} + \frac{\sqrt{21}}{2} \\ \frac{5}{2} - \frac{\sqrt{21}}{2} \end{cases} \end{aligned} \rightarrow \begin{aligned} x & = \begin{cases} \frac{\pi}{3} \\ - \frac{\pi}{3} \\ \frac{\pi}{2} + 2\, {\rm atan}\left( \frac{\sqrt{21}}{7} \right) \\ \frac{\pi}{2} - 2\, {\rm atan}\left( \frac{\sqrt{21}}{7} \right) \end{cases} \end{aligned} $$