I think you are simply misreading the definition. They perhaps should have used the words "starting with" rather than "taking." That is:
- $\sin x, \cos x$ and constant functions are rational trig functions
- If $p(x),q(x)$ are rational trig functions, then $p(x)\cdot q(x), p(x)+q(x),p(x)-q(x)$ are rational trig functions. If $q(x)\neq 0$ for some $x$, then also $p(x)/q(x)$ is a rational trig function.
Even with your reading of the text, since $\sin^2 x + \cos^2 x=1$, we can get $$\cos x =\frac{\cos x}{\sin^2 x + \cos^2 x}$$ But I suspect that it was not the author's intent for the paragraph to be interpreted that way.
Some care will need to be taken if you want to also include multiple variable rational trig functions.
You aren't explicitly allowed to take square roots, but that doesn't mean that $\sqrt{\sin x}$ is not a rational trig function. I suspect that the authors just meant to make it clear that they didn't include some operations in allowing you to create rational trig functions.
For example, while $\sqrt{\sin x}$ is not rational, $\sqrt{2+2\sin x-\cos^2 x}$ is rational, since it happens to be equal to $1+\sin x$. Proving that $\sqrt{\sin x}$ can't be written that way is actually some work, and probably most easily done with complex analysis. Essentially, we can show that a rational trig function can only be undefined for finitely many $x\in[0,2\pi]$, and when it is defined at $x$, it is differentiable at $x$. This breaks for $\sqrt{\sin x}$.
But again, I suspect the authors don't want you to go that far into it, and instead just note, "we only allow these operation, and square root wasn't one of them."
It is because $\sin x$ and $\cos x$ are linearly independent -- if you are given
$$A\sin x + B\cos x \equiv 0$$
then both $A=B=0$.
Proof: If given $A\sin x + B\cos x \equiv 0$, then
$$\begin{align*}
A\sin x + B\cos x &\equiv 0\\
A\sin0^\circ + B\cos 0^\circ &= 0 &\iff&&B = 0\\
A\sin 90^\circ + B\cos 90^\circ &= 0 &\iff &&A=0
\end{align*}$$
So you have made to the point that
$$\begin{align*}
5\cos x -3\sin x &\equiv k\cos\alpha\sin x - k\sin\alpha\cos x\\
5\cos x+k\sin\alpha\cos x -3\sin x-k\cos\alpha\sin x &\equiv 0\\
(5+ k\sin\alpha)\cos x + (-3-k\cos\alpha)\sin x &\equiv 0
\end{align*}$$
From the result above,
$$\begin{align*}
5+k\sin\alpha &= 0&\iff&&-k\sin\alpha &= 5\\
-3-k\cos\alpha &= 0 &\iff&&k\cos\alpha &= -3
\end{align*}$$
which is the result of directly matching the coefficients of $\cos x$ and $\sin x$.
Best Answer
If you are trying $A\sin\alpha + B\cos\alpha = C\sin\alpha$ you can rewrite as $$A\sin\alpha + B\cos\alpha -C\sin\alpha=0\\(A-C)\sin\alpha + B\cos\alpha=0 \to \div \cos\alpha\\(A-C)\tan \alpha =-B \\\tan \alpha =-\frac{B}{A-C}$$ but if you are trying $A\sin\alpha + B\cos\alpha = C$ you can use the fact $$A\sin\alpha + B\cos\alpha = \frac{A}{|A|}\sqrt{A^2+B^2}\sin(x+\arctan \frac{B}{A})=c$$it can turn the equation to $$\sin(x+\arctan \frac{B}{A})=k$$