You can use the well-known Rodriguez rotation formula (https://en.wikipedia.org/wiki/Rodrigues%27_rotation_formula), which states
$
\textbf{v}' = \textbf{v}\cos\theta + (\textbf{r}\times\textbf{v})\sin\theta + \textbf{r}(\textbf{v}\cdot\textbf{r})(1-\cos\theta)
$
where $\textbf{v}$ is the original vector, $\textbf{r}$ is the vector about which the rotation on angle $\theta$ is performed, and $\textbf{v}'$ is the vector after rotation.
Now, by applying this formula twice (first, for example, for the rotation on angle $\alpha$ about $x-$axis and then on angle $\beta$ around $y-$axis), we get
$\textbf{k}'' = \textbf{i}\cos\alpha\sin\beta -\textbf{j}\sin\alpha + \textbf{k}\cos\alpha\cos\beta$.
Now, to find the angle between $\textbf{k}$ and $\textbf{k}''$, you can take the dot-product of them
$\cos \angle(\textbf{k},\textbf{k}'') = \textbf{k}\cdot\textbf{k}'' = \cos\alpha\cos\beta$.
The compass direction is given by the vector $\textbf{k}''_{||} = \textbf{i}\cos\alpha\sin\beta -\textbf{j}\sin\alpha$.
Now, let consider specific example of $\alpha = 4^\circ$ and $\beta = 18^\circ$. In this case, $\cos \alpha = 0.9976$, $\sin\alpha = 0.07$, $\cos\beta = 0.951$, and $\sin\beta = 0.309$. As a result, combining all these numbers together, we get
$\angle(\textbf{k},\textbf{k}'') = 18.42^\circ$,
and the compass direction with respect to the the original $x-$axis is $-\tan^{-1}(\tan\alpha\sin\beta) = 1.24^\circ$.
Hope this helps.
The main problem is that your equations are not consistent with your coordinate system.
Your equations are consistent with this coordinates, which is much more commonly used in mathematics:
Unless otherwise specified, mathematical articles usually use this coordinate system when dealing with 2D problems.
There are two main characteristics:
The vertical axis increases upward, and the horizontal axis increases rightward.
The angles are consistent with the right-hand grip rule: angle increases in the anticlockwise direction.
For your coordinate system, you need to modify your equations, which are stated in one of the comments on your question.
Best Answer
Given a particular value $a = \tan(x)$ there are two angles in the interval $(-\pi, \pi]$ or $[0, 2\pi)$ that have $a$ as their tangent. (See graph.)
To determine which angle, you'll need to look at which quadrant your point is in, and you can do this by inspection. Then, if you end up with an angle that doesn't make sense, you simply add or subtract $\pi$ to get it in the right quadrant.
Or, as suggested in the comments, you can use the "atan2" function that takes two arguments (thereby specifying the quadrant for you) and you'll get the right answer straight off.