trigonometry
We know that $\arctan(\tan(x))=x$ when $x$ lies between $-\pi/2$ and $+\pi/2$; but do you know a way to transform the expression $\arctan(a\tan(x))$, where $a$ is a real number between $0$ and $1$?
I thought $a$ could be transformed with trigonometric functions, such as $a=\sin(\alpha)\cos(x)$, but $\arctan(\sin(\alpha)\sin(x))$ does not remind me anything.
Maybe there is no further possible transformation?
And then used the solution of Trigonometric Equation $\tan(θ)=\tan(β)$…
That sounds like a good approach to me. So what you're saying is that you got to
$$ 5\pi\cos\alpha = n\pi+\tfrac\pi2-5\pi\sin\alpha,\quad n\in\mathbb Z $$
and then solved this for $\alpha$? How exactly?
Personally I would use the tangent half-angle formulas to turn this trigonometric equation into a polynomial one:
$$ t:=\tan\frac\alpha2\quad \sin\alpha=\frac{2t}{1+t^2}\quad \cos\alpha=\frac{1-t^2}{1+t^2}\\ 5\frac{1-t^2}{1+t^2}=n+\frac12-5\frac{2t}{1+t^2}\\ 10-10t^2=2n+2nt^2+1+t^2-20t\\ $$
So what values of $n$ should you be considering? Let'suse the fact that $\sin\alpha\in[-1,1]$ and the same for $\cos\alpha$.
$$5[-1\ldots 1]=n+\tfrac12-5[-1\ldots 1]\\n=5[-1\ldots 1]+5[-1\ldots 1]-\tfrac12$$
So a conservative estimate would be $n\in\{-10,-9,-8,\ldots,7,8,9\}$. Since you can't have both $\sin\alpha$ and $\cos\alpha$ be close to $\pm1$ at the same time, not all of these $n$ will have solutions, but this is good enough for now. Take each $n$ and compute the resulting $t$ (at most two for each $n$). You get $28$ different values.
$$ \begin{array}{rl|rr|r} t && \alpha && n \\\hline -18.88819 = & -\sqrt{79} - 10 & -3.035805 = & -173.93882° & -6 \\ -5.18925 = & -\frac{1}{3} \, \sqrt{31} - \frac{10}{3} & -2.760848 = & -158.18495° & -7 \\ -1.47741 = & \frac{1}{3} \, \sqrt{31} - \frac{10}{3} & -1.951541 = & -111.81505° & -7 \\ -1.11181 = & \sqrt{79} - 10 & -1.676584 = & -96.06118° & -6 \\ -0.90871 = & -\sqrt{119} + 10 & -1.475215 = & -84.52361° & -5 \\ -0.76274 = & -\frac{1}{3} \, \sqrt{151} + \frac{10}{3} & -1.303204 = & -74.66809° & -4 \\ -0.64575 = & -\sqrt{7} + 2 & -1.146765 = & -65.70481° & -3 \\ -0.54575 = & -\frac{1}{7} \, \sqrt{191} + \frac{10}{7} & -0.999154 = & -57.24732° & -2 \\ -0.45630 = & -\frac{1}{9} \, \sqrt{199} + \frac{10}{9} & -0.856168 = & -49.05481° & -1 \\ -0.37334 = & -\frac{1}{11} \, \sqrt{199} + \frac{10}{11} & -0.714628 = & -40.94519° & 0 \\ -0.29387 = & -\frac{1}{13} \, \sqrt{191} + \frac{10}{13} & -0.571642 = & -32.75268° & 1 \\ -0.21525 = & -\frac{1}{3} \, \sqrt{7} + \frac{2}{3} & -0.424031 = & -24.29519° & 2 \\ -0.13460 = & -\frac{1}{17} \, \sqrt{151} + \frac{10}{17} & -0.267592 = & -15.33191° & 3 \\ -0.04783 = & -\frac{1}{19} \, \sqrt{119} + \frac{10}{19} & -0.095581 = & -5.47639° & 4 \\ 0.05294 = & -\frac{1}{21} \, \sqrt{79} + \frac{10}{21} & 0.105787 = & 6.06118° & 5 \\ 0.19271 = & -\frac{1}{23} \, \sqrt{31} + \frac{10}{23} & 0.380745 = & 21.81505° & 6 \\ 0.67686 = & \frac{1}{23} \, \sqrt{31} + \frac{10}{23} & 1.190052 = & 68.18495° & 6 \\ 0.89944 = & \frac{1}{21} \, \sqrt{79} + \frac{10}{21} & 1.465009 = & 83.93882° & 5 \\ 1.10046 = & \frac{1}{19} \, \sqrt{119} + \frac{10}{19} & 1.666377 = & 95.47639° & 4 \\ 1.31107 = & \frac{1}{17} \, \sqrt{151} + \frac{10}{17} & 1.838389 = & 105.33191° & 3 \\ 1.54858 = & \frac{1}{3} \, \sqrt{7} + \frac{2}{3} & 1.994827 = & 114.29519° & 2 \\ 1.83233 = & \frac{1}{13} \, \sqrt{191} + \frac{10}{13} & 2.142438 = & 122.75268° & 1 \\ 2.19152 = & \frac{1}{11} \, \sqrt{199} + \frac{10}{11} & 2.285425 = & 130.94519° & 0 \\ 2.67853 = & \frac{1}{9} \, \sqrt{199} + \frac{10}{9} & 2.426964 = & 139.05481° & -1 \\ 3.40290 = & \frac{1}{7} \, \sqrt{191} + \frac{10}{7} & 2.569951 = & 147.24732° & -2 \\ 4.64575 = & \sqrt{7} + 2 & 2.717562 = & 155.70481° & -3 \\ 7.42940 = & \frac{1}{3} \, \sqrt{151} + \frac{10}{3} & 2.874000 = & 164.66809° & -4 \\ 20.90871 = & \sqrt{119} + 10 & 3.046012 = & 174.52361° & -5 \end{array} $$
All of these look like valid solutions to me: they satisfy the initial equation. Since the tangent half-angle formulas can't represent $\alpha=\pi$ (it corresponds to $t=\infty$), we also need to check that this is not a solution. And of course these $\alpha$ are arguments to trigonometric functions, so adding any multiple of $2\pi$ will be a solution, too. The above are all the solutions in the $\alpha\in(-\pi,+\pi]$ range.
Update: After reading some other answers, and seeing how they avoid the tangent half-angle formulas, I wanted to look up the computation for that using well-established identities. Starting from the equation
\begin{align*} 5\pi\cos\alpha &= n\pi+\tfrac\pi2-5\pi\sin\alpha,\quad n\in\mathbb Z \\ \sin\alpha+\cos\alpha &= \frac{2n+1}{10} \end{align*}
the sum on the left hand side is the most interesting part. Wikipedia list of trigonometric identities lists your $\tan\left(\tfrac\pi2-\theta\right)=\cot\theta$ under Reflections and also some formulas you can use to tackle that sum.
One approach uses shifts to turn $\cos$ into $\sin$ and product to sum identities in reverse to turn the sum into a product:
\begin{align*} \cos\alpha &= \sin(\alpha+\tfrac\pi2) \\ \sin(\theta+\varphi)+\sin(\theta-\varphi)&=2\sin\theta\cos\varphi \qquad\text{with } \theta:=\alpha+\tfrac\pi4, \quad \varphi:=\tfrac\pi4 \\ \sin\alpha+\cos\alpha = \sin\alpha + \sin(\alpha+\tfrac\pi2) &= 2\sin(\alpha+\tfrac\pi4)\cos\tfrac\pi4 = \sqrt2\sin(\alpha+\tfrac\pi4) \end{align*}
You might also start from a formula for angle sums:
\begin{align*} \sin\alpha\cos\beta + \cos\alpha\sin\beta &= \sin(\alpha+\beta) \\ \beta := \tfrac\pi4 \qquad & \cos\beta=\sin\beta=\tfrac1{\sqrt2} \\ \tfrac1{\sqrt2}\left(\sin\alpha+\cos\alpha\right) &= \sin\left(\alpha+\tfrac\pi4\right) \end{align*}
Either way you get
$$ \sin\alpha+\cos\alpha = \sqrt2\sin(\alpha+\tfrac\pi4) = \frac{2n+1}{10} \\ \sin(\alpha+\tfrac\pi4) = \frac{2n+1}{10\sqrt2} \\ \alpha = \arcsin\frac{2n+1}{10\sqrt2}-\frac14\pi \qquad\text{or}\qquad \alpha = \frac34\pi-\arcsin\frac{2n+1}{10\sqrt2} \qquad\pmod{2\pi} $$
where the second solution accounts for the fact that $\arcsin$ should be considered a multi-valued function, and I'd like to get all solution angles in some $2\pi$-wide interval. You'd consider any $n\in\mathbb Z$ for which
$$ -1\le\frac{2n+1}{10\sqrt2}\le1\\ -7.57\approx\frac{-10\sqrt2-1}2\le n\le\frac{10\sqrt2-1}2\approx6.57 $$
which matches the list in my original table of solutions.
But the basic condition of using the above result is that $\beta$ lies between $\left(-\frac π2,\frac π2\right)$.
I'm not sure where you get this condition from. Neither the move from $\cot$ to $\tan$ nor the approach for solving $\tan\theta=\tan\beta$ does warrant such a restriction, as far as I can reason about it.
And so gives $\sin \alpha $ lies between $\left(0,\frac 15\right)$
Since some of the solutions in the above table are outside that range and appear to be valid, that's not the case.
Hint
Let $$ t=\arcsin(\cos(x))$$
then
$$-\frac{\pi}{2}\le t\le \frac{\pi}{2}$$
and
$$\sin(t)=\cos(x)=\sin(\frac{\pi}{2}+x)$$
thus
$$t=\arcsin(\cos(x))=x+\frac{\pi}{2}+2k\pi$$ or $$t=\frac{\pi}{2}-x+2k\pi$$
we choose the the value in $ [-\frac{\pi}{2},\frac{\pi}{2}]$.
Best Answer
Well, you can convert the arctan to other inverse-trig functions, such as arcsin:
$$ \arctan(a \tan(x)) = \arcsin\left(\frac{a \sin(x)}{\sqrt{\cos(x)^2 + a^2 \sin(x)^2}}\right) $$