When $x > 1,$ as in this case, $\arctan x > \frac\pi4,$
and therefore $2 \arctan x > \frac\pi2.$
That is, $2 \arctan x$ is in the second quadrant.
But $\arcsin$ can only produce angles in the first and fourth quadrants.
There is a similar difficulty when $x < -1.$
That's why the formula $2\tan^{-1}x=\sin^{-1}\frac{2x}{1+x^2}$
is good only for $\lvert x\rvert \leq 1.$
But when $x > 1,$ you need a different formula. One correct formula is
$$2\tan^{-1}x = \pi - \sin^{-1}\frac{2x}{1+x^2}.$$
And $\sin(\pi - \theta) = \sin \theta,$ so you get the same result in the end.
Postscript:
The following is a kind of extended comment on the question. You could also view it as an answer to a question that was not literally asked.
Note that for any real number $x$ (without restriction) we have
\begin{align}
\sin\left(\tan^{-1} x\right) &= \frac{x}{\sqrt{1+x^2}}, \\
\cos\left(\tan^{-1} x\right) &= \frac{1}{\sqrt{1+x^2}}.
\end{align}
So if $\tan^{-1} x = \alpha$ then
\begin{align}
\sin\left(2\tan^{-1} x\right) = \sin(2\alpha)
&= 2\sin\alpha \cos\alpha \\
&= 2\left(\frac{x}{\sqrt{1+x^2}}\right)\left(\frac{1}{\sqrt{1+x^2}}\right)
\\[1ex]
&= \frac{2x}{1+x^2}.
\end{align}
So this is in fact a general formula that does not require any restrictions or special cases in order to deal with the quadrants in which the trig functions fall.
Let $x=e^{ix}$, $y=e^{iy}$, $z=e^{iz}$. We want to find $\text{Im}(xyz)$.
We can express this system of equations as
$$x+ y+z=\omega :=\frac{11}{5}+2i,$$
and
$$\left(x - \frac{1}{x}\right)\left(y + \frac{1}{y}\right)\left(z + \frac{1}{z}\right) + \left(x + \frac{1}{x}\right)\left(y - \frac{1}{y}\right)\left(z + \frac{1}{z}\right) +\left(x + \frac{1}{x}\right)\left(y + \frac{1}{y}\right)\left(z - \frac{1}{z}\right) = \frac{17i}{6}\left(x + \frac{1}{x}\right)\left(y + \frac{1}{y}\right)\left(z + \frac{1}{z}\right),$$
which after expanding and simplifying, gives
$$(18-17i)x y z +(6-17i)\left(\frac{x y}{z} + \frac{x z}{y}+ \frac{y z}{x}\right) -(6+17i)\left(\frac{x}{yz} + \frac{y}{xz}+ \frac{z }{xy}\right) -\frac{18+17i}{xyz} = 0.$$
Noting the symmetry, this implies
$$\text{Im}\left((18-17i)x y z +(6-17i)\underbrace{\left(\frac{x y}{z} + \frac{x z}{y}+ \frac{y z}{x}\right)}_{A}\right) = 0.$$
The plan is to express $A$ in terms of $xyz$, then find solutions for $xyz$.
Taking the conjugate, the first condition implies that
$$\frac{1}{x}+\frac{1}{y}+\frac{1}{z}=\overline \omega.$$
Squaring, we have
$$\frac{1}{x^2}+\frac{1}{y^2}+\frac{1}{z^2}+\frac{2}{xy}+\frac{2}{xz}+\frac{2}{yz}=\overline\omega^2,$$
which gives for $A$
$$\frac{x y}{z} + \frac{x z}{y}+ \frac{y z}{x} = \overline\omega^2 xyz-2\omega.$$
Substituting, we have
$$\text{Im}\left((18-17i)x y z +(6-17i)\left(\overline\omega^2 xyz-2\omega\right)\right) = 0,$$
or
$$\text{Im}\left(\left(-\frac{3164}{25}-\frac{2102}{25}i\right)x y z -\frac{472}{5}+\frac{254}{5}i\right)=0.$$
This implies
$$1051\text{Re}(xyz)+1582\text{Im}(xyz) = 635.$$
The only solutions to
\begin{equation}
\begin{cases}
1051 a + 1582b =635\\
a^2+b^2=1
\end{cases}
\end{equation}
are $\left(-\frac{3}{5}, \frac{4}{5}\right)$ and $\left(\frac{699833}{721465},-\frac{175344}{721465}\right)$. However, as other solutions have already pointed out, we know all the angles are in the first quadrant, so the second solution is extraneous. We conclude that $\text{Im}(xyz)=\frac{4}{5}$.
This last step is admittedly tedious, but potentially doable by hand: using the unit circle parameterization $t\mapsto\left(\frac{1-t^2}{1+t^2}, \frac{2t}{1+t^2}\right)$, this boils down to solving $843 t^2 -1582t -208=0.$ Either notice that this factors as $(t-2)(843t+104)$, or use the quadratic formula to get
$t = \frac{1582\pm\sqrt{3204100}}{2(843)} = \frac{1582\pm 1790}{2(843)}$.
Best Answer
Your original (set of) equations implies $\tan x=\frac{4}{3}$ but not the other way around. When you solve $\tan x=\frac{4}{3}$ you get the solutions of your original equation and the solutions $\sin x=\frac{-4}{5}$, $\cos x=\frac{-3}{5}$.