The reason for domain restrictions is mainly because we want the "trig functions" to truly be functions in the strict mathematical sense.
That means for every element in the domain the function must produce exactly one function value.
Now one thing about functions is they don't always work equally well in both directions.
The definition of a function says you can get from any point in the domain to a unique point in the range; it says nothing about going from the range to the domain.
As a real-life analogy, there are machines that can turn standing trees into wood chips, but not (yet) any machine that can turn wood chips into a standing tree.
When a function $f$ has a genuine inverse function $f^{-1},$ the inverse really does reverse the effect of the original function:
$f^{-1}(f(x)) = x$ for any $x$ in the domain of $f.$
This works only for a function that is one-to-one, that is, when each value in the range comes from just one unique value in the domain.
As soon as you find even two values in the domain of a function that give the same output value when you use them as input to your function, you know there will not be a genuine inverse function.
The best we can do in cases like that is choose part of the domain of the original function, throw the rest away, and construct a limited "inverse" function that maps the range back to that part of the original function's domain.
So it is reasonable to expect that the domain of the inverse function will be the range of the original function (it certainly cannot be more than that), but it is not reasonable to expect in general that the range of the inverse function will be the domain of the original function. That only works when the original function is one-to-one.
The sine and cosine functions are not one-to-one, so it will not be the case that the range of $\arcsin(y)$ will be the domain of $\sin(x)$
or that you will have $\arccos(\cos(x)) = x$ for every number $x.$
The solution is not to redefine the original function so that it is invertible,
such as by making $\cos(x)$ undefined except when $0 \leq x \leq \pi$
(the range of $\arccos$).
Functions like the sine and cosine are far too useful when we allow them to be many-to-one (multiple input values $x$ that can produce the same output result $f(x)$);
we'd be giving up too much to make them truly invertible.
So we put up with an asymmetric situation: $\cos(x)$ is defined for all $x,$
but $\arccos(y)$ can only "get back" to a small subset of the values for which
$\cos(x)$ is defined, and we choose that subset to be $[0,\pi].$
The OP's system in the $x_i$,
$$\begin{cases}
-x_2^2+x_1x_3-x_4^2-x_2x_5-x_4x_5=0 \\
x_1^2-x_2^2+x_1x_4-3x_3x_4-x_2x_5-3x_3x_5=0 \\
x_1x_2-x_2x_3+x_2x_4-x_4x_5-x_5^2 =0 \\
3x_2x_3-x_2x_4-x_1x_5=0\\
3x_3^2-x_1x_4-x_4^2-x_2x_5-x_5^2=0
\end{cases}$$
DOES have non-trivial integer solutions. Some small examples $x_1, x_2, x_3, x_4, x_5$ are,
$$18, 7, -19, 15, -28$$
$$133, 57, 46, -37, 75$$
$$380, 665, -97, 81, -651$$
and probably an infinite more.
Added details:
Given a system of $n$ equations in $n$ unknowns, we can generally resolve it into one equation in one unknown using resultants.
The trick is to find the simplest equation to "cleave" the system. In the OP's case, it is the 4th one. Solve for its $x_5$, and we find,
$$x_5 =\frac{3 x_2 x_3 - x_2 x_4}{x_1}$$
Substitute this into the 1st and solve for $x_3$,
$$x_3 = \frac{x_1 x_2^2 - x_2^2 x_4 + x_1 x_4^2 - x_2 x_4^2}{x_1^2 - 3 x_2^2 - 3 x_2 x_4}$$
Substitute this $x_5, x_3$ and we find the remaining 3 equations become the SAME cubic in 3 variables $x_4, x_1, x_2$. Why this is so is a peculiarity of this system.
It was then easy to use Mathematica to test small values such that the cubic in $x_4$ factors.
Best Answer
There is a way to construct a possible solution using only conjugates , without applying squaring.
To get something useful, we want to multiply both side of the equation by the conjugate $3+\sqrt {15x+9}\neq 0\thinspace $ :
$$\begin{align}x\left(3+\sqrt {15x+9}\right)\left(4x^2+13x-14\right)=-15x\end{align}$$
Since $3+\sqrt {15x+9}\neq 0$ for all $x\geq -\frac 35$, this implies that $x_1=0$ is a solution. Therefore, to find other possible roots, we can proceed by dividing both side of the equation by $x\thinspace (x\neq 0\thinspace)$ :
$$ \begin{align}\left(3+\sqrt {15x+9}\right)\left(4x^2+13x-14\right)=-15\end{align} $$
By rearranging the left-hand side of the equation, we have :
$$ \begin{align}\left(3+\sqrt {15x+9}\right)\left(4x^2+13x-12\right)-2\left(3+\sqrt {15x+9}\right)=-15\end{align} $$
$$ \begin{align}\left(3+\sqrt {15x+9}\right)\left(4x^2+13x-12\right)=2\sqrt{15x+9}-9\end{align} $$
Then, multiplying both side of the equation by the conjugate $2\sqrt{15x+9}+9\neq 0\thinspace$, yields :
$$ \begin{align}\left(2\sqrt{15x+9}+9\right)\left(3+\sqrt{15x+9}\right)\left(x+4\right)\left(4x-3\right)=15\left(4x-3\right)\end{align} $$
Thus, based on the equivalence between the mathematical steps, we determine that $x_2=\frac 34$ is the second real root of the original equation, since $\thinspace 4x-3\thinspace$ is the common factor of the left and right sides of the equation.
Finally, we need to solve :
$$ \begin{align}\overbrace{\left(9+2\sqrt{15x+9}\right)}^{\ge 9}\thinspace \overbrace {\left(3+\sqrt {15x+9}\right)}^{\ge 3}\thinspace\overbrace {\left(x+4\right)}^{>3}=15\end{align} $$
However, we see that the last equation we obtained above has no real roots. Therefore, the original equation has only $2$ real roots : $\thinspace x\in\left\{0,\frac 34\right\}\thinspace.$
This completes the solution.
$\rm {Comment:}$
Remember that, this is not correct to generalize the method we used. The method only works on specific instances. Indeed, replace $\sqrt {15x+9}$ with the radical expression $\sqrt {7x+9}$ in the original equation, then we will definitely have to apply squaring operations and use Galois theory.