The expressions $$F(x) = \tan^{-1} \frac{\sqrt{x^2+4x-4}-x}{2}$$ and $$G(x) = \frac{1}{2} \sin^{-1} \frac{x-2}{x\sqrt{2}}$$ cannot be directly transformed into each other because they are not in fact equal; they differ by a constant of integration on the interval $x \ge 2(\sqrt{2}-1)$. Moreover, on the interval $x < -2(1+\sqrt{2})$, the two functions do not agree: their sum is constant, thus one or the other antiderivative has the incorrect sign. To see which one it is, let's look at $h(x) = \frac{x-2}{x \sqrt{2}}$ on this interval: clearly, it is positive (since if $x < 0$, $h(x) > 0$), and increasing (since $h'(x) > 0$). Furthermore, as $x \to -2(1+\sqrt{2})$ from the left, $h(x) \to 1$. So if we define the inverse sine in the usual way $\sin^{-1} : [-1,1] \to \left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$, then clearly $G(x)$ is an increasing function on $(-\infty, -2(1+\sqrt{2})]$. But this contradicts the requirement that $f(x) = (x \sqrt{x^2+4x-4})^{-1} < 0$ on this interval, for $G(x)$ is increasing implies $G'(x) = f(x) > 0$. Thus the answer in the book is incorrect for $x < -2(1+\sqrt{2})$.
The answer you obtained, $F$, does not have this problem. It is correct for all $x \in (-\infty, -2(1+\sqrt{2})] \cup [2(\sqrt{2}-1), \infty)$.
Now, to show that $F$ and $G$ are equivalent on $[2(\sqrt{2}-1),\infty)$, we claim that $F(x) - G(x) = \frac{\pi}{8}$. Then it suffices to compute $$ \tan \left( G(x) + \frac{\pi}{8} \right)$$ and show this is $(\sqrt{x^2+4x-4} - x)/2$, which is an exercise I leave to you.
This discrepancy between $F$ and $G$ that arises out of different methods of integration naturally suggests a question: was there an error in the method that produced $G$? Or was there an assumption that was made that did not actually hold? Is there a simple way to fix $G$ such that it does work on the entire interval for which the integrand is defined? Again, I leave these questions to you as an instructive exercise. I hope your investigation of this particular problem leads you to a deeper understanding of the meaning of antiderivatives, techniques of integration, and how to check if your computations make sense.
Best Answer
We can make the $u$-substitution $$u=5-\sqrt{x}$$ and solve for $x$ here. We get $x=(u-5)^2$ so $$\mathrm{d}x=(2u-10) \ \mathrm{d}u$$ Now to find the new bounds we use $u=5-\sqrt{x}$. So the bounds are $u=5-\sqrt{0} =5$ and $u=5-\sqrt{5}$. The integral is now $$\int_5^{5-\sqrt{5}}\! \sqrt{u} \cdot (2u-10)\ \mathrm{d}u$$ Which is easy to solve.