In a case like this, you have to contend with a residue at infinity. You can see this from the contour in your hint: as the radius of the circular contour $R \to \infty$, the integral about that contour approaches
$$i R \int_0^{2 \pi} d\phi \, \frac{e^{i \phi}}{\left(R^2 e^{i 2 \phi}-R^3 e^{i 3 \phi}\right)^{1/3}} \sim i 2 \pi (-1)^{-1/3}$$
However, you have to subtract out that dumbbell piece which excludes the branch points from the interior. This is where things get tricky. To this effect, we define
$$z^{-2/3} = e^{-(2/3) \log{z}}$$
such that $\arg{z} \in [-\pi,\pi)$. This definition is a result of the original branch cut of this factor being $(-\infty,0]$. The reaosn the branch is defined this way is because the argument of the log is negative real along the branch cut. Further define
$$(1-z)^{-1/3} = e^{-(1/3) \log{(1-z)}}$$
such that $\arg{(1-z)} \in [0,2\pi)$. This definition is a result of the original branch cut of this factor being $(-\infty,1]$. The reason the branch is defined like this is because, along the branch, $1-z$ is positive real along the branch cut.
To summarize, on the lines above and below the real axis, $z=x \in [0,1]$ and therefore $\arg{z} = 0$. On the line above the real axis, however, $\arg{(1-z)} = 2 \pi$. Therefore above the real axis, $z^{-2/3} (1-z)^{-1/3} = x^{-2/3} (1-x)^{-1/3} e^{-i 2 \pi/3}$ Below the real axis, $z^{-2/3} (1-z)^{-1/3} = x^{-2/3} (1-x)^{-1/3}$ because there, $\arg{(1-z)} = 0$.
Further, it should be clear that the integrals about the small circular arcs of radius $\epsilon$ around the branch points vanish as $\epsilon^{1/3}$.
Therefore, we may write
$$\left ( 1-e^{-i 2 \pi/3}\right) \int_0^1 dx \: x^{-2/3} (1-x)^{-1/3} = i 2 \pi (-1)^{-1/3}$$
Because that residue was calculated from the $1-z$ term, then $-1=e^{i \pi}$ and we have
$$\int_0^1 dx \: x^{-2/3} (1-x)^{-1/3} = i 2 \pi \frac{e^{-i \pi/3}}{1-e^{-i 2 \pi/3}} = \frac{\pi}{\sin{(\pi/3)}} = \frac{2 \pi}{\sqrt{3}}$$
You really don't want to break the square root up; it's not about separate branch cuts, but about a single branch cut for the square root of the quadratic. Presumably, you want the branch of $g(z)=\sqrt{1-z^2}$ with $g(i)=+\sqrt2$. When you then compute the residue of $f$ at $\infty$, you have to interpret the residue at $0$ of
$$\sqrt{1-\big(\frac1u\big)^2} = \frac{\sqrt{u^2-1}}{\sqrt{u^2}},$$
and one has to check that the ambiguities in the two square roots cancel out.
Best Answer
Choose the branch cuts as $(-\infty,-1]$ for $(z+1)^{-1/2}$ and $(-\infty,+1]$ for $(z-1)^{-1/2}$.
Then, $f(z) =(z^2-1)^{-1/2}$ is continuous across the negative real axis and the "effective" branch cut is $[-1,+1]$.
We will integrate $f$ on the clockwise contour $C$, which is the "dog-bone" clock-wise contour that encompasses $z=\pm 1$. To that end, we have
$$\begin{align} \oint_C f(z) dz &= \oint_C (z+1)^{-1/2} (z-1)^{-1/2} dz\\\\ &=\int_{-1}^1 \frac{dx}{+\sqrt{x^2-1}}\,dx+\int_{1}^{-1} \frac{dx}{-\sqrt{x^2-1}}\,dx\\\\ &=4\int_{0}^1 \frac{dx}{\sqrt{x^2-1}}\,dx \end{align}$$
Note that we tacitly used the fact that the contributions to the small "circles" (i.e., at the ends of the contour) around $z=\pm 1$ tend to zero as the radii of those circles approach zero.
We now compute the residue at infinity (Note: This is equivalent to evaluating the integral of $f$ on a counter-clockwise spherical contour of radius $R$ in the limit as $R \to \infty$). This is given by
$$\text{Res}_{z=\infty} f(z)=\text{Res}_{z=0} \left(-\frac{1}{z^2}f\left(\frac{1}{z}\right)\right)=-1$$
Putting it together gives
$$4\int_{0}^1 \frac{dx}{\sqrt{x^2-1}}\,dx=-2\pi i(-1)$$
from which we have
$$\int_{0}^1 \frac{dx}{\sqrt{x^2-1}}\,dx=i\pi/2$$