At first an example which I know how to treat.
Let's say we have the following integral
$\int dx \frac{1}{x^2+a^2}$
Now we do an analytic continuation of the constant $a$ to the complex numbers: $a \rightarrow ia$. This gives
$\int dx \frac{1}{x^2-a^2}$
which now has poles at $x = \pm a$. Let's say I want to evade these poles during the integration (small half-circles into the complex plane around the singular points), then I have to take the half-circle at $x = -a$ into the lower half-plane (integration counter-clockwise), and the half-circle at $x = a$ into the upper half-plane (integration clockwise). As can be seen in this picture:
>click<
Whether the integration of a half-circle is clockwise or counter-clockwise, determines the extra sign before the residual part of the integral to be minus or plus respectively.
Now let's get to my actual question.
Imagine, instead of the simple example above we have the following integral:
$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(a^2)\sin^2(x)}$
where $f(a^2) = \frac{\text{polynomial in }a^2}{\text{other polynomial in }a^2}$ and $0 < f(a^2) < 1$ for all $a$.
Since $0 < f(a^2) < 1$, there are no singularities in the integral. Now we do the analytic continuation $a \rightarrow ia$. This gives
$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(-a^2)\sin^2(x)}$
and it is given that always $f(-a^2) > 1$. That causes a singularity to appear in $x = \arcsin(\frac{1}{\sqrt{f(-a^2)}})$. (Before the analytic continuation there was no singularity, because $\arcsin(y) $ is defined only for $-1\leq y\leq 1$). It is not hard to compute the residue at this singularity, but now it is less transparent, whether we should evade the pole drawing the small half-circle into the upper or into the lower half-plane. This little detail determines the sign of the residual part of the integral (I would like to separate it in the residual and principal parts). How can I determine whether I should draw the half-circle in the lower or in the upper half-plane in the general case, or in this situation at least?
Best Answer
Notice that in the picture you linked to, the poles moved from $\pm ia$ toward the real axis along counter-clockwise arcs that did not cross the real axis itself (where the integration contour lies). It is this path of approach of the poles toward the real axis that determines how the integration contour needs to be deformed. Informally, the poles make little "dents" in the integration contour as they approach it. If the poles were moved from $\pm ia$ toward the real axis along clockwise arcs, then these dents would be in the other direction and you would get a different sign in front of the residual part of the integral and hence a different answer overall. This shows that the final answer for the integral has branch cuts in its analytic dependence on $a$.
The answer to your main question depends on the way the location of the pole at $x_z=\arcsin(f(z^2)^{-1/2})$ as a function of $z$. You need to explicitly pick a path in the complex $z$-plane that takes you from $a$ to $-ia$ such that the pole $x_z$ does not cross your integration contour. As the pole approaches the real axis, it will make a dent in your integration contour, which will determine the way you need to deform it.