If $a<0$ we have $b:=-a>0$, and
$$
J(a):=\int_{-\infty}^\infty e^{iax^2}\,dx=\int_{-\infty}^\infty e^{-ibx^2}\,dx=\overline{J(b)}=\overline{J(-a)}.
$$
Therefore, it is enough to evaluate $J(a)$ for $a>0$. So from now on we suppose that $a>0$.
Notice that
$$
J(a)=\int_{-\infty}^\infty e^{iax^2}\,dx=\frac{1}{\sqrt{a}}\int_{-\infty}^\infty e^{ix^2}\,dx=\frac{1}{\sqrt{a}}J(1).
$$
Hence, we only have to compute $J(1)$.
Given $r>0$ we denote by $\Gamma_R$ the boundary of
$$
\Omega_r=\left\{z \in \mathbb{C}:\ |z|\le r,\ 0\le \arg z\le \frac{\pi}{4}\right\}.
$$
Consider the parametrization $\gamma_r:[0,1] \to \mathbb{C}$ of $\Gamma_r$ given by
$$
\gamma_r(t)=\begin{cases}
\gamma_1(3t) & \text{ for } 0 \le t \le \frac13\\
\gamma_2(3t-1) & \text{ for } \frac13 < t \le \frac23\\
\gamma_3(3t-2) & \text{ for } \frac23 < t \le 1
\end{cases},
$$
where
$$
\gamma_1(t)= rt,\
\gamma_2(t)= r\exp\left(i\frac{\pi t}{4}\right),\
\gamma_3(t)= r(1-t)\exp\left(i\frac{\pi}{4}\right).
$$
Since
$$
f: \mathbb{C} \to \mathbb{C},\ f(z):=e^{iz^2}
$$
is holomorphic and $\Omega_r$ is simply connected, we have thanks to the Cauchy Integral formula:
$$
0=:\int_{\Gamma_r}f(z)\,dz=\sum_{k=1}^3I_k(r),
$$
with
$$
I_k(r):=\int_{\frac{k-1}{3}}^{\frac{k}{3}}\gamma_R'(t)f(\gamma_R(t))\,dt, \ k=1,2,3.
$$
I claim that
$$
\lim_{r\to \infty}I_2(r)=0.
$$
In fact, since $\sin \theta \ge \frac{\theta}{2}$ for $\theta \le \theta \le \frac{\pi}{2}$, we have:
\begin{eqnarray}
|I_2(r)|&=&\left|\int_0^13\gamma_2'(3t-1)f(\gamma_2(3t-1))\,dt\right|=\left|\int_0^1\gamma_2'(t)f(\gamma_2(t))\,dt\right|\\
&\le& \int_0^1\frac{\pi r}{4}\left|\exp\left\{ir^2\cos\left(\frac{\pi t}{2}\right)-r^2\sin\left(\frac{\pi t}{2}\right)\right\}\right|\,dt\\
&=&\int_0^1\frac{r\pi}{4}\exp\left[-r^2\sin\left(\frac{\pi t}{2}\right)\right]\,dt\\
&\le&\int_0^1\frac{r\pi}{4}\exp\left(-\frac{r^2\pi t}{4}\right)\,dt
=\frac{1}{r}\left[1-\exp\left(-\frac{\pi r^2}{4}\right)\right].
\end{eqnarray}
Thus $\lim_{r \to \infty}I_2(r)=0$ as claimed.
Clearly
\begin{eqnarray}
I_3(r)&=&\int_{\frac23}^13\gamma_3'(3t-2)f(\gamma_3(3t-2))\,dt=\int_0^1\gamma_3'(t)f(\gamma_3(t))\,dt\\
&=&-r\exp\left(i\frac{\pi}{4}\right)\int_0^1\exp\left(-r^2t^2\right)\,dt=-\exp\left(i\frac{\pi}{4}\right)\int_0^{r}\exp\left(-x^2\right)\,dx\\
I_1(r)&=&\int_0^{\frac13}3\gamma_1'(3t)f(\gamma_1(3t))\,dt=r\int_0^1\exp\left(ir^2t^2\right)\,dt=\int_0^{r}\exp\left(ix^2\right)\,dx.
\end{eqnarray}
So we have
$$
0=\sum_{k=1}^3I_k(r)=\int_0^{r} e^{ix^2}\,dx-\exp\left(i\frac{\pi}{4}\right)\int_0^{r} e^{-x^2}\,dx+I_2(r).
$$
This implies that
\begin{eqnarray}
J(1)&=&2\int_0^\infty e^{ix^2}\,dx=2\exp\left(i\frac{\pi}{4}\right)\int_0^\infty e^{-x^2}\,dx=\exp\left(i\frac{\pi}{4}\right)\int_{-\infty}^\infty e^{-x^2}\,dx\\
&=&\sqrt{\pi}\exp\left(i\frac{\pi}{4}\right)=(1+i)\sqrt{\frac{\pi}{2}}.
\end{eqnarray}
Thus we have
$$
J(a)=\begin{cases}
(1+i)\sqrt{\frac{\pi}{2a}} & \text{ for } a>0\\
(1-i)\sqrt{\frac{\pi}{-2a}} & \text{ for } a<0
\end{cases}.
$$
For the contour you describe in your text, you have to indent about the poles at $z=0$ and $z=1$. In that case, the contour integral
$$\oint_C dz \frac{e^{i a z}}{e^{2 \pi z}-1}$$
is split into $6$ segments:
$$\int_{\epsilon}^R dx \frac{e^{i a x}}{e^{2 \pi x}-1} + i \int_{\epsilon}^{1-\epsilon} dy \frac{e^{i a R} e^{-a y}}{e^{2 \pi R} e^{i 2 \pi y}-1} + \int_R^{\epsilon} dx \frac{e^{-a} e^{i a x}}{e^{2 \pi x}-1} \\+ i \int_{1-\epsilon}^{\epsilon} dy \frac{ e^{-a y}}{e^{i 2 \pi y}-1} + i \epsilon \int_{\pi/2}^0 d\phi \:e^{i \phi} \frac{e^{i a \epsilon e^{i \phi}}}{e^{2 \pi \epsilon e^{i \phi}}-1}+ i \epsilon \int_{2\pi}^{3 \pi/2} d\phi\: e^{-a} e^{i \phi} \frac{e^{i a \epsilon e^{i \phi}}}{e^{2 \pi \epsilon e^{i \phi}}-1}$$
The first integral is on the real axis, away from the indent at the origin. The second integral is along the right vertical segment. The third is on the horizontal upper segment. The fourth is on the left vertical segment. The fifth is around the lower indent (about the origin), and the sixth is around the upper indent, about $z=i$.
We will be interested in the limits as $R \rightarrow \infty$ and $\epsilon \rightarrow 0$. The first and third integrals combine to form, in this limit,
$$(1-e^{-a}) \int_0^{\infty} dx \frac{e^{i a x}}{e^{2 \pi x}-1}$$
The fifth and sixth integrals combine to form, as $\epsilon \rightarrow 0$:
$$\frac{i \epsilon}{2 \pi \epsilon} \left ( -\frac{\pi}{2}\right) + e^{-a} \frac{i \epsilon}{2 \pi \epsilon} \left ( -\frac{\pi}{2}\right) = -\frac{i}{4} (1+e^{-a}) $$
The second integral vanishes as $R \rightarrow \infty$. The fourth integral, however, does not, and must be evaluated, at least partially. We rewrite it, as $\epsilon \rightarrow 0$:
$$-\frac{1}{2} \int_0^1 dy \frac{e^{-a y} e^{- i \pi y}}{\sin{\pi y}} = -\frac{1}{2} PV\int_0^1 dy \: e^{-a y} \cot{\pi y} + \frac{i}{2} \frac{1-e^{-a}}{a}$$
where
$$PV\int_0^1 dy \: e^{-a y} \cot{\pi y} = \lim_{\epsilon \to 0} \int_{\epsilon}^{1-\epsilon} dy \: e^{-a y} \cot{\pi y}$$
is the Cauchy principal value of that integral. By Cauchy's theorem, the contour integral is zero because there are no poles within the contour. Thus,
$$(1-e^{-a}) \int_0^{\infty} dx \frac{e^{i a x}}{e^{2 \pi x}-1} -\frac{i}{4} (1+e^{-a}) -\frac{1}{2} PV \int_0^1 dy \: e^{-a y} \cot{\pi y} + \frac{i}{2} \frac{1-e^{-a}}{a}=0$$
Now take the imaginary part of the above equation - note that the nasty Cauchy PV integral drops out - and get
$$(1-e^{-a}) \int_0^{\infty} dx \frac{\sin{ a x}}{e^{2 \pi x}-1} = \frac{1}{4} (1+e^{-a})-\frac{1}{2} \frac{1-e^{-a}}{a}$$
or, after a little algebra and simplifying things, we get:
$$\int_0^{\infty} dx \frac{\sin{ a x}}{e^{2 \pi x}-1} = \frac14 \coth{\left (\frac{a}{2}\right )} - \frac{1}{2 a}$$
Best Answer
Combine the 4th and 6th integrals to get the Cauchy principal value:
$$PV \int_{\pi}^0 dy \frac{e^{-a y}}{1+e^{i y}} = PV \int_{\pi}^0dy \frac{e^{-a y}}{2 \cos{(y/2)}} e^{-iy/2} $$
You still need to evaluate the imaginary part of the integral.