Without loss of generality we can assume $c>0$ , since $c<0$ only changes the sign of the integral.
As already discussed in the comments, Glasser's master theorem does not apply here. We can instead write the integral as
$$ I = \mathrm{e}^{2ab} \int \limits_0^c \mathrm{e}^{-(a x + b/x)^2} \, \mathrm{d} x \, .$$
Now we would like to let $a x + b/x = t$ . Since this map is not bijective, we need to be careful here! For $0<x<\sqrt{b/a}$ the correct inverse is
$$x = \frac{1}{2a} \left(t-\sqrt{t^2-4ab}\right) \, ,$$
while for $x > \sqrt{b/a}$ we have to use
$$x = \frac{1}{2a} \left(t+\sqrt{t^2-4ab}\right) \, .$$
For $c \leq \sqrt{b/a} \, \Leftrightarrow \, a c - b/c \leq 0$ we only need the first version. The substitution yields
$$ I = \frac{\mathrm{e}^{2ab}}{2a} \int \limits_{ac+b/c}^\infty \left[\frac{t}{\sqrt{t^2-4ab}} - 1\right]\mathrm{e}^{-t^2} \, \mathrm{d} t \, . $$
Changing variables once more for the first part ($u = \sqrt{t^2 - 4ab}$) and using the complementary error function $\operatorname{erfc} = 1-\operatorname{erf}$ we obtain
$$ I = \frac{\sqrt{\pi}}{4a} \left[\mathrm{e}^{-2ab} \operatorname{erfc}(|ac - b/c|) - \mathrm{e}^{2ab} \operatorname{erfc}(ac + b/c)\right] \, .$$
For $c > \sqrt{b/a} \, \Leftrightarrow \, a c - b/c > 0$ we have to split the integral:
\begin{align}
I &= \mathrm{e}^{2ab} \left\{\int \limits_0^\sqrt{b/a} \mathrm{e}^{-(a x + b/x)^2} \, \mathrm{d} x + \int \limits_\sqrt{b/a}^c \mathrm{e}^{-(a x + b/x)^2} \, \mathrm{d} x \right\} \\
&= \frac{\mathrm{e}^{2ab}}{2a} \left\{\int \limits_{2\sqrt{ab}}^\infty \left[\frac{t}{\sqrt{t^2-4ab}} - 1\right]\mathrm{e}^{-t^2} \, \mathrm{d} t + \int \limits_{2\sqrt{ab}}^{ac+b/c} \left[1 + \frac{t}{\sqrt{t^2-4ab}}\right]\mathrm{e}^{-t^2} \, \mathrm{d} t \right\} \, .
\end{align}
Evaluating the remaining integrals as in the other case we find
$$ I = \frac{\sqrt{\pi}}{4a} \left[\mathrm{e}^{-2ab} [1+\operatorname{erf}(ac - b/c)] - \mathrm{e}^{2ab} \operatorname{erfc}(ac + b/c)\right] \, .$$
Both results can be simplified and the final result for arbitrary $c>0$ is
\begin{align}
I &= \frac{\sqrt{\pi}}{4a} \left[\mathrm{e}^{-2ab} \operatorname{erfc}(b/c - ac) - \mathrm{e}^{2ab} \operatorname{erfc}(b/c + ac)\right] \\
&= \frac{\sqrt{\pi}}{4a} \left[\mathrm{e}^{2ab} \operatorname{erf}(ac+b/c) + \mathrm{e}^{-2ab} \operatorname{erf}(ac-b/c) - 2 \sinh(2ab)\right] \, ,
\end{align}
which agrees with the result from the table (except for a constant, of course). Letting $c \to \infty$ we obtain the integral
$$ \int \limits_{-\infty}^\infty \mathrm{e}^{-a^2 x^2 - b^2 /x^2} \, \mathrm{d} x = \frac{\sqrt{\pi}}{a} \mathrm{e}^{-2ab} \, , $$
which can also be computed using the master theorem.
Together with the answer to your other question in terms of Owen's T-function we can write the final result as
$$I(a,b) = \pi \left[\arctan \left(\frac{b}{a}\right) - 2 \pi \operatorname{T} \left(\sqrt{2} a, \frac{b}{a}\right)\right] \, .$$
In particular, $I(a,b) \sim \frac{\pi b}{a}$ as $a \to \infty$,
$$ I(a,\infty) = \frac{\pi^2}{2} \operatorname{erf}(a)$$
and
$$ I(a,a) = \frac{\pi^2}{4} \operatorname{erf}^2(a) \, .$$
Best Answer
With the substitution $x=\frac12(1+\tanh t)$, we obtain $$\int^1_0 e^{-\frac1{c\,x\,(1-x)}}\,dx=\frac12\,\int^\infty_{-\infty}\frac1{\cosh^2 t}\,e^{-\frac4c\,\cosh^2 t}\,dt=\int^\infty_0\frac1{\cosh^2 t}\,e^{-\frac4c\,\cosh^2 t}\,dt,$$ so let's investigate $$I(x)=\int^\infty_0\frac1{\cosh^2 t}\,e^{-x\,\cosh^2 t}\,dt.$$ Since $$\frac{d}{dt}\frac1{\cosh t}e^{-x\,\cosh^2t}=\left(-\frac{\sinh t}{\cosh^2t}-2x\sinh t\right)e^{-x\,\cosh^2t},$$ we have $$\int^\infty_0\left(-\frac{\sinh^2 t}{\cosh^2t}-2x\sinh^2 t\right)e^{-x\,\cosh^2t}\,dt=\int^\infty_0\sinh t\,\frac{d}{dt}\frac1{\cosh t}e^{-x\,\cosh^2t}\,dt=-\int^\infty_0e^{-x\,\cosh^2t}\,dt$$ by partial integration. Now we know that $$1-\frac{\sinh^2 t}{\cosh^2t}=\frac1{\cosh^2t},$$ this means $$\int^\infty_0\frac1{\cosh^2 t}\,e^{-x\,\cosh^2 t}\,dt=2x\int^\infty_0\sinh^2t\,e^{-x\,\cosh^2 t}\,dt.$$ We make use of the well-known identities $$\sinh^2t=\frac12(\cosh2t-1)$$ and $$\cosh^2t=\frac12(\cosh2t+1)$$ to get $$I(x)=x\,e^{-x/2}\int^\infty_0(\cosh2t-1)\,e^{-x/2\,\cosh2t}\,dt.$$ Substituting $t\to t/2$ and taking into account the integral representation $$K_\nu(z)=\int^\infty_0e^{-z\cosh t}\,\cosh(\nu t)\,dt$$ (this is formula 10.32.9 in https://dlmf.nist.gov/10.32), we finally arrive at
$$I(x)=\frac{x}2\,e^{-x/2}(K_1(x/2)-K_0(x/2)).$$ As this almost looked too good to be true, I checked it numerically, with $x=4/c$ and various values of $c$ between $0.1$ and $10$, and found perfect agreement.